Commit c3c4a398 authored by Pierre Smeyers's avatar Pierre Smeyers
Browse files

Merge branch 'feat/shared-bucket-support' into 'master' 

multiple improvements

See merge request to-be-continuous/s3!31
parents e85674cf a9bdd09b

README.md

+72 −18
Original line number Diff line number Diff line
@@ -5,6 +5,8 @@ This project implements a generic GitLab CI template for any [S3](https://en.wik 


This is a basic and very cheap solution to host static pages websites as well as [progressive web applications](https://en.wikipedia.org/wiki/Progressive_web_application).



The template uses [s3cmd](https://github.com/s3tools/s3cmd) to control the S3 API endpoint and uploading objects.



## Overview



This template implements continuous delivery/continuous deployment for projects hosted on S3 platforms.
@@ -63,12 +65,25 @@ The S3 template uses some global configuration used throughout all jobs. 
| Name                   | description                                   | default value     |

| ---------------------- | --------------------------------------------- | ----------------- |

| `S3_CMD_IMAGE`         | The Docker image used to run [s3cmd](https://s3tools.org/usage) commands | `d3fk/s3cmd:latest` |

| `S3_ENDPOINT_HOST`     | Default S3 endpoint hostname (and port)       | **has to be defined** |

| `S3_ENDPOINT_HOST`     | Default S3 endpoint hostname (with port)      | `s3.amazonaws.com` (AWS) |

| `S3_HOST_BUCKET`       | Default DNS-style bucket+hostname:port template for accessing a bucket | `%(bucket)s.$S3_ENDPOINT_HOST` |

| `S3_REGION`            | Default region to create the buckets in (if not defined, the template won't create any) | _none_ |

| :lock: `S3_ACCESS_KEY` | Default S3 service Access Key                 | **has to be defined** |

| :lock: `S3_SECRET_KEY` | Default S3 service Secret Key                 | **has to be defined** |

| `S3_BASE_BUCKET_NAME`  | Base bucket name                              | `$CI_PROJECT_NAME` ([see GitLab doc](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)) |

| `S3_ROOT_PATH` | Default root path where files will be uploaded in the S3 bucket (:warning: don't forget the starting `/`)  | _none_ |

| `S3_PREFIX`            | Default S3 prefix to use as a root destination to upload objects in the S3 bucket | _none_ |

| `S3_SCRIPTS_DIR`       | Directory where S3 hook scripts are located   | `.`|



#### Using other S3-compatible systems than AWS



The template might be used with other storage systems provided they are implementing a compatible API. 

In that case, you'll have to override the default `$S3_ENDPOINT_HOST` and `$S3_WEBSITE_ENDPOINT`variables.



| Provider               | `S3_ENDPOINT_HOST`   | `S3_WEBSITE_ENDPOINT` |

| ---------------------- | -------------------- | --------------------- |

| [Google Cloud Platform](https://cloud.google.com/storage/docs/interoperability) | `storage.googleapis.com` | _website hosting in GCP not supported by `s3cmd`_ |

| Microsoft Azure | requires using [Minio](https://min.io/)<br/>Read [this article](https://cloudblogs.microsoft.com/opensource/2017/11/09/s3cmd-amazon-s3-compatible-apps-azure-storage/) for further information | N/A |

| [Flexible Engine (Orange Business Services)](https://docs.prod-cloud-ocb.orange-business.com/endpoint/) | `oss.<region>.prod-cloud-ocb.orange-business.com` (:warning: `<region>` must be set) | `https://%(bucket)s.oss-website.%(location)s.prod-cloud-ocb.orange-business.com` |



### Secrets management
@@ -110,13 +125,12 @@ Here are variables supported to configure review environments: 
| Name                     | description                            | default value     |

| ------------------------ | -------------------------------------- | ----------------- |

| `S3_REVIEW_DISABLED`     | Set to `true` to disable `review` environments         | _none_ (enabled) |

| `S3_REVIEW_ENDPOINT_HOST`| S3 endpoint hostname (and port) for `review` env  _(only define if different from default)_ | `$S3_ENDPOINT_HOST` |

| `S3_REVIEW_ENDPOINT_HOST`| S3 endpoint hostname (with port) for `review` env  _(only define if different from default)_ | `$S3_ENDPOINT_HOST` |

| `S3_REVIEW_REGION`       | Region to create the `review` buckets in (if not defined, the template won't create any) | `$S3_REGION` |

| :lock: `S3_REVIEW_ACCESS_KEY` | S3 service Access Key for `review` env  _(only define if different from default)_    | `$S3_ACCESS_KEY` |

| :lock: `S3_REVIEW_SECRET_KEY` | S3 service Secret Key for `review` env  _(only define if different from default)_    | `$S3_SECRET_KEY` |

| `S3_REVIEW_BUCKET_NAME`  | Bucket name for `review` env      | `"${S3_BASE_BUCKET_NAME}-${CI_ENVIRONMENT_SLUG}"` (ex: `myproject-review-fix-bug-12`) |

| `S3_REVIEW_ENVIRONMENT_SCHEME` | The review environment protocol scheme | `https` |

| `S3_REVIEW_ENVIRONMENT_DOMAIN` | The review environment domain. | _none_ |

| `S3_REVIEW_ROOT_PATH`  |  S3 bucket root path (prefix) for `review` env _(only define if different from default)_ | `S3_ROOT_PATH` |

| `S3_REVIEW_PREFIX`  |  S3 prefix to use for `review` env _(only define if different from default)_ | `S3_PREFIX` |



Note: By default, review `environment.url` will be built as `${S3_REVIEW_ENVIRONMENT_SCHEME}://${$CI_PROJECT_NAME}-${CI_ENVIRONMENT_SLUG}.${S3_REVIEW_ENVIRONMENT_DOMAIN}`
@@ -131,12 +145,12 @@ Here are variables supported to configure the integration environment: 
| Name                     | description                            | default value     |

| ------------------------ | -------------------------------------- | ----------------- |

| `S3_INTEG_DISABLED`      | Set to `true` to disable the `integration` environment | _none_ (enabled) |

| `S3_INTEG_ENDPOINT_HOST` | S3 endpoint hostname (and port) for `integration` env  _(only define if different from default)_    | `$S3_ENDPOINT_HOST` |

| `S3_INTEG_ENDPOINT_HOST` | S3 endpoint hostname (with port) for `integration` env  _(only define if different from default)_    | `$S3_ENDPOINT_HOST` |

| `S3_INTEG_REGION`        | Region to create the `integration` bucket in | `$S3_REGION` |

| :lock: `S3_INTEG_ACCESS_KEY` | S3 service Access Key for `integration` env  _(only define if different from default)_    | `$S3_ACCESS_KEY` |

| :lock: `S3_INTEG_SECRET_KEY` | S3 service Secret Key for `integration` env  _(only define if different from default)_    | `$S3_SECRET_KEY` |

| `S3_INTEG_BUCKET_NAME`   | Bucket name for `integration` env | `${S3_BASE_BUCKET_NAME}-integration` |

| `S3_INTEG_ENVIRONMENT_URL` | The integration environment url **including scheme** (ex: `https://my-project-integration.s3-website.nonpublic.domain.com`). Do not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that. | _none_ |

| `S3_INTEG_ROOT_PATH`  |  S3 bucket root path (prefix) for `integration` env _(only define if different from default)_ | `S3_ROOT_PATH` |

| `S3_INTEG_PREFIX`  |  S3 prefix to use for `integration` env _(only define if different from default)_ | `S3_PREFIX` |



#### Staging environment
@@ -150,12 +164,12 @@ Here are variables supported to configure the staging environment: 
| Name                     | description                            | default value     |

| ------------------------ | -------------------------------------- | ----------------- |

| `S3_STAGING_DISABLED`    | Set to `true` to disable the `staging` environment     | _none_ (enabled) |

| `S3_STAGING_ENDPOINT_HOST`| S3 endpoint hostname (and port) for `staging` env  _(only define if different from default)_   | `$S3_ENDPOINT_HOST` |

| `S3_STAGING_ENDPOINT_HOST`| S3 endpoint hostname (with port) for `staging` env  _(only define if different from default)_   | `$S3_ENDPOINT_HOST` |

| `S3_STAGING_REGION`      | Region to create the `staging` bucket in | `$S3_REGION` |

| :lock: `S3_STAGING_ACCESS_KEY` | S3 service Access Key for `staging` env  _(only define if different from default)_    | `$S3_ACCESS_KEY` |

| :lock: `S3_STAGING_SECRET_KEY` | S3 service Secret Key for `staging` env  _(only define if different from default)_    | `$S3_SECRET_KEY` |

| `S3_STAGING_BUCKET_NAME` | Bucket name for `staging` env     | `${S3_BASE_BUCKET_NAME}-staging` |

| `S3_STAGING_ENVIRONMENT_URL` | The staging environment url **including scheme** (ex: `https://my-project-staging.s3-website.nonpublic.domain`). Do not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that. | _none_ |

| `S3_STAGING_ROOT_PATH`  |  S3 bucket root path (prefix) for `staging` env _(only define if different from default)_ | `S3_ROOT_PATH` |

| `S3_STAGING_PREFIX`  |  S3 prefix to use for `staging` env _(only define if different from default)_ | `S3_PREFIX` |



#### Production environment
@@ -168,13 +182,49 @@ Here are variables supported to configure the production environment: 
| Name                     | description                            | default value     |

| ------------------------ | -------------------------------------- | ----------------- |

| `S3_PROD_DISABLED`       | Set to `true` to disable the `production` environment  | _none_ (enabled) |

| `S3_PROD_ENDPOINT_HOST`  | S3 endpoint hostname (and port) for `production` env  _(only define if different from default)_| `$S3_ENDPOINT_HOST` |

| `S3_PROD_ENDPOINT_HOST`  | S3 endpoint hostname (with port) for `production` env  _(only define if different from default)_| `$S3_ENDPOINT_HOST` |

| `S3_PROD_REGION`         | Region to create the `production` bucket in | `$S3_REGION` |

| :lock: `S3_PROD_ACCESS_KEY` | S3 service Access Key for `production` env  _(only define if different from default)_    | `$S3_ACCESS_KEY` |

| :lock: `S3_PROD_SECRET_KEY` | S3 service Secret Key for `production` env  _(only define if different from default)_    | `$S3_SECRET_KEY` |

| `S3_PROD_BUCKET_NAME`    | Bucket name for `production` env  | `$S3_BASE_BUCKET_NAME` |

| `S3_PROD_ENVIRONMENT_URL`| The production environment url **including scheme** (ex: `https://my-project.s3-website.public.domain.com`) Do not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that. | _none_ |

| `AUTODEPLOY_TO_PROD`     | Set this variable to auto-deploy to production. If not set deployment to production will be `manual` (default behaviour). | _none_ (disabled) |

| `S3_PROD_ROOT_PATH`  |  S3 bucket root path (prefix) for `production` env _(only define if different from default)_ | `S3_ROOT_PATH` |

| `S3_PROD_PREFIX`  |  S3 prefix to use for `production` env _(only define if different from default)_ | `S3_PREFIX` |



### Buckets namespacing



In its default configuration, the template manages (create/sync/delete) one S3 bucket per environment.



But you may also configure it to implement other policies. Here are several examples of alternate policies.



#### A single bucket with separate prefixes for each env



Here the `.gitlab-ci.yml` configuration for one shared bucket for all envs, each separated by prefix:



```yaml

variables:

  # use same bucket for all envs

  S3_REVIEW_BUCKET_NAME: "acme-bucket-shared"

  S3_INTEG_BUCKET_NAME: "acme-bucket-shared"

  S3_STAGING_BUCKET_NAME: "acme-bucket-shared"

  S3_PROD_BUCKET_NAME: "acme-bucket-shared"

  # segregate envs with prefixes

  S3_PREFIX: "$CI_ENVIRONMENT_SLUG"

```



#### Hybrid policy



Here the `.gitlab-ci.yml` configuration for one shared bucket for review envs and separate buckets for others:



```yaml

variables:

  # use same bucket for all review envs

  S3_REVIEW_BUCKET_NAME: "acme-bucket-review"

  S3_INTEG_BUCKET_NAME: "acme-bucket-integ"

  S3_STAGING_BUCKET_NAME: "acme-bucket-staging"

  S3_PROD_BUCKET_NAME: "acme-bucket-prod"

  # segregate review envs with prefixes

  S3_REVIEW_PREFIX: "$CI_ENVIRONMENT_SLUG"

```



### Deployment jobs
@@ -184,17 +234,21 @@ It uses the following variables: 


| Name                   | description                                   | default value     |

| ---------------------- | --------------------------------------------- | ----------------- |

| `S3_DEPLOY_ARGS`       | [s3cmd](https://s3tools.org/usage) command and options to deploy files to the bucket | `put --acl-public --no-mime-magic --guess-mime-type --recursive` |

| `S3_DEPLOY_ARGS`       | [s3cmd](https://s3tools.org/usage) command and options to deploy files to the bucket | `sync --recursive --delete-removed --acl-public --no-mime-magic --guess-mime-type` |

| `S3_DEPLOY_FILES`      | Pattern(s) of files to deploy to the S3 bucket| `public/` _(all files from `public` directory)_ |

| `S3_WEBSITE_DISABLED`  | Set to `true` to disable WebSite hosting by your S3 bucket    | _none_ (enabled by default) |

| `S3_WEBSITE_ARGS`      | [s3cmd](https://s3tools.org/usage) command and options to enable WebSite hosting on the bucket | `ws-create --ws-index=index.html` |

| `S3_WEBSITE_ARGS`      | [s3cmd](https://s3tools.org/usage) command and options to enable WebSite hosting on the bucket | `ws-create --ws-index=index.html --ws-error=404.html` |

| `S3_WEBSITE_ENDPOINT`  | Default WebSite endpoint url pattern (supports `%(bucket)s` and `%(location)s` placeholders).<br/>_only required when website hosting is not disabled_ | `http://%(bucket)s.s3-website.%(location)s.amazonaws.com` |



If need be you could add your own hook script `s3-pre-deploy.sh` that will be triggered right before deploying files to

the S3 bucket.



If the target bucket doesn't appear to exist, the template tries to create it.



### `s3-cleanup-review` job



This job allows destroying each review environment. Simply deletes the associated bucket.

This job allows destroying each review environment. Simply deletes the associated objects in the bucket.

After objects removal, if the bucket appears to be empty, also tries to delete the bucket.



### `s3-cleanup-all-review` job

kicker.json

+50 −47
Original line number Diff line number Diff line
@@ -12,7 +12,8 @@ 
    },

    {

      "name": "S3_ENDPOINT_HOST",

      "description": "Default S3 endpoint hostname (and port)",

      "description": "Default S3 endpoint hostname (with port)",

      "default": "s3.amazonaws.com",

      "mandatory": true

    },

    {
@@ -21,6 +22,10 @@ 
      "default": "%(bucket)s.$S3_ENDPOINT_HOST",

      "mandatory": true

    },

    {

      "name": "S3_REGION",

      "description": "Default region to create the buckets in (if not defined, the template won't create any)"

    },

    {

      "name": "S3_ACCESS_KEY",

      "description": "Default S3 service Access Key",
@@ -41,8 +46,8 @@ 
    },

    {

      "name": "S3_DEPLOY_ARGS",

      "description": "directory where S3 scripts (templates, hook scripts) are located",

      "default": "put --acl-public --no-mime-magic --guess-mime-type --recursive",

      "description": "[s3cmd](https://s3tools.org/usage) command and options to deploy files to the bucket",

      "default": "sync --recursive --delete-removed --acl-public --no-mime-magic --guess-mime-type",

      "advanced": true

    },

    {
@@ -55,15 +60,26 @@ 
      "description": "Disables WebSite hosting by your S3 bucket",

      "type": "boolean"

    },

    {

      "name": "S3_WEBSITE_ENDPOINT",

      "description": "Default WebSite endpoint url pattern (supports `%(bucket)s` and `%(location)s` placeholders)",

      "default": "http://%(bucket)s.s3-website.%(location)s.amazonaws.com"

    },

    {

      "name": "S3_WEBSITE_ARGS",

      "description": "[s3cmd](https://s3tools.org/usage) command and options to enable WebSite hosting on the bucket",

      "default": "ws-create --ws-index=index.html",

      "default": "ws-create --ws-index=index.html --ws-error=404.html",

      "advanced": true

    },

    {

      "name": "S3_ROOT_PATH",

      "description": "Default root path where files will be uploaded in the S3 bucket (:warning: don't forget the starting `/`)"

      "name": "S3_PREFIX",

      "description": "Default S3 prefix to use as a root destination to upload objects in the S3 bucket"

    },

    {

      "name": "S3_SCRIPTS_DIR",

      "description": "Directory where S3 hook scripts are located",

      "default": ".",

      "advanced": true

    }

  ],

  "features": [
@@ -75,7 +91,11 @@ 
      "variables": [

        {

          "name": "S3_REVIEW_ENDPOINT_HOST",

          "description": "S3 endpoint hostname (and port) for `review` env  _(only define if different from default)_"

          "description": "S3 endpoint hostname (with port) for `review` env  _(only define if different from default)_"

        },

        {

          "name": "S3_REVIEW_REGION",

          "description": "Region to create the `review` buckets in (if not defined, the template won't create any)"

        },

        {

          "name": "S3_REVIEW_ACCESS_KEY",
@@ -93,25 +113,14 @@ 
          "default": "${S3_BASE_BUCKET_NAME}-${CI_ENVIRONMENT_SLUG}",

          "advanced": true

        },

        {

          "name": "S3_REVIEW_ENVIRONMENT_SCHEME",

          "description": "The review environment protocol scheme",

          "default": "https",

          "mandatory": true

        },

        {

          "name": "S3_REVIEW_ENVIRONMENT_DOMAIN",

          "description": "The review environment domain (ex: `s3-website.nonpublic.domain`).\n\nBy default review `environment.url` will be built as `${S3_REVIEW_ENVIRONMENT_SCHEME}://${$CI_PROJECT_NAME}-${CI_ENVIRONMENT_SLUG}.${S3_REVIEW_ENVIRONMENT_DOMAIN}`",

          "mandatory": true

        },

        {

          "name": "CLEANUP_ALL_REVIEW",

          "description": "Enables a **manual** job to cleanup all review envs at once.\n\nYou may also use it to [schedule](https://docs.gitlab.com/ee/ci/pipelines/schedules.html) cloud resources cleanup. See documentation.",

          "type": "boolean"

        },

        {

          "name": "S3_REVIEW_ROOT_PATH",

          "description": "S3 bucket root path (prefix) for `review` env _(only define if different from default)_",

          "name": "S3_REVIEW_PREFIX",

          "description": "S3 prefix to use for `review` env _(only define if different from default)_",

          "advanced": true

        }

      ]
@@ -124,7 +133,11 @@ 
      "variables": [

        {

          "name": "S3_INTEG_ENDPOINT_HOST",

          "description": "S3 endpoint hostname (and port) for `integration` env  _(only define if different from default)_"

          "description": "S3 endpoint hostname (with port) for `integration` env  _(only define if different from default)_"

        },

        {

          "name": "S3_INTEG_REGION",

          "description": "Region to create the `integration` bucket in"

        },

        {

          "name": "S3_INTEG_ACCESS_KEY",
@@ -143,14 +156,8 @@ 
          "advanced": true

        },

        {

          "name": "S3_INTEG_ENVIRONMENT_URL",

          "type": "url",

          "description": "The integration environment url including scheme (ex: `https://my-project-integration.s3-website.nonpublic.domain`).\n\nDo not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that.",

          "mandatory": true

        },

        {

          "name": "S3_INTEG_ROOT_PATH",

          "description": "S3 bucket root path (prefix) for `integration` env _(only define if different from default)_",

          "name": "S3_INTEG_PREFIX",

          "description": "S3 prefix to use for `integration` env _(only define if different from default)_",

          "advanced": true

        }

      ]
@@ -163,7 +170,11 @@ 
      "variables": [

        {

          "name": "S3_STAGING_ENDPOINT_HOST",

          "description": "S3 endpoint hostname (and port) for `staging` env  _(only define if different from default)_"

          "description": "S3 endpoint hostname (with port) for `staging` env  _(only define if different from default)_"

        },

        {

          "name": "S3_STAGING_REGION",

          "description": "Region to create the `staging` bucket in"

        },

        {

          "name": "S3_STAGING_ACCESS_KEY",
@@ -182,14 +193,8 @@ 
          "advanced": true

        },

        {

          "name": "S3_STAGING_ENVIRONMENT_URL",

          "type": "url",

          "description": "The staging environment url including scheme (ex: `https://my-project-staging.s3-website.nonpublic.domain`).\n\nDo not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that.",

          "mandatory": true

        },

        {

          "name": "S3_STAGING_ROOT_PATH",

          "description": "S3 bucket root path (prefix) for `staging` env _(only define if different from default)_",

          "name": "S3_STAGING_PREFIX",

          "description": "S3 prefix to use for `staging` env _(only define if different from default)_",

          "advanced": true

        }

      ]
@@ -202,7 +207,11 @@ 
      "variables": [

        {

          "name": "S3_PROD_ENDPOINT_HOST",

          "description": "S3 endpoint hostname (and port) for `production` env  _(only define if different from default)_"

          "description": "S3 endpoint hostname (with port) for `production` env  _(only define if different from default)_"

        },

        {

          "name": "S3_PROD_REGION",

          "description": "Region to create the `production` bucket in"

        },

        {

          "name": "S3_PROD_ACCESS_KEY",
@@ -220,20 +229,14 @@ 
          "default": "${S3_BASE_BUCKET_NAME}",

          "advanced": true

        },

        {

          "name": "S3_PROD_ENVIRONMENT_URL",

          "type": "url",

          "description": "The production environment url including scheme (ex: `https://my-project.s3-website.public.domain.com`).\n\nDo not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that.",

          "mandatory": true

        },

        {

          "name": "AUTODEPLOY_TO_PROD",

          "type": "boolean",

          "description": "Set this variable to auto-deploy to production. If not set deployment to production will be manual (default behaviour)."

        },

        {

          "name": "S3_PROD_ROOT_PATH",

          "description": "S3 bucket root path (prefix) for `production` env _(only define if different from default)_",

          "name": "S3_PROD_PREFIX",

          "description": "S3 prefix to use for `production` env _(only define if different from default)_",

          "advanced": true

        }

      ]

templates/gitlab-ci-s3.yml

+86 −47

File changed.

Preview size limit exceeded, changes collapsed.