Commit 5237d6b0 authored by Cédric OLIVIER's avatar Cédric OLIVIER
Browse files

Merge branch 'feat/component' into 'master' 

feat: migrate to CI/CD component

See merge request to-be-continuous/s3!51
parents d17eb3b9 3123652e

.gitlab-ci.yml

+1 −1
Original line number Diff line number Diff line
@@ -10,7 +10,7 @@ include: 
    file: '/templates/validation.yml'

  - project: 'to-be-continuous/bash'

    ref: '3.3'

    file: 'templates/gitlab-ci-bash.yml'

    file: '/templates/gitlab-ci-bash.yml'

  - project: 'to-be-continuous/semantic-release'

    ref: '3.7'

    file: '/templates/gitlab-ci-semrel.yml'

README.md

+103 −67
Original line number Diff line number Diff line
@@ -8,13 +8,52 @@ It uses [s3cmd](https://github.com/s3tools/s3cmd) to control the S3 API endpoint 


## Usage



In order to include this template in your project, add the following to your `gitlab-ci.yml`:

This template can be used both as a [CI/CD component](https://docs.gitlab.com/ee/ci/components/#use-a-component-in-a-cicd-configuration) 

or using the legacy [`include:project`](https://docs.gitlab.com/ee/ci/yaml/index.html#includeproject) syntax.



### Use as a CI/CD component



Add the following to your `gitlab-ci.yml`:



```yaml

include:

  # 1: include the component

  - component: gitlab.com/to-be-continuous/s3/gitlab-ci-s3@7.1.0

    # 2: set/override component inputs

    inputs:

      # ⚠ this is only an example

      deploy-files: "website/"

      staging-disabled: "true"

      base-bucket-name: "wonder-doc"

      # use same bucket for all review envs

      review-bucket-name: "wonder-doc-review"

      # segregate review envs with prefixes

      review-prefix: "$CI_ENVIRONMENT_SLUG"

      region: "eu-west-0"

```



### Use as a CI/CD template (legacy)



Add the following to your `gitlab-ci.yml`:



```yaml

include:

  # 1: include the template

  - project: 'to-be-continuous/s3'

    ref: '7.1.0'

    file: '/templates/gitlab-ci-s3.yml'



variables:

  # 2: set/override template variables

  # ⚠ this is only an example

  S3_DEPLOY_FILES: "website/"

  S3_STAGING_DISABLED: "true"

  S3_BASE_BUCKET_NAME: "wonder-doc"

  # use same bucket for all review envs

  S3_REVIEW_BUCKET_NAME: "wonder-doc-review"

  # segregate review envs with prefixes

  S3_REVIEW_PREFIX: "$CI_ENVIRONMENT_SLUG"

  S3_REGION: "eu-west-0"

```



## Understand
@@ -66,7 +105,7 @@ You're free to enable whichever or both, and you can also choose your deployment 
The template might be used with other storage systems provided they are implementing a compatible API. 

In that case, you'll have to override the default `$S3_ENDPOINT_HOST` and `$S3_WEBSITE_ENDPOINT`variables.



| Provider               | `S3_ENDPOINT_HOST`   | `S3_WEBSITE_ENDPOINT` |

| Provider               | `endpoint-host` / `S3_ENDPOINT_HOST` | `website-endpoint` / `S3_WEBSITE_ENDPOINT` |

| ---------------------- | -------------------- | --------------------- |

| [Google Cloud Platform](https://cloud.google.com/storage/docs/interoperability) | `storage.googleapis.com` | _website hosting in GCP not supported by `s3cmd`_ |

| Microsoft Azure | requires using [Minio](https://min.io/)<br/>Read [this article](https://cloudblogs.microsoft.com/opensource/2017/11/09/s3cmd-amazon-s3-compatible-apps-azure-storage/) for further information | N/A |
@@ -140,17 +179,17 @@ Here are some advices about your **secrets** (variables marked with a :lock:): 


The S3 template uses some global configuration used throughout all jobs.



| Name                   | Description                                   | Default value     |

| Input / Variable | Description                                   | Default value     |

| ---------------------- | --------------------------------------------- | ----------------- |

| `S3_CMD_IMAGE`         | The Docker image used to run [s3cmd](https://s3tools.org/usage) commands | `registry.hub.docker.com/d3fk/s3cmd:latest` |

| `S3_ENDPOINT_HOST`     | Default S3 endpoint hostname (with port)      | `s3.amazonaws.com` (AWS) |

| `S3_HOST_BUCKET`       | Default DNS-style bucket+hostname:port template for accessing a bucket | `%(bucket)s.$S3_ENDPOINT_HOST` |

| `S3_REGION`            | Default region to create the buckets in (if not defined, the template won't create any) | _none_ |

| `cmd-image` / `S3_CMD_IMAGE` | The Docker image used to run [s3cmd](https://s3tools.org/usage) commands | `registry.hub.docker.com/d3fk/s3cmd:latest` |

| `endpoint-host` / `S3_ENDPOINT_HOST` | Default S3 endpoint hostname (with port)      | `s3.amazonaws.com` (AWS) |

| `host-bucket` / `S3_HOST_BUCKET` | Default DNS-style bucket+hostname:port template for accessing a bucket | `%(bucket)s.$S3_ENDPOINT_HOST` |

| `region` / `S3_REGION` | Default region to create the buckets in (if not defined, the template won't create any) | _none_ |

| :lock: `S3_ACCESS_KEY` | Default S3 service Access Key                 | **has to be defined** |

| :lock: `S3_SECRET_KEY` | Default S3 service Secret Key                 | **has to be defined** |

| `S3_BASE_BUCKET_NAME`  | Base bucket name                              | `$CI_PROJECT_NAME` ([see GitLab doc](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)) |

| `S3_PREFIX`            | Default S3 prefix to use as a root destination to upload objects in the S3 bucket | _none_ |

| `S3_SCRIPTS_DIR`       | Directory where S3 hook scripts are located   | `.`|

| `base-bucket-name` / `S3_BASE_BUCKET_NAME` | Base bucket name                              | `$CI_PROJECT_NAME` ([see GitLab doc](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)) |

| `prefix` / `S3_PREFIX` | Default S3 prefix to use as a root destination to upload objects in the S3 bucket | _none_ |

| `scripts-dir` / `S3_SCRIPTS_DIR` | Directory where S3 hook scripts are located   | `.`|



### Deployment jobs
@@ -158,13 +197,13 @@ Each environment has its own deployment job (associated with the right branch). 


It uses the following variables:



| Name                   | Description                                   | Default value     |

| Input / Variable | Description                                   | Default value     |

| ---------------------- | --------------------------------------------- | ----------------- |

| `S3_DEPLOY_ARGS`       | [s3cmd](https://s3tools.org/usage) command and options to deploy files to the bucket | `sync --recursive --delete-removed --acl-public --no-mime-magic --guess-mime-type` |

| `S3_DEPLOY_FILES`      | Pattern(s) of files to deploy to the S3 bucket| `public/` _(all files from `public` directory)_ |

| `S3_WEBSITE_DISABLED`  | Set to `true` to disable WebSite hosting by your S3 bucket    | _none_ (enabled by default) |

| `S3_WEBSITE_ARGS`      | [s3cmd](https://s3tools.org/usage) command and options to enable WebSite hosting on the bucket | `ws-create --ws-index=index.html --ws-error=404.html` |

| `S3_WEBSITE_ENDPOINT`  | Default WebSite endpoint url pattern (supports `%(bucket)s` and `%(location)s` placeholders).<br/>_only required when website hosting is not disabled_ | `http://%(bucket)s.s3-website.%(location)s.amazonaws.com` |

| `deploy-args` / `S3_DEPLOY_ARGS` | [s3cmd](https://s3tools.org/usage) command and options to deploy files to the bucket | `sync --recursive --delete-removed --acl-public --no-mime-magic --guess-mime-type` |

| `deploy-files` / `S3_DEPLOY_FILES` | Pattern(s) of files to deploy to the S3 bucket| `public/` _(all files from `public` directory)_ |

| `website-disabled` / `S3_WEBSITE_DISABLED` | Set to `true` to disable WebSite hosting by your S3 bucket    | _none_ (enabled by default) |

| `website-args` / `S3_WEBSITE_ARGS` | [s3cmd](https://s3tools.org/usage) command and options to enable WebSite hosting on the bucket | `ws-create --ws-index=index.html --ws-error=404.html` |

| `website-endpoint` / `S3_WEBSITE_ENDPOINT` | Default WebSite endpoint url pattern (supports `%(bucket)s` and `%(location)s` placeholders).<br/>_only required when website hosting is not disabled_ | `http://%(bucket)s.s3-website.%(location)s.amazonaws.com` |



If need be you could add your own hook script `s3-pre-deploy.sh` that will be triggered right before deploying files to

the S3 bucket.
@@ -205,16 +244,16 @@ They are **enabled by default** and can be disabled by setting the `S3_REVIEW_DI 


Here are variables supported to configure review environments:



| Name                     | Description                            | Default value     |

| Input / Variable | Description                            | Default value     |

| ------------------------ | -------------------------------------- | ----------------- |

| `S3_REVIEW_DISABLED`     | Set to `true` to disable `review` environments         | _none_ (enabled) |

| `S3_REVIEW_ENDPOINT_HOST`| S3 endpoint hostname (with port) for `review` env  _(only define if different from default)_ | `$S3_ENDPOINT_HOST` |

| `S3_REVIEW_REGION`       | Region to create the `review` buckets in (if not defined, the template won't create any) | `$S3_REGION` |

| :lock: `S3_REVIEW_ACCESS_KEY` | S3 service Access Key for `review` env  _(only define if different from default)_    | `$S3_ACCESS_KEY` |

| :lock: `S3_REVIEW_SECRET_KEY` | S3 service Secret Key for `review` env  _(only define if different from default)_    | `$S3_SECRET_KEY` |

| `S3_REVIEW_BUCKET_NAME`  | Bucket name for `review` env      | `"${S3_BASE_BUCKET_NAME}-${CI_ENVIRONMENT_SLUG}"` (ex: `myproject-review-fix-bug-12`) |

| `S3_REVIEW_PREFIX`  |  S3 prefix to use for `review` env _(only define if different from default)_ | `S3_PREFIX` |

| `S3_REVIEW_AUTOSTOP_DURATION`| The amount of time before GitLab will automatically stop `review` environments | `4 hours` |

| `review-disabled` / `S3_REVIEW_DISABLED` | Set to `true` to disable `review` environments         | _none_ (enabled) |

| `review-endpoint-host` / `S3_REVIEW_ENDPOINT_HOST` | S3 endpoint hostname (with port) for `review` env  _(only define to override default)_ | `$S3_ENDPOINT_HOST` |

| `review-region` / `S3_REVIEW_REGION` | Region to create the `review` buckets in (if not defined, the template won't create any) | `$S3_REGION` |

| :lock: `S3_REVIEW_ACCESS_KEY` | S3 service Access Key for `review` env  _(only define to override default)_    | `$S3_ACCESS_KEY` |

| :lock: `S3_REVIEW_SECRET_KEY` | S3 service Secret Key for `review` env  _(only define to override default)_    | `$S3_SECRET_KEY` |

| `review-bucket-name` / `S3_REVIEW_BUCKET_NAME` | Bucket name for `review` env      | `"${S3_BASE_BUCKET_NAME}-${CI_ENVIRONMENT_SLUG}"` (ex: `myproject-review-fix-bug-12`) |

| `review-prefix` / `S3_REVIEW_PREFIX` |  S3 prefix to use for `review` env _(only define to override default)_ | `prefix` / `S3_PREFIX` |

| `review-autostop-duration` / `S3_REVIEW_AUTOSTOP_DURATION` | The amount of time before GitLab will automatically stop `review` environments | `4 hours` |



### Integration environment configuration
@@ -224,15 +263,15 @@ It is **enabled by default** and can be disabled by setting the `S3_INTEG_DISABL 


Here are variables supported to configure the integration environment:



| Name                     | Description                            | Default value     |

| Input / Variable | Description                            | Default value     |

| ------------------------ | -------------------------------------- | ----------------- |

| `S3_INTEG_DISABLED`      | Set to `true` to disable the `integration` environment | _none_ (enabled) |

| `S3_INTEG_ENDPOINT_HOST` | S3 endpoint hostname (with port) for `integration` env  _(only define if different from default)_    | `$S3_ENDPOINT_HOST` |

| `S3_INTEG_REGION`        | Region to create the `integration` bucket in | `$S3_REGION` |

| :lock: `S3_INTEG_ACCESS_KEY` | S3 service Access Key for `integration` env  _(only define if different from default)_    | `$S3_ACCESS_KEY` |

| :lock: `S3_INTEG_SECRET_KEY` | S3 service Secret Key for `integration` env  _(only define if different from default)_    | `$S3_SECRET_KEY` |

| `S3_INTEG_BUCKET_NAME`   | Bucket name for `integration` env | `${S3_BASE_BUCKET_NAME}-integration` |

| `S3_INTEG_PREFIX`  |  S3 prefix to use for `integration` env _(only define if different from default)_ | `S3_PREFIX` |

| `integ-disabled` / `S3_INTEG_DISABLED` | Set to `true` to disable the `integration` environment | _none_ (enabled) |

| `integ-endpoint-host` / `S3_INTEG_ENDPOINT_HOST` | S3 endpoint hostname (with port) for `integration` env  _(only define to override default)_    | `$S3_ENDPOINT_HOST` |

| `integ-region` / `S3_INTEG_REGION` | Region to create the `integration` bucket in | `$S3_REGION` |

| :lock: `S3_INTEG_ACCESS_KEY` | S3 service Access Key for `integration` env  _(only define to override default)_    | `$S3_ACCESS_KEY` |

| :lock: `S3_INTEG_SECRET_KEY` | S3 service Secret Key for `integration` env  _(only define to override default)_    | `$S3_SECRET_KEY` |

| `integ-bucket-name` / `S3_INTEG_BUCKET_NAME` | Bucket name for `integration` env | `${S3_BASE_BUCKET_NAME}-integration` |

| `integ-prefix` / `S3_INTEG_PREFIX` |  S3 prefix to use for `integration` env _(only define to override default)_ | `prefix` / `S3_PREFIX` |



### Staging environment configuration
@@ -243,15 +282,15 @@ It is **enabled by default** and can be disabled by setting the `S3_STAGING_DISA 


Here are variables supported to configure the staging environment:



| Name                     | Description                            | Default value     |

| Input / Variable | Description                            | Default value     |

| ------------------------ | -------------------------------------- | ----------------- |

| `S3_STAGING_DISABLED`    | Set to `true` to disable the `staging` environment     | _none_ (enabled) |

| `S3_STAGING_ENDPOINT_HOST`| S3 endpoint hostname (with port) for `staging` env  _(only define if different from default)_   | `$S3_ENDPOINT_HOST` |

| `S3_STAGING_REGION`      | Region to create the `staging` bucket in | `$S3_REGION` |

| :lock: `S3_STAGING_ACCESS_KEY` | S3 service Access Key for `staging` env  _(only define if different from default)_    | `$S3_ACCESS_KEY` |

| :lock: `S3_STAGING_SECRET_KEY` | S3 service Secret Key for `staging` env  _(only define if different from default)_    | `$S3_SECRET_KEY` |

| `S3_STAGING_BUCKET_NAME` | Bucket name for `staging` env     | `${S3_BASE_BUCKET_NAME}-staging` |

| `S3_STAGING_PREFIX`  |  S3 prefix to use for `staging` env _(only define if different from default)_ | `S3_PREFIX` |

| `staging-disabled` / `S3_STAGING_DISABLED` | Set to `true` to disable the `staging` environment     | _none_ (enabled) |

| `staging-endpoint-host` / `S3_STAGING_ENDPOINT_HOST` | S3 endpoint hostname (with port) for `staging` env  _(only define to override default)_   | `$S3_ENDPOINT_HOST` |

| `staging-region` / `S3_STAGING_REGION` | Region to create the `staging` bucket in | `$S3_REGION` |

| :lock: `S3_STAGING_ACCESS_KEY` | S3 service Access Key for `staging` env  _(only define to override default)_    | `$S3_ACCESS_KEY` |

| :lock: `S3_STAGING_SECRET_KEY` | S3 service Secret Key for `staging` env  _(only define to override default)_    | `$S3_SECRET_KEY` |

| `staging-bucket-name` / `S3_STAGING_BUCKET_NAME` | Bucket name for `staging` env     | `${S3_BASE_BUCKET_NAME}-staging` |

| `staging-prefix` / `S3_STAGING_PREFIX` |  S3 prefix to use for `staging` env _(only define to override default)_ | `prefix` / `S3_PREFIX` |



### Production environment configuration
@@ -261,16 +300,16 @@ It is **enabled by default** and can be disabled by setting the `S3_PROD_DISABLE 


Here are variables supported to configure the production environment:



| Name                     | Description                            | Default value     |

| Input / Variable | Description                            | Default value     |

| ------------------------ | -------------------------------------- | ----------------- |

| `S3_PROD_DISABLED`       | Set to `true` to disable the `production` environment  | _none_ (enabled) |

| `S3_PROD_ENDPOINT_HOST`  | S3 endpoint hostname (with port) for `production` env  _(only define if different from default)_| `$S3_ENDPOINT_HOST` |

| `S3_PROD_REGION`         | Region to create the `production` bucket in | `$S3_REGION` |

| :lock: `S3_PROD_ACCESS_KEY` | S3 service Access Key for `production` env  _(only define if different from default)_    | `$S3_ACCESS_KEY` |

| :lock: `S3_PROD_SECRET_KEY` | S3 service Secret Key for `production` env  _(only define if different from default)_    | `$S3_SECRET_KEY` |

| `S3_PROD_BUCKET_NAME`    | Bucket name for `production` env  | `$S3_BASE_BUCKET_NAME` |

| `S3_PROD_DEPLOY_STRATEGY`| Defines the deployment to production strategy. One of `manual` (i.e. _one-click_) or `auto`. | `manual` |

| `S3_PROD_PREFIX`  |  S3 prefix to use for `production` env _(only define if different from default)_ | `S3_PREFIX` |

| `prod-disabled` / `S3_PROD_DISABLED` | Set to `true` to disable the `production` environment  | _none_ (enabled) |

| `prod-endpoint-host` / `S3_PROD_ENDPOINT_HOST` | S3 endpoint hostname (with port) for `production` env  _(only define to override default)_| `$S3_ENDPOINT_HOST` |

| `prod-region` / `S3_PROD_REGION` | Region to create the `production` bucket in | `$S3_REGION` |

| :lock: `S3_PROD_ACCESS_KEY` | S3 service Access Key for `production` env  _(only define to override default)_    | `$S3_ACCESS_KEY` |

| :lock: `S3_PROD_SECRET_KEY` | S3 service Secret Key for `production` env  _(only define to override default)_    | `$S3_SECRET_KEY` |

| `prod-bucket-name` / `S3_PROD_BUCKET_NAME` | Bucket name for `production` env  | `$S3_BASE_BUCKET_NAME` |

| `prod-deploy-strategy` / `S3_PROD_DEPLOY_STRATEGY` | Defines the deployment to production strategy. One of `manual` (i.e. _one-click_) or `auto`. | `manual` |

| `prod-prefix` / `S3_PROD_PREFIX` |  S3 prefix to use for `production` env _(only define to override default)_ | `prefix` / `S3_PREFIX` |



## Variants
@@ -282,11 +321,11 @@ This variant allows delegating your secrets management to a [Vault](https://www. 


In order to be able to communicate with the Vault server, the variant requires the additional configuration parameters:



| Name              | Description                            | Default value     |

| Input / Variable | Description                            | Default value     |

| ----------------- | -------------------------------------- | ----------------- |

| `TBC_VAULT_IMAGE` | The [Vault Secrets Provider](https://gitlab.com/to-be-continuous/tools/vault-secrets-provider) image to use (can be overridden) | `registry.gitlab.com/to-be-continuous/tools/vault-secrets-provider:master` |

| `VAULT_BASE_URL`  | The Vault server base API url          | _none_ |

| `VAULT_OIDC_AUD`  | The `aud` claim for the JWT | `$CI_SERVER_URL` |

| `vault-base-url` / `VAULT_BASE_URL` | The Vault server base API url          | _none_ |

| `vault-oidc-aud` / `VAULT_OIDC_AUD` | The `aud` claim for the JWT | `$CI_SERVER_URL` |

| :lock: `VAULT_ROLE_ID`   | The [AppRole](https://www.vaultproject.io/docs/auth/approle) RoleID | **must be defined** |

| :lock: `VAULT_SECRET_ID` | The [AppRole](https://www.vaultproject.io/docs/auth/approle) SecretID | **must be defined** |
@@ -300,7 +339,7 @@ Then you may retrieve any of your secret(s) from Vault using the following synta 


With:



| Name                             | Description                            |

| Parameter                        | Description                            |

| -------------------------------- | -------------------------------------- |

| `secret_path` (_path parameter_) | this is your secret location in the Vault server |

| `field` (_query parameter_)      | parameter to access a single basic field from the secret JSON payload |
@@ -310,20 +349,17 @@ With: 
```yaml

include:

  # main template

  - project: 'to-be-continuous/s3'

    ref: '7.1.0'

    file: '/templates/gitlab-ci-s3.yml'

  - component: gitlab.com/to-be-continuous/s3/gitlab-ci-s3@7.1.0

  # Vault variant

  - project: 'to-be-continuous/s3'

    ref: '7.1.0'

    file: '/templates/gitlab-ci-s3-vault.yml'

  - component: gitlab.com/to-be-continuous/s3/gitlab-ci-s3-vault@7.1.0

    inputs:

      # audience claim for JWT

      vault-oidc-aud: "https://vault.acme.host"

      vault-base-url: "https://vault.acme.host/v1"

      # $VAULT_ROLE_ID and $VAULT_SECRET_ID defined as a secret CI/CD variable



variables:

    # audience claim for JWT

    VAULT_OIDC_AUD: "https://vault.acme.host"

  # Secrets managed by Vault

  S3_ACCESS_KEY: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/my-backend/s3?field=access_key"

  S3_SECRET_KEY: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/my-backend/s3?field=secret_key"

    VAULT_BASE_URL: "https://vault.acme.host/v1"

    # $VAULT_ROLE_ID and $VAULT_SECRET_ID defined as a secret CI/CD variable

```

bumpversion.sh

+2 −2
Original line number Diff line number Diff line
@@ -27,13 +27,13 @@ if [[ "$curVer" ]]; then 
  log_info "Bump version from \\e[33;1m${curVer}\\e[0m to \\e[33;1m${nextVer}\\e[0m (release type: $relType)..."



  # replace in README

  sed -e "s/ref: '$curVer'/ref: '$nextVer'/" README.md > README.md.next

  sed -e "s/ref: *'$curVer'/ref: '$nextVer'/" -e "s/ref: *\"$curVer\”/ref: \”$nextVer\”/" -e "s/component: *\(.*\)@$curVer/component: \1@$nextVer/" README.md > README.md.next

  mv -f README.md.next README.md



  # replace in template and variants

  for tmpl in templates/*.yml

  do

    sed -e "s/\"$curVer\"/\"$nextVer\"/" "$tmpl" > "$tmpl.next"

    sed -e "s/command: *\[\"--service\", \"\(.*\)\", \"$curVer\"\]/command: [\"--service\", \"\1\", \"$nextVer\"]/" "$tmpl" > "$tmpl.next"

    mv -f "$tmpl.next" "$tmpl"

  done

else

kicker.json

+22 −24

File changed.

Preview size limit exceeded, changes collapsed.

logo.png

+14.6 KiB (19 KiB)
Loading image diff...