Commit 5cfcb659 authored by Pierre Smeyers's avatar Pierre Smeyers
Browse files

feat: add parallel:matrix support

parent 99de8f04

README.md

+61 −0
Original line number Diff line number Diff line
@@ -147,6 +147,56 @@ variables: 
  S3_REVIEW_PREFIX: "$CI_ENVIRONMENT_SLUG"

```



### Multiple environments support



In some situations, you might want to deploy multiple instances of your application to the same environment

(for instance in a monorepo context, or to deploy the same application with different configurations).



The S3 template supports this use case by leveraging GitLab's [`parallel:matrix`](https://docs.gitlab.com/ci/yaml/#parallelmatrix)

feature combined with the `S3_ENVIRONMENT_NAMESPACE` variable.



The `.s3-env-base` hidden job (extended by both `.s3-deploy` and `.s3-cleanup`) supports

`parallel:matrix` with the `S3_ENVIRONMENT_NAMESPACE` variable (that **shall begin with `/`**).



Here is an example that deploys both a front-end and a back-end to S3:



The S3 template allows deploying multiple environments in parallel. Use cases of this include:



- monorepo, where a single Git repository might host several separate deployable components or apps,

- multi-instances deployment of the same application.



This feature can be enabled using the [parallel matrix jobs](https://docs.gitlab.com/ee/ci/yaml/#parallelmatrix)

pattern at the `.s3-env-base` job level (this is the top parent job of all deployment jobs).

Environments namespacing is ensured by the `S3_ENVIRONMENT_NAMESPACE` variable (must start with a slash `/`).



Here is the example of the `.gitlab-ci.yml` file for a project deploying 2 static websites (_doc_ and _dev-guide_):



```yaml

.s3-env-base:

  parallel:

    matrix:

      - S3_ENVIRONMENT_NAMESPACE: /doc

        S3_SCRIPTS_DIR: doc

        S3_PREFIX: doc

        S3_DEPLOY_FILES: doc/public/

      - S3_ENVIRONMENT_NAMESPACE: /dev-guide

        S3_SCRIPTS_DIR: dev-guide

        S3_PREFIX: dev-guide

        S3_DEPLOY_FILES: dev-guide/public/



# ⚠ on_stop must be unset when defining parallel:matrix environments

# see: https://gitlab.com/gitlab-org/gitlab/-/issues/332247

s3-review:

  environment:

    on_stop: null

```



The above configuration will deploy 2 environments on each pipeline:



- on feature branches: `review/doc/$CI_COMMIT_REF_NAME` and `review/dev-guide/$CI_COMMIT_REF_NAME`

- on the integration branch: `integration/doc` and `integration/dev-guide` 

- on the production branch: `staging/doc` and `staging/dev-guide` (and finally `production/doc` and `production/dev-guide`)



### Deployment output variables



As seen above, the S3 template may support up to 4 environments (`review`, `integration`, `staging` and `production`).
@@ -159,6 +209,16 @@ Each deployment job produces _output variables_ that are propagated to downstrea 


They may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).



> [!important]

> If [multiple environments](#multiple-environments-support) are configured, the output variables are prefixed with a 

> sluggified value of the `S3_ENVIRONMENT_NAMESPACE` variable (stripped of punctuation characters and converted to lowercase):

> 

> * `<namespace_slug>_environment_type`: set to the type of environment (`review`, `integration`, `staging` or `production`),

> * `<namespace_slug>_environment_name`: the application name (see below),

> * `<namespace_slug>_environment_url`: set to the environment URL (whether determined statically or dynamically).

> 

> The output dotenv file will be `s3.env.<namespace_slug>` instead, and the dynamic variable `${environment_namespace}` can be used in your scripts and manifests to access the contextual value of `<namespace_slug>`.



## Configuration reference



### Secrets management
@@ -189,6 +249,7 @@ The S3 template uses some global configuration used throughout all jobs. 
| :lock: `S3_SECRET_KEY` | Default S3 service Secret Key                 | **has to be defined** |

| `base-bucket-name` / `S3_BASE_BUCKET_NAME` | Base bucket name                              | `$CI_PROJECT_NAME` ([see GitLab doc](https://docs.gitlab.com/ci/variables/predefined_variables/)) |

| `prefix` / `S3_PREFIX` | Default S3 prefix to use as a root destination to upload objects in the S3 bucket | _none_ |

| `environment-namespace` / `S3_ENVIRONMENT_NAMESPACE` | Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying [multiple environments](#multiple-environments-support))_<br/>:warning: must start with a slash `/` | _none_ |

| `scripts-dir` / `S3_SCRIPTS_DIR` | Directory where S3 hook scripts are located   | `.`|



### Deployment jobs

kicker.json

+5 −0
Original line number Diff line number Diff line
@@ -77,6 +77,11 @@ 
      "name": "S3_PREFIX",

      "description": "Default S3 prefix to use as a root destination to upload objects in the S3 bucket"

    },

    {

      "name": "S3_ENVIRONMENT_NAMESPACE",

      "description": "Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying multiple environments)_\n\n:warning: must start with a slash `/`",

      "advanced": true

    },

    {

      "name": "S3_SCRIPTS_DIR",

      "description": "Directory where S3 hook scripts are located",

templates/gitlab-ci-s3.yml

+46 −33
Original line number Diff line number Diff line
@@ -50,6 +50,12 @@ spec: 
    prefix:

      description: Default S3 prefix to use as a root destination to upload objects in the S3 bucket

      default: ''

    environment-namespace:

      description: |-

        Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying multiple environments)_

        

        :warning: must start with a slash `/`

      default: ''

    scripts-dir:

      description: Directory where S3 hook scripts are located

      default: .
@@ -183,6 +189,7 @@ variables: 
  S3_ENDPOINT_HOST: $[[ inputs.endpoint-host ]]

  S3_HOST_BUCKET: $[[ inputs.host-bucket ]]

  S3_BASE_BUCKET_NAME: $[[ inputs.base-bucket-name ]]

  S3_ENVIRONMENT_NAMESPACE: $[[ inputs.environment-namespace ]]

  S3_SCRIPTS_DIR: $[[ inputs.scripts-dir ]]

  S3_BASE_TEMPLATE_NAME: s3

  S3_REVIEW_AUTOSTOP_DURATION: $[[ inputs.review-autostop-duration ]]
@@ -487,6 +494,8 @@ stages: 
  # upload/sync files to bucket

  function s3_deploy() {

    export environment_type=$ENV_TYPE

    environment_namespace=$(echo "$S3_ENVIRONMENT_NAMESPACE" | tr -d '[:punct:]' | tr '[:upper:]' '[:lower:]')

    export environment_namespace

    bucket=${ENV_BUCKET_NAME:-${S3_BASE_BUCKET_NAME}${ENV_APP_SUFFIX}}

    prefix=${ENV_PREFIX:-${S3_PREFIX}}

    region=${ENV_REGION:-${S3_REGION}}
@@ -505,7 +514,7 @@ stages: 
    log_info "--- \$prefix: \\e[33;1m${prefix:-(none)}\\e[0m"



    # unset any upstream deployment env & artifacts

    rm -f s3.env

    rm -f s3.env*

    rm -f environment_url.txt



    # maybe create bucket
@@ -553,8 +562,15 @@ stages: 
      environment_url="$website_url$prefix"

      # finally persist environment url

      echo "$environment_url" > environment_url.txt

      echo -e "environment_type=$environment_type\\nenvironment_name=$bucket\\nenvironment_url=$environment_url" > s3.env

      chmod 644 environment_url.txt s3.env

      # var prefix ('_' if namespace)

      prefix="${environment_namespace:+${environment_namespace}_}"

      dotenvfile="s3.env${environment_namespace:+.${environment_namespace}}"

      {

        echo "${prefix}environment_type=${environment_type}"

        echo "${prefix}environment_name=${bucket}"

        echo "${prefix}environment_url=${environment_url}"

      } >> "$dotenvfile"

      chmod 644 environment_url.txt "$dotenvfile"

    fi

  }
@@ -647,9 +663,8 @@ stages: 
  before_script:

    - !reference [.s3-scripts]

    - install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}"

    - s3_login



# Deploy job prototype

# Env management job prototype

# Can be extended to define a concrete environment

#

# @arg ENV_TYPE             : environment type
@@ -657,11 +672,23 @@ stages: 
# @arg ENV_REGION           : env-specific region

# @arg ENV_PREFIX           : env-specific S3 prefix

# @arg ENV_APP_SUFFIX       : env-specific application suffix

.s3-deploy:

.s3-env-base:

  extends: .s3-base

  stage: deploy

  variables:

    ENV_APP_SUFFIX: "-$CI_ENVIRONMENT_SLUG"

  before_script:

    - !reference [.s3-base, before_script]

    - s3_login

  environment:

    name: ${ENV_TYPE}${S3_ENVIRONMENT_NAMESPACE}

  resource_group: ${ENV_TYPE}${S3_ENVIRONMENT_NAMESPACE}



# Deploy job prototype

# Can be extended for each deployable environment

# @arg ENV_URL : env-specific application url

.s3-deploy:

  extends: .s3-env-base

  script:

    - s3_deploy

  artifacts:
@@ -669,26 +696,19 @@ stages: 
    paths:

      - environment_url.txt

    reports:

      dotenv: s3.env

      dotenv: s3.env*

  environment:

    url: "$environment_url" # can be either static or dynamic



# Cleanup job prototype

# Can be extended for each deletable environment

#

# @arg ENV_TYPE         : environment type

# @arg ENV_BUCKET_NAME  : env-specific S3 bucket name

# @arg ENV_PREFIX       : env-specific S3 prefix

# @arg ENV_APP_SUFFIX   : env-specific application suffix

.s3-cleanup:

  extends: .s3-base

  stage: deploy

  extends: .s3-env-base

  # force no dependencies

  dependencies: []

  # no need to clone repository

  variables:

    GIT_STRATEGY: none

    ENV_APP_SUFFIX: "-$CI_ENVIRONMENT_SLUG"

  script:

    - s3_delete

  environment:
@@ -706,10 +726,12 @@ s3-review: 
    ENV_ACCESS_KEY: "$S3_REVIEW_ACCESS_KEY"

    ENV_SECRET_KEY: "$S3_REVIEW_SECRET_KEY"

  environment:

    name: review/$CI_COMMIT_REF_NAME

    name: ${ENV_TYPE}${S3_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}

    # ⚠ on_stop must be unset when defining parallel:matrix environments

    # see: https://gitlab.com/gitlab-org/gitlab/-/issues/332247

    on_stop: s3-cleanup-review

    auto_stop_in: "$S3_REVIEW_AUTOSTOP_DURATION"

  resource_group: review/$CI_COMMIT_REF_NAME

  resource_group: ${ENV_TYPE}${S3_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}

  rules:

    # exclude tags

    - if: $CI_COMMIT_TAG
@@ -734,9 +756,9 @@ s3-cleanup-review: 
    ENV_ACCESS_KEY: "$S3_REVIEW_ACCESS_KEY"

    ENV_SECRET_KEY: "$S3_REVIEW_SECRET_KEY"

  environment:

    name: review/$CI_COMMIT_REF_NAME

    name: ${ENV_TYPE}${S3_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}

    action: stop

  resource_group: review/$CI_COMMIT_REF_NAME

  resource_group: ${ENV_TYPE}${S3_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}

  rules:

    # exclude tags

    - if: $CI_COMMIT_TAG
@@ -794,9 +816,6 @@ s3-integration: 
    ENV_ENDPOINT_HOST: "$S3_INTEG_ENDPOINT_HOST"

    ENV_ACCESS_KEY: "$S3_INTEG_ACCESS_KEY"

    ENV_SECRET_KEY: "$S3_INTEG_SECRET_KEY"

  environment:

    name: integration

  resource_group: integration

  rules:

    # exclude if $S3_INTEG_DISABLED set

    - if: '$S3_INTEG_DISABLED == "true"'
@@ -816,9 +835,6 @@ s3-staging: 
    ENV_ENDPOINT_HOST: "$S3_STAGING_ENDPOINT_HOST"

    ENV_ACCESS_KEY: "$S3_STAGING_ACCESS_KEY"

    ENV_SECRET_KEY: "$S3_STAGING_SECRET_KEY"

  environment:

    name: staging

  resource_group: staging

  rules:

    # exclude if $S3_STAGING_DISABLED set

    - if: '$S3_STAGING_DISABLED == "true"'
@@ -839,9 +855,6 @@ s3-production: 
    ENV_ENDPOINT_HOST: "$S3_PROD_ENDPOINT_HOST"

    ENV_ACCESS_KEY: "$S3_PROD_ACCESS_KEY"

    ENV_SECRET_KEY: "$S3_PROD_SECRET_KEY"

  environment:

    name: production

  resource_group: production

  rules:

    # exclude non-production branches

    - if: '$CI_COMMIT_REF_NAME !~ $PROD_REF'