Commit a3b40c8b authored by Pierre Smeyers's avatar Pierre Smeyers
Browse files

feat: add parallel:matrix support

parent 1d2e03c3
Loading
Loading
Loading
Loading
+49 −1
Original line number Diff line number Diff line
@@ -243,7 +243,7 @@ that you might use in your hook scripts, deployment manifests, and other deploym
#### Generated environment name

The `${environment_name}` variable is generated to designate each deployment environment with a unique and meaningful application name.
By construction, it is suitable for inclusion in DNS, URLs, Kubernetes labels...
By construction, it is suitable for inclusion in DNS, URLs, {{cookiecutter.template_name}} labels...
It is built from:

* the application _base name_ (defaults to `$CI_PROJECT_NAME` but can be overridden globally and/or per deployment environment - _see configuration variables_)
@@ -331,6 +331,43 @@ The **static way** can be implemented simply by setting the appropriate configur
To implement the **dynamic way**, your deployment script shall simply generate a `environment_url.txt` file in the working directory, containing only
the dynamically generated url. When detected by the template, it will use it as the newly deployed environment url.

### Multiple environments support

The {{cookiecutter.template_name}} template allows deploying multiple environments in parallel. Use cases of this include:

- monorepo, where a single Git repository might host several separate deployable components or apps,
- multi-instances deployment of the same application.

This feature can be enabled using the [parallel matrix jobs](https://docs.gitlab.com/ee/ci/yaml/#parallelmatrix)
pattern at the `.{{cookiecutter.template_prefix}}-base` job level (the template's base job).
Environments namespacing is ensured by the `{{cookiecutter.template_PREFIX}}_ENVIRONMENT_NAMESPACE` variable (must start with a slash `/`).

Here is the example of the `.gitlab-ci.yml` file for a project deploying both a frontend and a backend applications:

```yaml
.{{cookiecutter.template_prefix}}-base:
  parallel:
    matrix:
      - {{cookiecutter.template_PREFIX}}_ENVIRONMENT_NAMESPACE: "/front"
        # {{cookiecutter.template_name}} deployment scripts are located in the ./front/ directory
        {{cookiecutter.template_PREFIX}}_SCRIPTS_DIR: "front"
      - {{cookiecutter.template_PREFIX}}_ENVIRONMENT_NAMESPACE: "/back"
        # {{cookiecutter.template_name}} deployment scripts are located in the ./back/ directory
        {{cookiecutter.template_PREFIX}}_SCRIPTS_DIR: "back"

# ⚠ on_stop must be unset when defining parallel:matrix environments
# see: https://gitlab.com/gitlab-org/gitlab/-/issues/332247
{{cookiecutter.template_prefix}}-review:
  environment:
    on_stop: null
```

The above configuration will deploy 2 environments on each pipeline:

- on feature branches: `review/front/$CI_COMMIT_REF_NAME` and `review/back/$CI_COMMIT_REF_NAME`
- on the integration branch: `integration/front` and `integration/back`
- on the production branch: `staging/front` and `staging/back` (and finally `production/front` and `production/back`)

### Deployment output variables

Each deployment job produces _output variables_ that are propagated to downstream jobs (using [dotenv artifacts](https://docs.gitlab.com/ci/yaml/artifacts_reports/#artifactsreportsdotenv)):
@@ -343,6 +380,16 @@ Those variables may be freely used in downstream jobs (for instance to run accep

You may also add and propagate your own custom variables, by pushing them to the `{{ cookiecutter.project_slug }}.env` file in your [deployment scripts or hooks](#deployment-and-cleanup).

> [!important]
> If [multiple environments](#multiple-environments-support) are configured, the output variables are prefixed with a 
> sluggified value of the `{{cookiecutter.template_PREFIX}}_ENVIRONMENT_NAMESPACE` variable (stripped of punctuation characters and converted to lowercase):
> 
> * `<namespace_slug>_environment_type`: set to the type of environment (`review`, `integration`, `staging` or `production`),
> * `<namespace_slug>_environment_name`: the application name (see below),
> * `<namespace_slug>_environment_url`: set to the environment URL (whether determined statically or dynamically).
> 
> The output dotenv file will be `{{cookiecutter.project_slug}}.env.<namespace_slug>` instead, and the dynamic variable `${environment_namespace}` can be used in your scripts and manifests to access the contextual value of `<namespace_slug>`.

## Configuration reference

### Secrets management
@@ -370,6 +417,7 @@ The {{cookiecutter.template_name}} template uses some global configuration used
| `api-url` / `{{cookiecutter.template_PREFIX}}_API_URL`            | Default {{cookiecutter.template_name}} API url | _none_ |
| :lock: `{{cookiecutter.template_PREFIX}}_API_TOKEN`          | Default {{cookiecutter.template_name}} API token | _none_ |
| `environment-url` / `{{cookiecutter.template_PREFIX}}_ENVIRONMENT_URL`    | Default environments url _(only define for static environment URLs declaration)_<br/>_supports late variable expansion (e.g.: `https://%{environment_name}.{{cookiecutter.project_slug}}.acme.com`)_ | _none_ |
| `environment-namespace` / `{{cookiecutter.template_PREFIX}}_ENVIRONMENT_NAMESPACE` | Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying [multiple environments](#multiple-environments-support))_<br/>:warning: must start with a slash `/` | _none_ |
| `scripts-dir` / `{{cookiecutter.template_PREFIX}}_SCRIPTS_DIR`        | Directory where deploy & cleanup scripts are located | `.` _(root project dir)_ |

### Review environments configuration
+5 −0
Original line number Diff line number Diff line
@@ -118,6 +118,11 @@
      "type": "url",
      "description": "The default environments url _(only define for static environment URLs declaration)_\n\n_supports late variable expansion (ex: `https://%{environment_name}.{{cookiecutter.template_prefix}}.acme.com`)_"
    },
    {
      "name": "{{cookiecutter.template_PREFIX}}_ENVIRONMENT_NAMESPACE",
      "description": "Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying multiple environments)_\n\n:warning: must start with a slash `/`",
      "advanced": true
    },
    {
      "name": "{{cookiecutter.template_PREFIX}}_SCRIPTS_DIR",
      "description": "Directory where deploy & cleanup scripts are located",
+48 −33
Original line number Diff line number Diff line
@@ -75,6 +75,12 @@ spec:

        _supports late variable expansion (ex: `https://%{environment_name}.{{cookiecutter.template_prefix}}.acme.com`)_
      default: ''
    environment-namespace:
      description: |-
        Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying multiple environments)_
        
        :warning: must start with a slash `/`
      default: ''
    scripts-dir:
      description: Directory where deploy & cleanup scripts are located
      default: .
@@ -242,6 +248,7 @@ variables:
  {{ cookiecutter.template_PREFIX }}_BASE_APP_NAME: $[[ inputs.base-app-name ]]
  {{ cookiecutter.template_PREFIX }}_API_URL: $[[ inputs.api-url ]]
  {{ cookiecutter.template_PREFIX }}_ENVIRONMENT_URL: $[[ inputs.environment-url ]]
  {{ cookiecutter.template_PREFIX }}_ENVIRONMENT_NAMESPACE: $[[ inputs.environment-namespace ]]
  {{ cookiecutter.template_PREFIX }}_SCRIPTS_DIR: $[[ inputs.scripts-dir ]]

  {{ cookiecutter.template_PREFIX }}_REVIEW_PROJECT: $[[ inputs.review-project ]]
@@ -637,6 +644,8 @@ stages:
    # variables expansion in $environment_url
    environment_url=$(echo "$environment_url" | TBC_ENVSUBST_ENCODING=uricomp tbc_envsubst)
    export environment_url
    environment_namespace=$(echo "${{ cookiecutter.template_PREFIX }}_ENVIRONMENT_NAMESPACE" | tr -d '[:punct:]' | tr '[:upper:]' '[:lower:]')
    export environment_namespace
    # extract hostname from $environment_url
    hostname=$(echo "$environment_url" | awk -F[/:] '{print $4}')
    export hostname
@@ -648,7 +657,7 @@ stages:
    log_info "--- \$hostname: \\e[33;1m${hostname}\\e[0m"

    # unset any upstream deployment env & artifacts
    rm -f {{ cookiecutter.project_slug }}.env
    rm -f {{ cookiecutter.project_slug }}.env*
    rm -f environment_url.txt

    # TODO: implement the deployment here
@@ -662,8 +671,17 @@ stages:
    else
      echo "$environment_url" > environment_url.txt
    fi
    echo -e "environment_type=$environment_type\\nenvironment_name=$environment_name\\nenvironment_url=$environment_url" >> {{ cookiecutter.project_slug }}.env
    chmod 644 environment_url.txt {{ cookiecutter.project_slug }}.env
    # var prefix ('_' if namespace)
    prefix="${environment_namespace:+${environment_namespace}_}"
    dotenvfile="{{ cookiecutter.project_slug }}.env${environment_namespace:+.${environment_namespace}}"
    {
      echo "${prefix}environment_type=${environment_type}"
      echo "${prefix}environment_name=${environment_name}"
      echo "${prefix}environment_url=${environment_url}"
      # '$environment_url' is required by GitLab (dynamic env URL)
      if [[ "$environment_namespace" ]]; then echo "environment_url=${environment_url}"; fi
    } >> "$dotenvfile"
    chmod 644 environment_url.txt "$dotenvfile"
  }

  # environment cleanup function
@@ -832,20 +850,31 @@ stages:
  before_script:
    - !reference [.{{ cookiecutter.template_prefix }}-scripts]
    - install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}"
    - {{ cookiecutter.template_prefix }}_login

# Deploy job prototype
# Env management job prototype
# Can be extended to define a concrete environment
#
# @var ENV_TYPE      : environment type
# @var ENV_APP_NAME  : env-specific application name
# @var ENV_APP_SUFFIX: env-specific application suffix
# @var ENV_URL       : env-specific application url
.{{ cookiecutter.template_prefix }}-deploy:
# @arg ENV_TYPE      : environment type
# @arg ENV_APP_NAME  : env-specific application name
# @arg ENV_APP_SUFFIX: env-specific application suffix
.{{ cookiecutter.template_prefix }}-env-base:
  extends: .{{ cookiecutter.template_prefix }}-base
  stage: deploy
  variables:
    ENV_APP_SUFFIX: "-$CI_ENVIRONMENT_SLUG"
  before_script:
    - !reference [.{{ cookiecutter.template_prefix }}-base, before_script]
    - {{ cookiecutter.template_prefix }}_login
  environment:
    name: ${ENV_TYPE}{{ '${' }}{{ cookiecutter.template_PREFIX }}_ENVIRONMENT_NAMESPACE}
  resource_group: ${ENV_TYPE}{{ '${' }}{{ cookiecutter.template_PREFIX }}_ENVIRONMENT_NAMESPACE}

# Deploy job prototype
# Can be extended to define a concrete environment
#
# @var ENV_URL       : env-specific application url
.{{ cookiecutter.template_prefix }}-deploy:
  extends: .{{ cookiecutter.template_prefix }}-env-base
  script:
    - {{ cookiecutter.template_prefix }}_deploy
  artifacts:
@@ -855,8 +884,9 @@ stages:
      - environment_url.txt
    reports:
      # TODO: propagate deployed env info in a dotenv artifact
      dotenv: {{ cookiecutter.project_slug }}.env
      dotenv: {{ cookiecutter.project_slug }}.env*
  environment:
    action: start
    url: "$environment_url" # can be either static or dynamic

# Cleanup job prototype
@@ -866,12 +896,9 @@ stages:
# @var ENV_APP_NAME  : env-specific application name
# @var ENV_APP_SUFFIX: env-specific application suffix
.{{ cookiecutter.template_prefix }}-cleanup:
  extends: .{{ cookiecutter.template_prefix }}-base
  stage: deploy
  extends: .{{ cookiecutter.template_prefix }}-env-base
  # force no dependencies
  dependencies: []
  variables:
    ENV_APP_SUFFIX: "-$CI_ENVIRONMENT_SLUG"
  script:
    - {{ cookiecutter.template_prefix }}_delete
  environment:
@@ -889,10 +916,12 @@ stages:
    ENV_API_TOKEN: "${{ cookiecutter.template_PREFIX }}_REVIEW_API_TOKEN"
    ENV_PROJECT: "${{ cookiecutter.template_PREFIX }}_REVIEW_PROJECT"
  environment:
    name: review/$CI_COMMIT_REF_NAME
    name: ${ENV_TYPE}{{ '${' }}{{ cookiecutter.template_PREFIX }}_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}
    # ⚠ on_stop must be unset when defining parallel:matrix environments
    # see: https://gitlab.com/gitlab-org/gitlab/-/issues/332247
    on_stop: {{ cookiecutter.template_prefix }}-cleanup-review
    auto_stop_in: "${{ cookiecutter.template_PREFIX }}_REVIEW_AUTOSTOP_DURATION"
  resource_group: review/$CI_COMMIT_REF_NAME
  resource_group: ${ENV_TYPE}{{ '${' }}{{ cookiecutter.template_PREFIX }}_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}
  rules:
    # exclude tags
    - if: $CI_COMMIT_TAG
@@ -916,10 +945,8 @@ stages:
    ENV_API_TOKEN: "${{ cookiecutter.template_PREFIX }}_REVIEW_API_TOKEN"
    ENV_PROJECT: "${{ cookiecutter.template_PREFIX }}_REVIEW_PROJECT"
  environment:
    name: review/$CI_COMMIT_REF_NAME
    action: stop
  # TODO: use resource group
  resource_group: review/$CI_COMMIT_REF_NAME
    name: ${ENV_TYPE}{{ '${' }}{{ cookiecutter.template_PREFIX }}_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}
  resource_group: ${ENV_TYPE}{{ '${' }}{{ cookiecutter.template_PREFIX }}_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}
  rules:
    # exclude tags
    - if: $CI_COMMIT_TAG
@@ -942,10 +969,6 @@ stages:
    ENV_API_URL: "${{ cookiecutter.template_PREFIX }}_INTEG_API_URL"
    ENV_API_TOKEN: "${{ cookiecutter.template_PREFIX }}_INTEG_API_TOKEN"
    ENV_PROJECT: "${{ cookiecutter.template_PREFIX }}_INTEG_PROJECT"
  environment:
    name: integration
  # TODO: use resource group
  resource_group: integration
  rules:
    # exclude if ${{ cookiecutter.template_PREFIX }}_INTEG_PROJECT not set
    - if: '${{ cookiecutter.template_PREFIX }}_INTEG_PROJECT == null || ${{ cookiecutter.template_PREFIX }}_INTEG_PROJECT == ""'
@@ -963,10 +986,6 @@ stages:
    ENV_API_URL: "${{ cookiecutter.template_PREFIX }}_STAGING_API_URL"
    ENV_API_TOKEN: "${{ cookiecutter.template_PREFIX }}_STAGING_API_TOKEN"
    ENV_PROJECT: "${{ cookiecutter.template_PREFIX }}_STAGING_PROJECT"
  environment:
    name: staging
  # TODO: use resource group
  resource_group: staging
  rules:
    # exclude if ${{ cookiecutter.template_PREFIX }}_STAGING_PROJECT not set
    - if: '${{ cookiecutter.template_PREFIX }}_STAGING_PROJECT == null || ${{ cookiecutter.template_PREFIX }}_STAGING_PROJECT == ""'
@@ -986,10 +1005,6 @@ stages:
    ENV_API_URL: "${{ cookiecutter.template_PREFIX }}_PROD_API_URL"
    ENV_API_TOKEN: "${{ cookiecutter.template_PREFIX }}_PROD_API_TOKEN"
    ENV_PROJECT: "${{ cookiecutter.template_PREFIX }}_PROD_PROJECT"
  environment:
    name: production
  # TODO: use resource group
  resource_group: production
  rules:
    # exclude if ${{ cookiecutter.template_PREFIX }}_PROD_PROJECT not set
    - if: '${{ cookiecutter.template_PREFIX }}_PROD_PROJECT == null || ${{ cookiecutter.template_PREFIX }}_PROD_PROJECT == ""'