Commit 3569dcf4 authored by Pierre Smeyers's avatar Pierre Smeyers
Browse files

Merge branch 'feat/support-parallel-matrix' into 'main' 

Add parallel:matrix support

See merge request to-be-continuous/docker-compose!25
parents dc0aedcd 94b081c6

README.md

+49 −0
Original line number Diff line number Diff line
@@ -283,6 +283,45 @@ The **static way** can be implemented simply by setting the appropriate configur 
To implement the **dynamic way**, your post deployment hook script shall simply generate a `environment_url.txt` file in the working directory, containing only

the dynamically generated url. When detected by the template, it will use it as the newly deployed environment url.



### Multiple environments support



The Docker Compose template allows deploying multiple environments in parallel. Use cases of this include:



- monorepo, where a single Git repository might host several separate deployable components or apps,

- multi-instances deployment of the same application.



This feature can be enabled using the [parallel matrix jobs](https://docs.gitlab.com/ee/ci/yaml/#parallelmatrix)

pattern at the `.aws-env-base` job level (this is the top parent job of all deployment jobs). 

Environments namespacing is ensured by the `DCMP_ENVIRONMENT_NAMESPACE` variable (must start with a slash `/`).



Here is the example of the `.gitlab-ci.yml` file for a project deploying both a frontend and a backend applications:



```yaml

.compose-env-base:

  parallel:

    matrix:

      - DCMP_ENVIRONMENT_NAMESPACE: "/front"

        # Compose deployment scripts are located in the ./front/ directory

        DCMP_SCRIPTS_DIR: "front"

        DCMP_BASE_APP_NAME: "wonderfront"

      - DCMP_ENVIRONMENT_NAMESPACE: "/back"

        # Compose deployment scripts are located in the ./back/ directory

        DCMP_SCRIPTS_DIR: "back"

        DCMP_BASE_APP_NAME: "wonderback"



# ⚠ on_stop must be unset when defining parallel:matrix environments

# see: https://gitlab.com/gitlab-org/gitlab/-/issues/332247

compose-review:

  environment:

    on_stop: null

```



The above configuration will deploy 2 environments on each pipeline:



- on feature branches: `review/front/$CI_COMMIT_REF_NAME` and `review/back/$CI_COMMIT_REF_NAME`

- on the integration branch: `integration/front` and `integration/back` 

- on the production branch: `staging/front` and `staging/back` (and finally `production/front` and `production/back`)



### Deployment output variables



Each deployment job produces _output variables_ that are propagated to downstream jobs (using [dotenv artifacts](https://docs.gitlab.com/ci/yaml/artifacts_reports/#artifactsreportsdotenv)):
@@ -294,6 +333,15 @@ Each deployment job produces _output variables_ that are propagated to downstrea 
Those variables may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).



You may also add and propagate your own custom variables, by pushing them to the `docker-compose.out.env` file in your [deployment scripts or hooks](#deployment-and-cleanup).

> [!important]

> If [multiple environments](#multiple-environments-support) are configured, the output variables are prefixed with a 

> sluggified value of the `DCMP_ENVIRONMENT_NAMESPACE` variable (stripped of punctuation characters and converted to lowercase):

> 

> * `<namespace_slug>_environment_type`: set to the type of environment (`review`, `integration`, `staging` or `production`),

> * `<namespace_slug>_environment_name`: the application name (see below),

> * `<namespace_slug>_environment_url`: set to the environment URL (whether determined statically or dynamically).

> 

> The output dotenv file will be `docker-compose.out.env.<namespace_slug>` instead, and the dynamic variable `${environment_namespace}` can be used in your scripts and manifests to access the contextual value of `<namespace_slug>`.



## Configuration reference
@@ -321,6 +369,7 @@ The Docker Compose template uses some global configuration used throughout all j 
| `cmd` / `DCMP_CMD`     | The docker compose or stack command (`docker compose`, `docker-compose` or `docker stack`)                                                                                          | _none_ (auto) |

| `base-app-name` / `DCMP_BASE_APP_NAME`| Base application name                                                                                                                                                               | `$CI_PROJECT_NAME` ([see GitLab doc](https://docs.gitlab.com/ci/variables/predefined_variables/)) |

| `environment-url` / `DCMP_ENVIRONMENT_URL`| Default environments url _(only define for static environment URLs declaration)_<br/>_supports late variable expansion (ex: `https://%{environment_name}.docker-compose.acme.com`)_ | _none_ |

| `environment-namespace` / `DCMP_ENVIRONMENT_NAMESPACE` | Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying [multiple environments](#multiple-environments-support))_<br/>:warning: must start with a slash `/` | _none_ |

| `scripts-dir` / `DCMP_SCRIPTS_DIR`| Directory where Compose files, dotenv files and hook scripts are located                                                                                                            | `.` _(root project dir)_ |

| `up-opts` / `DCMP_UP_OPTS` | [`compose up` options](https://docs.docker.com/reference/cli/docker/compose/up/#options) (only when using Docker Compose)                                                           | `--no-build --remove-orphans --wait --wait-timeout 180` |

| `down-opts`/ `DCMP_DOWN_OPTS` | [`compose down` options](https://docs.docker.com/reference/cli/docker/compose/down/#options) (only when using Docker Compose)                                                       | `--volumes --remove-orphans --rmi all` |

kicker.json

+5 −0
Original line number Diff line number Diff line
@@ -29,6 +29,11 @@ 
      "type": "url",

      "description": "The default environments url _(only define for static environment URLs declaration)_\n\n_supports late variable expansion (ex: `https://%{environment_name}.dcmp.acme.com`)_"

    },

    {

      "name": "DCMP_ENVIRONMENT_NAMESPACE",

      "description": "Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying multiple environments)_\n\n:warning: must start with a slash `/`",

      "advanced": true

    },

    {

      "name": "DCMP_SCRIPTS_DIR",

      "description": "Directory where Compose files, dotenv files and hook scripts are located",

templates/gitlab-ci-docker-compose.yml

+47 −44
Original line number Diff line number Diff line
@@ -35,6 +35,12 @@ spec: 


        _supports late variable expansion (ex: `https://%{environment_name}.dcmp.acme.com`)_

      default: ''

    environment-namespace:

      description: |-

        Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying multiple environments)_

        

        :warning: must start with a slash `/`

      default: ''

    scripts-dir:

      description: Directory where Compose files, dotenv files and hook scripts are located

      default: .
@@ -189,6 +195,7 @@ variables: 


  DCMP_BASE_APP_NAME: $[[ inputs.base-app-name ]]

  DCMP_ENVIRONMENT_URL: $[[ inputs.environment-url ]]

  DCMP_ENVIRONMENT_NAMESPACE: $[[ inputs.environment-namespace ]]

  DCMP_SCRIPTS_DIR: $[[ inputs.scripts-dir ]]

  DCMP_SSH_KNOWN_HOSTS: $[[ inputs.ssh-known-hosts ]]

  DCMP_CONFIG_OPTS: $[[ inputs.config-opts ]]
@@ -769,6 +776,9 @@ stages: 


  # application deployment function

  function compose_up() {

    # compute environment namespace (strip punctuation, lowercase)

    environment_namespace=$(echo "$DCMP_ENVIRONMENT_NAMESPACE" | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]')

    export environment_namespace

    environment_url=${ENV_URL:-$DCMP_ENVIRONMENT_URL}

    # variables expansion in $environment_url

    environment_url=$(echo "$environment_url" | TBC_ENVSUBST_ENCODING=uricomp tbc_envsubst)
@@ -784,7 +794,7 @@ stages: 
    log_info "--- \$hostname: \\e[33;1m${hostname}\\e[0m"



    # unset any upstream deployment env & artifacts

    rm -f docker-compose.out.env

    rm -f docker-compose.out.env*

    rm -f environment_url.txt



    # maybe execute pre compose-up script
@@ -831,8 +841,15 @@ stages: 
    else

      echo "$environment_url" > environment_url.txt

    fi

    echo -e "environment_type=$environment_type\\nenvironment_name=$environment_name\\nenvironment_url=$environment_url" >> docker-compose.out.env

    chmod 644 environment_url.txt docker-compose.out.env

    # var prefix ('_' if namespace)

    prefix="${environment_namespace:+${environment_namespace}_}"

    dotenvfile="docker-compose.out.env${environment_namespace:+.${environment_namespace}}"

    {

      echo "${prefix}environment_type=${environment_type}"

      echo "${prefix}environment_name=${environment_name}"

      echo "${prefix}environment_url=${environment_url}"

    } >> "$dotenvfile"

    chmod 644 environment_url.txt "$dotenvfile"

  }



  # environment cleanup function
@@ -874,13 +891,8 @@ stages: 


  # ENDSCRIPT



# job prototype

# Base job prototype

# defines default Docker image, services, cache policy and init scripts

# Required vars for login:

# @var ENV_TYPE      : environment type

# @var ENV_APP_NAME  : env-specific application name

# @var ENV_APP_SUFFIX: env-specific application suffix

# @var DOCKER_HOST   : env-specific DOCKER_HOST

.compose-base:

  image: $DCMP_IMAGE

  services:
@@ -889,24 +901,31 @@ stages: 
  before_script:

    - !reference [.compose-scripts]

    - install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}"

    - compose_init



# base job for online compose operations

.compose-online:

# Env management job prototype

# Can be extended to define a concrete environment

# @var ENV_TYPE      : environment type

# @var ENV_APP_NAME  : env-specific application name

# @var ENV_APP_SUFFIX: env-specific application suffix

.compose-env-base:

  extends: .compose-base

  stage: deploy

  variables:

    ENV_APP_SUFFIX: "-$CI_ENVIRONMENT_SLUG"

  before_script:

    - !reference [.compose-base, before_script]

    - compose_init

    - configure_network

    - configure_registries_auth

  environment:

    name: "${ENV_TYPE}${DCMP_ENVIRONMENT_NAMESPACE}"

  resource_group: "${ENV_TYPE}${DCMP_ENVIRONMENT_NAMESPACE}"



# Deploy job prototype

# Can be extended to define a concrete environment

# @var ENV_URL              : env-specific application url

.compose-deploy:

  extends: .compose-online

  stage: deploy

  variables:

    ENV_APP_SUFFIX: "-$CI_ENVIRONMENT_SLUG"

  extends: .compose-env-base

  script:

    - compose_up

  artifacts:
@@ -916,33 +935,28 @@ stages: 
      - environment_url.txt

    reports:

      # propagate deployed env info in a dotenv artifact

      dotenv: docker-compose.out.env

      dotenv: docker-compose.out.env*

  environment:

    url: "$environment_url" # can be either static or dynamic



# Cleanup job prototype

# Can be extended for each deletable environment

.compose-cleanup:

  extends: .compose-online

  stage: deploy

  extends: .compose-env-base

  # force no dependencies

  dependencies: []

  variables:

    ENV_APP_SUFFIX: "-$CI_ENVIRONMENT_SLUG"

  script:

    - compose_down

  environment:

    action: stop



# compose config job prototype

# Can be extended for each concrete environment

#

# @var ENV_TYPE      : environment type

# @var ENV_APP_NAME  : env-specific application name

# @var ENV_APP_SUFFIX: env-specific application suffix

.compose-config:

  extends: .compose-base

  stage: package-test

  before_script:

    - !reference [.compose-base, before_script]

    - compose_init

  script:

    - |

      if [[ "$DCMP_CMD" == "docker stack" ]]; then
@@ -1065,10 +1079,12 @@ compose-review: 
    ENV_APP_NAME: "$DCMP_REVIEW_APP_NAME"

    ENV_URL: "$DCMP_REVIEW_ENVIRONMENT_URL"

  environment:

    name: review/$CI_COMMIT_REF_NAME

    name: ${ENV_TYPE}${DCMP_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}

    # ⚠ on_stop must be unset when defining parallel:matrix environments

    # see: https://gitlab.com/gitlab-org/gitlab/-/issues/332247

    on_stop: compose-cleanup-review

    auto_stop_in: "$DCMP_REVIEW_AUTOSTOP_DURATION"

  resource_group: review/$CI_COMMIT_REF_NAME

  resource_group: ${ENV_TYPE}${DCMP_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}

  rules:

    # exclude tags

    - if: $CI_COMMIT_TAG
@@ -1088,10 +1104,9 @@ compose-cleanup-review: 
    ENV_TYPE: review

    ENV_APP_NAME: "$DCMP_REVIEW_APP_NAME"

  environment:

    name: review/$CI_COMMIT_REF_NAME

    name: ${ENV_TYPE}${DCMP_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}

    action: stop

  # TODO: use resource group

  resource_group: review/$CI_COMMIT_REF_NAME

  resource_group: ${ENV_TYPE}${DCMP_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}

  rules:

    # exclude tags

    - if: $CI_COMMIT_TAG
@@ -1113,10 +1128,6 @@ compose-integration: 
    ENV_TYPE: integration

    ENV_APP_NAME: "$DCMP_INTEG_APP_NAME"

    ENV_URL: "$DCMP_INTEG_ENVIRONMENT_URL"

  environment:

    name: integration

  # TODO: use resource group

  resource_group: integration

  rules:

    # exclude if $DCMP_INTEG_DOCKER_HOST not set

    - if: '$DCMP_INTEG_DOCKER_HOST == null || $DCMP_INTEG_DOCKER_HOST == ""'
@@ -1133,10 +1144,6 @@ compose-staging: 
    ENV_TYPE: staging

    ENV_APP_NAME: "$DCMP_STAGING_APP_NAME"

    ENV_URL: "$DCMP_STAGING_ENVIRONMENT_URL"

  environment:

    name: staging

  # TODO: use resource group

  resource_group: staging

  rules:

    # exclude if $DCMP_STAGING_DOCKER_HOST not set

    - if: '$DCMP_STAGING_DOCKER_HOST == null || $DCMP_STAGING_DOCKER_HOST == ""'
@@ -1155,10 +1162,6 @@ compose-production: 
    ENV_APP_SUFFIX: "" # no suffix for prod

    ENV_APP_NAME: "$DCMP_PROD_APP_NAME"

    ENV_URL: "$DCMP_PROD_ENVIRONMENT_URL"

  environment:

    name: production

  # TODO: use resource group

  resource_group: production

  rules:

    # exclude if $DCMP_PROD_DOCKER_HOST not set

    - if: '$DCMP_PROD_DOCKER_HOST == null || $DCMP_PROD_DOCKER_HOST == ""'