Commit 0c75a6b2 authored by Pierre Smeyers's avatar Pierre Smeyers
Browse files

feat: add parallel:matrix support

parent 013b44ff

README.md

+48 −0
Original line number Diff line number Diff line
@@ -330,6 +330,43 @@ cf cups my-ups -p '{"user":"service-user","password":"service-password","url":"h 


Then at service binding, the application gets the three credential parameters as specified in the descriptor file, that are `user`, `password` and `url`.



### Multiple environments support



The Cloud Foundry template allows deploying multiple environments in parallel. Use cases of this include:



- monorepo, where a single Git repository might host several separate deployable components or apps,

- multi-instances deployment of the same application.



This feature can be enabled using the [parallel matrix jobs](https://docs.gitlab.com/ee/ci/yaml/#parallelmatrix)

pattern at the `.cf-env-base` job level (this is the top parent job of all deployment jobs).

Environments namespacing is ensured by the `CF_ENVIRONMENT_NAMESPACE` variable (must start with a slash `/`).



Here is the example of the `.gitlab-ci.yml` file for a project deploying both a frontend and a backend applications:



```yaml

.cf-env-base:

  parallel:

    matrix:

      - CF_ENVIRONMENT_NAMESPACE: "/front"

        # Cloud Foundry scripts are located in the ./front/ directory

        CF_SCRIPTS_DIR: "front"

      - CF_ENVIRONMENT_NAMESPACE: "/back"

        # Cloud Foundry scripts are located in the ./back/ directory

        CF_SCRIPTS_DIR: "back"



# ⚠ on_stop must be unset when defining parallel:matrix environments

# see: https://gitlab.com/gitlab-org/gitlab/-/issues/332247

cf-review:

  environment:

    on_stop: null

```



The above configuration will deploy 2 environments on each pipeline:



- on feature branches: `review/front/$CI_COMMIT_REF_NAME` and `review/back/$CI_COMMIT_REF_NAME`

- on the integration branch: `integration/front` and `integration/back`

- on the production branch: `staging/front` and `staging/back` (and finally `production/front` and `production/back`)



### Deployment output variables



Each deployment job produces _output variables_ that are propagated to downstream jobs (using [dotenv artifacts](https://docs.gitlab.com/ci/yaml/artifacts_reports/#artifactsreportsdotenv)):
@@ -342,6 +379,16 @@ Those variables may be freely used in downstream jobs (for instance to run accep 


You may also add and propagate your own custom variables, by pushing them to the `cloudfoundry.env` file in your [deployment scripts](#hook-scripts).



> [!important]

> If [multiple environments](#multiple-environments-support) are configured, the output variables are prefixed with a

> sluggified value of the `CF_ENVIRONMENT_NAMESPACE` variable (stripped of punctuation characters and converted to lowercase):

>

> * `<namespace_slug>_environment_type`: set to the type of environment (`review`, `integration`, `staging` or `production`),

> * `<namespace_slug>_environment_name`: the application name (see below),

> * `<namespace_slug>_environment_url`: set to the environment URL (whether determined statically or dynamically).

>

> The output dotenv file will be `cloudfoundry.env.<namespace_slug>` instead, and the dynamic variable `${environment_namespace}` can be used in your scripts and manifests to access the contextual value of `<namespace_slug>`.



## Configuration reference



### Secrets management
@@ -365,6 +412,7 @@ The Cloud Foundry template uses some global configuration used throughout all jo 
| Input / Variable                               | Description                                                                                                                                                                                     | Default Value                                                                                            |

| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |

| `cli-image` / `CF_CLI_IMAGE`                   | The Docker image used to run CF CLI commands <br/>:warning: **set the version required by your Cloud Foundry server**                                                                           | `docker.io/governmentpaas/cf-cli`                                                          <br/>[![Trivy Badge](https://to-be-continuous.gitlab.io/doc/secu/trivy-badge-CF_CLI_IMAGE.svg)](https://to-be-continuous.gitlab.io/doc/secu/trivy-CF_CLI_IMAGE) |

| `environment-namespace` / `CF_ENVIRONMENT_NAMESPACE`           | Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying [multiple environments](#multiple-environments-support))_<br/>:warning: must start with a slash `/` | _none_ |

| `manifest-basename` / `CF_MANIFEST_BASENAME`   | CF manifest file basename (without extension nor env suffix)                                                                                                                                    | `manifest`                                                                                               |

| `url`/ `CF_URL`                                | Default CF API url                                                                                                                                                                              | **has to be defined**                                                                                    |

| :lock: `CF_USER`                               | Default CF user name                                                                                                                                                                            | **has to be defined**                                                                                    |

kicker.json

+5 −0
Original line number Diff line number Diff line
@@ -11,6 +11,11 @@ 
      "description": "The Docker image used to run CF CLI commands - **set the version required by your Cloud Foundry server**",

      "default": "docker.io/governmentpaas/cf-cli"

    },

    {

      "name": "CF_ENVIRONMENT_NAMESPACE",

      "description": "Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying multiple environments)_\n\n:warning: must start with a slash `/`",

      "advanced": true

    },

    {

      "name": "CF_MANIFEST_BASENAME",

      "description": "CF manifest file basename (without extension nor env suffix)",

templates/gitlab-ci-cf.yml

+47 −49
Original line number Diff line number Diff line
@@ -21,6 +21,12 @@ spec: 
    manifest-basename:

      description: CF manifest file basename (without extension nor env suffix)

      default: manifest

    environment-namespace:

      description: |-

        Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying multiple environments)_

        

        :warning: must start with a slash `/`

      default: ''

    scripts-dir:

      description: Directory where Cloud Foundry scripts (manifest, hook scripts) are located

      default: "."
@@ -263,6 +269,7 @@ variables: 


  CF_MANIFEST_BASENAME: $[[ inputs.manifest-basename ]]

  CF_SCRIPTS_DIR: $[[ inputs.scripts-dir ]]

  CF_ENVIRONMENT_NAMESPACE: $[[ inputs.environment-namespace ]]

  CF_ROLLING_STRATEGY: $[[ inputs.rolling-strategy ]]



  CF_URL: $[[ inputs.url ]]
@@ -962,9 +969,11 @@ stages: 
    routepath=${ENV_ROUTE_PATH:-${CF_DEFAULT_ROUTE_PATH}}

    pushargs="${ENV_PUSH_ARGS:-${CF_DEFAULT_PUSH_ARGS}}"

    domain_tmp=${ENV_DOMAIN_TMP:-${domain}}

    environment_namespace=$(echo "$CF_ENVIRONMENT_NAMESPACE" | tr -d '[:punct:]' | tr '[:upper:]' '[:lower:]')

    export environment_namespace



    # unset any upstream deployment env & artifacts

    rm -f cloudfoundry.env

    rm -f cloudfoundry.env*

    rm -f environment_url.txt



    if [[ "$zdt" = "true" ]] && [[ "$CF_ROLLING_STRATEGY" != "true" ]]
@@ -978,8 +987,15 @@ stages: 


    # finally persist environment url

    echo "$CI_ENVIRONMENT_URL" > environment_url.txt

    echo -e "environment_type=$environment_type\\nenvironment_name=$environment_name\\nenvironment_url=$CI_ENVIRONMENT_URL" >> cloudfoundry.env

    chmod 644 environment_url.txt cloudfoundry.env

    # var prefix ('_' if namespace)

    prefix="${environment_namespace:+${environment_namespace}_}"

    dotenvfile="cloudfoundry.env${environment_namespace:+.${environment_namespace}}"

    {

      echo "${prefix}environment_type=${environment_type}"

      echo "${prefix}environment_name=${environment_name}"

      echo "${prefix}environment_url=${CI_ENVIRONMENT_URL}"

    } >> "$dotenvfile"

    chmod 644 environment_url.txt "$dotenvfile"

  }



  function pre_delete() {
@@ -1300,7 +1316,7 @@ stages: 
    - !reference [.cf-scripts]

    - install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}"



# Deploy job prototype

# Env management job prototype

# Can be extended to define a concrete environment

#

# @arg ENV_TYPE      : environment type
@@ -1311,27 +1327,35 @@ stages: 
# @arg ENV_PASSWORD  : env-specific Cloud Foundry password

# @arg ENV_ORG       : env-specific Cloud Foundry organization

# @arg ENV_SPACE     : env-specific Cloud Foundry space

# @arg ENV_ZERODOWNTIME: whether or not zero downtime deployment shall be used

# @arg ENV_DOMAIN    : env-specific domain

# @arg ENV_ROUTE_PATH: env-specific route path

# @arg ENV_HOST_NAME : env-specific application hostname to use

# @arg ENV_PUSH_ARGS : env-specific additional push command arguments

# @arg ENV_DOMAIN_TMP: env-specific domain during the -tmp app

# @arg ENV_RETIRED_APP_SUFFIX : If set, the app old version is not deleted/overriden but renamed with this suffix

.cf-deploy:

.cf-env-base:

  extends: .cf-base

  stage: deploy

  variables:

    ENV_APP_SUFFIX: "-$CI_ENVIRONMENT_SLUG"

  before_script:

    - !reference [.cf-scripts]

    - install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}"

    - !reference [.cf-base, before_script]

    - assert_defined "${ENV_URL:-$CF_URL}" 'Missing required Cloud Foundry url'

    - assert_defined "${ENV_USER:-$CF_USER}" 'Missing required Cloud Foundry user'

    - assert_defined "${ENV_PASSWORD:-$CF_PASSWORD}" 'Missing required Cloud Foundry password'

    - assert_defined "${ENV_ORG:-$CF_ORG}" 'Missing required Cloud Foundry organization'

    - assert_defined "$ENV_SPACE" 'Missing required Cloud Foundry space'

    - cf login -a ${ENV_URL:-$CF_URL} -u ${ENV_USER:-$CF_USER} -p "${ENV_PASSWORD:-$CF_PASSWORD}" -o ${ENV_ORG:-$CF_ORG} -s $ENV_SPACE

  environment:

    name: ${ENV_TYPE}${CF_ENVIRONMENT_NAMESPACE}

  resource_group: ${ENV_TYPE}${CF_ENVIRONMENT_NAMESPACE}



# Deploy job prototype

# Can be extended for each deployable environment

#

# @arg ENV_ZERODOWNTIME: whether or not zero downtime deployment shall be used

# @arg ENV_DOMAIN    : env-specific domain

# @arg ENV_ROUTE_PATH: env-specific route path

# @arg ENV_HOST_NAME : env-specific application hostname to use

# @arg ENV_PUSH_ARGS : env-specific additional push command arguments

# @arg ENV_DOMAIN_TMP: env-specific domain during the -tmp app

# @arg ENV_RETIRED_APP_SUFFIX : If set, the app old version is not deleted/overriden but renamed with this suffix

.cf-deploy:

  extends: .cf-env-base

  script:

    # use $CI_ENVIRONMENT_SLUG for environment_name to avoid service name constraints (<=50 chars)

    # use $CI_ENVIRONMENT_SLUG for hostname to avoid constraints (<=63 chars)
@@ -1344,35 +1368,14 @@ stages: 
      - environment_url.txt

      - cf-*.log

    reports:

      dotenv: cloudfoundry.env

      dotenv: cloudfoundry.env*



# Cleanup job prototype

# Can be extended for each deletable environment

#

# @arg ENV_TYPE      : environment type

# @arg ENV_APP_NAME  : env-specific application name

# @arg ENV_APP_SUFFIX: env-specific application suffix

# @arg ENV_URL       : env-specific Cloud Foundry API url

# @arg ENV_USER      : env-specific Cloud Foundry user

# @arg ENV_PASSWORD  : env-specific Cloud Foundry password

# @arg ENV_ORG       : env-specific Cloud Foundry organization

# @arg ENV_SPACE     : env-specific Cloud Foundry space

.cf-cleanup:

  extends: .cf-base

  stage: deploy

  extends: .cf-env-base

  # force no dependencies

  dependencies: []

  variables:

    ENV_APP_SUFFIX: "-$CI_ENVIRONMENT_SLUG"

  before_script:

    - !reference [.cf-scripts]

    - install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}"

    - assert_defined "${ENV_URL:-$CF_URL}" 'Missing required Cloud Foundry url'

    - assert_defined "${ENV_USER:-$CF_USER}" 'Missing required Cloud Foundry user'

    - assert_defined "${ENV_PASSWORD:-$CF_PASSWORD}" 'Missing required Cloud Foundry password'

    - assert_defined "${ENV_ORG:-$CF_ORG}" 'Missing required Cloud Foundry organization'

    - assert_defined "$ENV_SPACE" 'Missing required Cloud Foundry space'

    - cf login -a ${ENV_URL:-$CF_URL} -u ${ENV_USER:-$CF_USER} -p "${ENV_PASSWORD:-$CF_PASSWORD}" -o ${ENV_ORG:-$CF_ORG} -s $ENV_SPACE

  script:

    - cf_delete

    - manage_services "delete"
@@ -1400,11 +1403,13 @@ cf-review: 
    ENV_DOMAIN_TMP: "$CF_REVIEW_DOMAIN_TMP"

    ENV_RETIRED_APP_SUFFIX: "$CF_REVIEW_RETIRED_APP_SUFFIX"

  environment:

    name: review/$CI_COMMIT_REF_NAME

    name: ${ENV_TYPE}${CF_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}

    url: "${CF_REVIEW_ENVIRONMENT_SCHEME}://${CI_PROJECT_NAME}-${CI_ENVIRONMENT_SLUG}.${CF_REVIEW_ENVIRONMENT_DOMAIN}"

    # ⚠ on_stop must be unset when defining parallel:matrix environments

    # see: https://gitlab.com/gitlab-org/gitlab/-/issues/332247

    on_stop: cf-cleanup-review

    auto_stop_in: "$CF_REVIEW_AUTOSTOP_DURATION"

  resource_group: review/$CI_COMMIT_REF_NAME

  resource_group: ${ENV_TYPE}${CF_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}

  rules:

    # exclude tags

    - if: $CI_COMMIT_TAG
@@ -1427,9 +1432,9 @@ cf-cleanup-review: 
    ENV_ORG: "$CF_REVIEW_ORG"

    ENV_SPACE: "$CF_REVIEW_SPACE"

  environment:

    name: review/$CI_COMMIT_REF_NAME

    name: ${ENV_TYPE}${CF_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}

    action: stop

  resource_group: review/$CI_COMMIT_REF_NAME

  resource_group: ${ENV_TYPE}${CF_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}

  rules:

    # exclude tags

    - if: $CI_COMMIT_TAG
@@ -1448,8 +1453,7 @@ cf-cleanup-all-review: 
  stage: deploy

  dependencies: []

  before_script:

    - !reference [.cf-scripts]

    - install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}"

    - !reference [.cf-base, before_script]

    - assert_defined "${CF_REVIEW_URL:-$CF_URL}" 'Missing required env $CF_REVIEW_URL or $CF_URL'

    - assert_defined "${CF_REVIEW_USER:-$CF_USER}" 'Missing required env $CF_REVIEW_USER or $CF_USER'

    - assert_defined "${CF_REVIEW_PASSWORD:-$CF_PASSWORD}" 'Missing required env $CF_REVIEW_PASSWORD or $CF_PASSWORD'
@@ -1491,9 +1495,7 @@ cf-integration: 
    ENV_DOMAIN_TMP: "$CF_INTEG_DOMAIN_TMP"

    ENV_RETIRED_APP_SUFFIX: "$CF_INTEG_RETIRED_APP_SUFFIX"

  environment:

    name: integration

    url: "${CF_INTEG_ENVIRONMENT_URL}"

  resource_group: integration

  rules:

    # only on integration branch(es), with $CF_INTEG_SPACE set

    - if: '$CF_INTEG_SPACE != "" && $CI_COMMIT_REF_NAME =~ $INTEG_REF'
@@ -1521,9 +1523,7 @@ cf-staging: 
    ENV_DOMAIN_TMP: "$CF_STAGING_DOMAIN_TMP"

    ENV_RETIRED_APP_SUFFIX: "$CF_STAGING_RETIRED_APP_SUFFIX"

  environment:

    name: staging

    url: "${CF_STAGING_ENVIRONMENT_URL}"

  resource_group: staging

  rules:

    # exclude non-production branches

    - if: '$CI_COMMIT_REF_NAME !~ $PROD_REF'
@@ -1551,9 +1551,7 @@ cf-production: 
    ENV_DOMAIN_TMP: "$CF_PROD_DOMAIN_TMP"

    ENV_RETIRED_APP_SUFFIX: "$CF_PROD_RETIRED_APP_SUFFIX"

  environment:

    name: production

    url: "${CF_PROD_ENVIRONMENT_URL}"

  resource_group: production

  rules:

    # exclude non-production branches

    - if: '$CI_COMMIT_REF_NAME !~ $PROD_REF'