Commit 1130b17f authored by Pierre Smeyers's avatar Pierre Smeyers
Browse files

feat: manage static or dynamic environment urls

parent 61ec77ec
Loading
Loading
Loading
Loading
+56 −45
Original line number Diff line number Diff line
@@ -78,6 +78,62 @@ Here are some advices about your **secrets** (variables marked with a :lock:):
  it will then be possible to mask it and the template will automatically decode it prior to using it.
3. Don't forget to escape special characters (ex: `$` -> `$$`).

### Deployment and cleanup jobs

The GitLab CI template for Google Cloud requires you to provide a shell script that fully implements your application
deployment and cleanup using the [`gcloud` CLI](https://cloud.google.com/sdk/gcloud).

#### Lookup policy

The deployment script is searched as follows:

1. look for a specific `gcp-deploy-$env.sh` in the `$GCP_SCRIPTS_DIR` directory in your project (e.g. `gcp-deploy-staging.sh` for staging environment),
2. if not found: look for a default `gcp-deploy.sh` in the `$GCP_SCRIPTS_DIR` directory in your project,
3. if not found: the deployment job will fail.

The cleanup script is searched as follows:

1. look for a specific `gcp-cleanup-$env.sh` in the `$GCP_SCRIPTS_DIR` directory in your project (e.g. `gcp-cleanup-staging.sh` for staging environment),
2. if not found: look for a default `gcp-cleanup.sh` in the `$GCP_SCRIPTS_DIR` directory in your project,
3. if not found: the cleanup job will fail.

Your script(s) shall use available [dynamic variables](#dynamic-variables).

#### Dynamic Variables

You have to be aware that your deployment (and cleanup) scripts have to be able to cope with various environments 
(`review`, `integration`, `staging` and `production`), each with different application names, exposed routes, settings, ...

Part of this complexity can be handled by the lookup policies described above (ex: one resource per env).

In order to be able to implement some **genericity** in your scripts and templates, you should use available environment variables:

1. any [GitLab CI variable](https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables)
    (ex: `${CI_ENVIRONMENT_URL}` to retrieve the actual environment exposed route) 
2. any [custom variable](https://docs.gitlab.com/ee/ci/variables/#custom-environment-variables)
    (ex: `${SECRET_TOKEN}` that you have set in your project CI/CD variables)
3. **dynamic variables** set by the template:
    * `${appname}`: the application target name to use in this environment (ex: `myproject-review-fix-bug-12` or `myproject-staging`)
    * `${env}`: the environment type (`review`, `integration`, `staging` or `production`)
    * `${hostname}`: the environment hostname, extracted from `${CI_ENVIRONMENT_URL}` (has to be explicitly declared as [`environment:url`](https://docs.gitlab.com/ee/ci/yaml/#environmenturl) in your `.gitlab-ci.yml` file)
    * `${gcp_project_id}`: the current Google Cloud project ID associated to your environment

#### Static vs. Dynamic environment URLs

The Google Cloud template supports two ways of defining your environments url:

* a **static way**: when you know your environments url in advance, probably because you're exposing your routes through a DNS you manage,
* a [**dynamic way**](https://docs.gitlab.com/ee/ci/environments/#set-dynamic-environment-urls-after-a-job-finishes): when the url cannot be known before the
  deployment job is executed.

The static way can be implemented simply by setting the appropriate configuration variables depending on the environments (see environments configuration chapters below):

* `$GCP_REVIEW_ENVIRONMENT_SCHEME` and`$GCP_REVIEW_ENVIRONMENT_DOMAIN` for the review environments, 
* `$GCP_INTEG_ENVIRONMENT_URL`, `$GCP_STAGING_ENVIRONMENT_URL` and `$GCP_PROD_ENVIRONMENT_URL` for others.

To implement the dynamic way, your deployment script shall simply generate a `environment_url.txt` file, containing only
the dynamically generated url.

### Environments configuration

As seen above, the Google Cloud template may support up to 4 environments (`review`, `integration`, `staging` and `production`).
@@ -158,51 +214,6 @@ Here are variables supported to configure the production environment:
| `GCP_PROD_ENVIRONMENT_URL`| The production environment url **including scheme** (ex: `https://my-application.public.domain.com`) Do not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that. | _none_ |
| `AUTODEPLOY_TO_PROD`      | Set this variable to auto-deploy to production. If not set deployment to production will be `manual` (default behaviour). | _none_ (disabled) |

### Deployment jobs

The GitLab CI template for Google Cloud requires you to provide a shell script that fully implements your application
deployment using the [`gcloud` CLI](https://cloud.google.com/sdk/gcloud).

The deployment script is searched as follows:

1. look for a specific `gcp-deploy-$env.sh` in the `$GCP_SCRIPTS_DIR` directory in your project (e.g. `gcp-deploy-staging.sh` for staging environment),
2. if not found: look for a default `gcp-deploy.sh` in the `$GCP_SCRIPTS_DIR` directory in your project,
3. if not found: the deployment job will fail.

Your script(s) shall use available [dynamic variables](#dynamic-variables).

### Cleanup jobs

The GitLab CI template for Google Cloud requires you to provide a shell script that fully implements your application
cleanup using the [`gcloud` CLI](https://cloud.google.com/sdk/gcloud).

The cleanup script is searched as follows:

1. look for a specific `gcp-cleanup-$env.sh` in the `$GCP_SCRIPTS_DIR` directory in your project (e.g. `gcp-cleanup-staging.sh` for staging environment),
2. if not found: look for a default `gcp-cleanup.sh` in the `$GCP_SCRIPTS_DIR` directory in your project,
3. if not found: the cleanup job will fail.

Your script(s) shall use available [dynamic variables](#dynamic-variables).

### Dynamic Variables

You have to be aware that your deployment (and cleanup) scripts have to be able to cope with various environments 
(`review`, `integration`, `staging` and `production`), each with different application names, exposed routes, settings, ...

Part of this complexity can be handled by the lookup policies described above (ex: one resource per env).

In order to be able to implement some **genericity** in your scripts and templates, you should use available environment variables:

1. any [GitLab CI variable](https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables)
    (ex: `${CI_ENVIRONMENT_URL}` to retrieve the actual environment exposed route) 
2. any [custom variable](https://docs.gitlab.com/ee/ci/variables/#custom-environment-variables)
    (ex: `${SECRET_TOKEN}` that you have set in your project CI/CD variables)
3. **dynamic variables** set by the template:
    * `${appname}`: the application target name to use in this environment (ex: `myproject-review-fix-bug-12` or `myproject-staging`)
    * `${env}`: the environment type (`review`, `integration`, `staging` or `production`)
    * `${hostname}`: the environment hostname, extracted from `${CI_ENVIRONMENT_URL}` (has to be explicitly declared as [`environment:url`](https://docs.gitlab.com/ee/ci/yaml/#environmenturl) in your `.gitlab-ci.yml` file)
    * `${gcp_project_id}`: the current Google Cloud project ID associated to your environment

## Examples

### Google AppEngine application
+26 −9
Original line number Diff line number Diff line
@@ -241,8 +241,9 @@ stages:
    export env=$1
    export appname=$2
    export gcp_project_id=$3
    # extract hostname from $CI_ENVIRONMENT_URL
    hostname=$(echo "$CI_ENVIRONMENT_URL" | awk -F[/:] '{print $4}')
    export environment_url=$4
    # extract hostname from $environment_url
    hostname=$(echo "$environment_url" | awk -F[/:] '{print $4}')
    export hostname

    log_info "--- \\e[32mdeploy\\e[0m (env: \\e[33;1m${env}\\e[0m)"
@@ -251,6 +252,12 @@ stages:
    log_info "--- \$hostname: \\e[33;1m${hostname}\\e[0m"
    log_info "--- \$gcp_project_id: \\e[33;1m${gcp_project_id}\\e[0m"

    # unset any upstream deployment env & artifacts
    unset environment_name
    unset environment_type
    rm -f gcloud.env
    rm -f environment_url.txt

    deployscript=$(ls -1 "$GCP_SCRIPTS_DIR/gcp-deploy-${env}.sh" 2>/dev/null || ls -1 "$GCP_SCRIPTS_DIR/gcp-deploy.sh" 2>/dev/null || echo "")
    if [[ -f "$deployscript" ]]
    then
@@ -262,8 +269,15 @@ stages:
    fi

    # finally persist environment url
    echo "$CI_ENVIRONMENT_URL" > environment_url.txt
    echo -e "environment_type=$env\\nenvironment_name=$appname\\nenvironment_url=$CI_ENVIRONMENT_URL" > gcloud.env
    if [[ -f environment_url.txt ]]
    then
      environment_url=$(cat environment_url.txt)
      export environment_url
      log_info "--- dynamic environment url found: (\\e[33;1m$environment_url\\e[0m)"
    else
      echo "$environment_url" > environment_url.txt
    fi
    echo -e "environment_type=$env\\nenvironment_name=$appname\\nenvironment_url=$environment_url" > gcloud.env
  }

  # environment cleanup function
@@ -335,6 +349,7 @@ stages:
# @arg ENV_APP_SUFFIX: env-specific application suffix
# @arg ENV_PROJECT   : env-specific GCP Project ID
# @arg ENV_KEY_FILE  : env-specific GCP API key file (JSON)
# @arg ENV_URL       : env-specific application url
.gcp-deploy:
  extends: .gcp-base
  stage: deploy
@@ -346,13 +361,15 @@ stages:
    - assert_defined "${ENV_KEY_FILE:-$GCP_KEY_FILE}" 'Missing required GCP key file (JSON)'
    - gcloud auth activate-service-account --key-file ${ENV_KEY_FILE:-$GCP_KEY_FILE}
  script:
    - deploy "$ENV_TYPE" "${ENV_APP_NAME:-${GCP_BASE_APP_NAME}${ENV_APP_SUFFIX}}" "$ENV_PROJECT"
    - deploy "$ENV_TYPE" "${ENV_APP_NAME:-${GCP_BASE_APP_NAME}${ENV_APP_SUFFIX}}" "$ENV_PROJECT" "$ENV_URL"
  artifacts:
    name: "$ENV_TYPE env url for $CI_PROJECT_NAME on $CI_COMMIT_REF_SLUG"
    paths:
      - environment_url.txt
    reports:
      dotenv: gcloud.env
  environment:
    url: "$environment_url" # can be either static or dynamic

# Cleanup job prototype
# Can be extended for each deletable environment
@@ -389,9 +406,9 @@ gcp-review:
    ENV_APP_NAME: "$GCP_REVIEW_APP_NAME"
    ENV_PROJECT: "$GCP_REVIEW_PROJECT"
    ENV_KEY_FILE: "$GCP_REVIEW_KEY_FILE"
    ENV_URL: "${GCP_REVIEW_ENVIRONMENT_SCHEME}://${CI_PROJECT_NAME}-${CI_ENVIRONMENT_SLUG}.${GCP_REVIEW_ENVIRONMENT_DOMAIN}"
  environment:
    name: review/$CI_COMMIT_REF_NAME
    url: "${GCP_REVIEW_ENVIRONMENT_SCHEME}://${CI_PROJECT_NAME}-${CI_ENVIRONMENT_SLUG}.${GCP_REVIEW_ENVIRONMENT_DOMAIN}"
    on_stop: gcp-cleanup-review
  resource_group: review/$CI_COMMIT_REF_NAME
  rules:
@@ -439,9 +456,9 @@ gcp-integration:
    ENV_APP_NAME: "$GCP_INTEG_APP_NAME"
    ENV_PROJECT: "$GCP_INTEG_PROJECT"
    ENV_KEY_FILE: "$GCP_INTEG_KEY_FILE"
    ENV_URL: "${GCP_INTEG_ENVIRONMENT_URL}"
  environment:
    name: integration
    url: "${GCP_INTEG_ENVIRONMENT_URL}"
  resource_group: integration
  rules:
    # exclude merge requests
@@ -458,9 +475,9 @@ gcp-staging:
    ENV_APP_NAME: "$GCP_STAGING_APP_NAME"
    ENV_PROJECT: "$GCP_STAGING_PROJECT"
    ENV_KEY_FILE: "$GCP_STAGING_KEY_FILE"
    ENV_URL: "${GCP_STAGING_ENVIRONMENT_URL}"
  environment:
    name: staging
    url: "${GCP_STAGING_ENVIRONMENT_URL}"
  resource_group: staging
  rules:
    # exclude merge requests
@@ -479,9 +496,9 @@ gcp-production:
    ENV_APP_NAME: "$GCP_PROD_APP_NAME"
    ENV_PROJECT: "$GCP_PROD_PROJECT"
    ENV_KEY_FILE: "$GCP_PROD_KEY_FILE"
    ENV_URL: "${GCP_PROD_ENVIRONMENT_URL}"
  environment:
    name: production
    url: "${GCP_PROD_ENVIRONMENT_URL}"
  resource_group: production
  rules:
    # exclude merge requests