Merge branch 'feat/env-url-improvement' into 'master'

This project implements a generic GitLab CI template for [Google Cloud Platform](https://cloud.google.com/) environments.

## Overview: managed environments

## Usage

In order to include this template in your project, add the following to your `gitlab-ci.yml`:

```yaml

include:

  - project: 'to-be-continuous/gcloud'

    ref: '2.2.0'

    file: '/templates/gitlab-ci-gcloud.yml'

```

## Understand

This chapter introduces key notions and principle to understand how this template works.

### Managed deployment environments

This template implements continuous delivery/continuous deployment for projects hosted on Google Cloud Platform.

The following chapters present the managed predefined environments and their associated Git workflow.

### Review environments

#### Review environments

The template supports **review** environments: those are dynamic and ephemeral environments to deploy your

_ongoing developments_ (a.k.a. _feature_ or _topic_ branches).

It also comes with a _cleanup_ job (accessible either from the _environments_ page, or from the pipeline view).

### Integration environment

#### Integration environment

If you're using a Git Workflow with an integration branch (such as [Gitflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow)),

the template supports an **integration** environment.

When enabled, it deploys the result from upstream build stages to a dedicated environment.

It is only active for your integration branch (`develop` by default).

### Production environments

#### Production environments

Lastly, the template supports 2 environments associated to your production branch (`master` or `main` by default):

* **continuous deployment**: automatic deployment to production (when the upstream pipeline is successful),

* **continuous delivery**: deployment to production can be triggered manually (when the upstream pipeline is successful).

## Usage

### Supported authentication methods

### Include

The Google Cloud Platform template supports two kinds of authentication:

In order to include this template in your project, add the following to your `gitlab-ci.yml`:

1. basic authentication with [Service Account key file](https://cloud.google.com/bigquery/docs/authentication/service-account-file),

2. or [federated authentication using OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/).

```yaml

include:

  - project: 'to-be-continuous/gcloud'

    ref: '2.2.0'

    file: '/templates/gitlab-ci-gcloud.yml'

```

#### Service account authentication

### Global configuration

To use this authentication method, simply generate and provide [Service Account key file](https://cloud.google.com/bigquery/docs/authentication/service-account-file) as secret GitLab CI/CD variables (of type File), using the appropriate variables (see doc below).

The Google Cloud template uses some global configuration used throughout all jobs.

Can be provided globally and/or per environment.

| Name                     | description                            | default value     |

| ------------------------ | -------------------------------------- | ----------------- |

| `GCP_CLI_IMAGE`          | the Docker image used to run Google Cloud CLI commands| `google/cloud-sdk:latest` |

| :lock: `GCP_KEY_FILE`    | Default [Service Account key file](https://cloud.google.com/bigquery/docs/authentication/service-account-file) | _none_ |

| `GCP_OIDC_PROVIDER`      | Default Workload Identity Provider associated with GitLab to [authenticate with OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/) | none|

| `GCP_OIDC_ACCOUNT`       | Default Service Account to which impersonate with OpenID Connect authentication | none  |

| `GCP_BASE_APP_NAME`      | Base application name                  | `$CI_PROJECT_NAME` ([see GitLab doc](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)) |

| `GCP_SCRIPTS_DIR`        | Directory where Google Cloud scripts (deploy & cleanup) are located | `.` _(root project dir)_ |

### Secrets management

Here are some advices about your **secrets** (variables marked with a :lock:): 

1. Manage them as [project or group CI/CD variables](https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui):

    * [**masked**](https://docs.gitlab.com/ee/ci/variables/#mask-a-custom-variable) to prevent them from being inadvertently 

      displayed in your job logs,

    * [**protected**](https://docs.gitlab.com/ee/ci/variables/#protect-a-custom-variable) if you want to secure some secrets 

      you don't want everyone in the project to have access to (for instance production secrets).

2. In case a secret contains [characters that prevent it from being masked](https://docs.gitlab.com/ee/ci/variables/#masked-variable-requirements), 

  simply define its value as the [Base64](https://en.wikipedia.org/wiki/Base64) encoded value prefixed with `@b64@`: 

  it will then be possible to mask it and the template will automatically decode it prior to using it.

3. Don't forget to escape special characters (ex: `$` -> `$$`).

### Federated authentication using OpenID Connect

#### Federated authentication using OpenID Connect

The GCP template supports a [federated authentication using OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/).

- `gcloud iam workload-identity-pools list  --location=global --format="value(name)"` will list you POOL_IDs available on your `GCP_PROJECT`

- `gcloud iam workload-identity-pools providers list --workload-identity-pool=<my-pool>  --location=global --format="value(name)"` will return the list of available `PROVIDER_ID` for one `POOL_ID`

### Deployment and cleanup jobs

### Deployment context variables

In order to manage the various deployment environments, this template provides a couple of **dynamic variables**

that you might use in your hook scripts, deployment manifests and other deployment resources:

The GitLab CI template for Google Cloud requires you to provide a shell script that fully implements your application

deployment and cleanup using the [`gcloud` CLI](https://cloud.google.com/sdk/gcloud).

* `${environment_type}`: the current deployment environment type (`review`, `integration`, `staging` or `production`)

* `${environment_name}`: a generated application name to use for the current deployment environment (ex: `myapp-review-fix-bug-12` or `myapp-staging`) - _details below_

#### Lookup policy

#### Generated environment name

The `${environment_name}` variable is generated to designate each deployment environment with a unique and meaningful application name.

By construction, it is suitable for inclusion in DNS, URLs, Kubernetes labels...

It is built from:

* the application _base name_ (defaults to `$CI_PROJECT_NAME` but can be overridden globally and/or per deployment environment - _see configuration variables_)

* GitLab predefined `$CI_ENVIRONMENT_SLUG` variable ([sluggified](https://en.wikipedia.org/wiki/Clean_URL#Slug) name, truncated to 24 characters)

The `${environment_name}` variable is then evaluated as:

* `<app base name>` for the production environment

* `<app base name>-$CI_ENVIRONMENT_SLUG` for all other deployment environments

* :bulb: `${environment_name}` can also be overriden per environment with the appropriate configuration variable

Examples (with an application's base name `myapp`):

| `$environment_type` | Branch        | `$CI_ENVIRONMENT_SLUG`  | `$environment_name` |

|---------------------|---------------|-------------------------|---------------------|

| `review`            | `feat/blabla` | `review-feat-bla-xmuzs6`| `myapp-review-feat-bla-xmuzs6` |

| `integration`       | `develop`     | `integration`           | `myapp-integration` |

| `staging`           | `main`        | `staging`               | `myapp-staging` |

| `production`        | `main`        | `production`            | `myapp` |

### Deployment and cleanup scripts

The Google Cloud template requires you to provide a shell script that fully implements your application

deployment and cleanup using the [`gcloud` CLI](https://cloud.google.com/sdk/gcloud) and all other tools available in the selected Docker image.

The deployment script is searched as follows:

1. look for a specific `gcp-deploy-$env.sh` in the `$GCP_SCRIPTS_DIR` directory in your project (e.g. `gcp-deploy-staging.sh` for staging environment),

1. look for a specific `gcp-deploy-$environment_type.sh` in the `$GCP_SCRIPTS_DIR` directory in your project (e.g. `gcp-deploy-staging.sh` for staging environment),

2. if not found: look for a default `gcp-deploy.sh` in the `$GCP_SCRIPTS_DIR` directory in your project,

3. if not found: the deployment job will fail.

The cleanup script is searched as follows:

1. look for a specific `gcp-cleanup-$env.sh` in the `$GCP_SCRIPTS_DIR` directory in your project (e.g. `gcp-cleanup-staging.sh` for staging environment),

1. look for a specific `gcp-cleanup-$environment_type.sh` in the `$GCP_SCRIPTS_DIR` directory in your project (e.g. `gcp-cleanup-staging.sh` for staging environment),

2. if not found: look for a default `gcp-cleanup.sh` in the `$GCP_SCRIPTS_DIR` directory in your project,

3. if not found: the cleanup job will fail.

Your script(s) shall use available [dynamic variables](#dynamic-variables).

#### Dynamic Variables

You have to be aware that your deployment (and cleanup) scripts have to be able to cope with various environments 

(`review`, `integration`, `staging` and `production`), each with different application names, exposed routes, settings, ...

Part of this complexity can be handled by the lookup policies described above (ex: one resource per env).

In order to be able to implement some **genericity** in your scripts and templates, you should use available environment variables:

> :information_source: Your deployment (and cleanup) scripts have to be able to cope with various environments, each with different application names, exposed routes, settings, ...

> Part of this complexity can be handled by the lookup policies described above (ex: one script per env) and also by using available environment variables:

>

> 1. [deployment context variables](#deployment-context-variables) provided by the template:

>     * `${environment_type}`: the current environment type (`review`, `integration`, `staging` or `production`)

>     * `${environment_name}`: the application name to use for the current environment (ex: `myproject-review-fix-bug-12` or `myproject-staging`)

>     * `${hostname}`: the environment hostname, extracted from the current environment url (after late variable expansion - see below)

> 2. any [GitLab CI variable](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)

> 3. any [custom variable](https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project)

>     (ex: `${SECRET_TOKEN}` that you have set in your project CI/CD variables)

1. any [GitLab CI variable](https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables)

    (ex: `${CI_ENVIRONMENT_URL}` to retrieve the actual environment exposed route) 

2. any [custom variable](https://docs.gitlab.com/ee/ci/variables/#custom-environment-variables)

    (ex: `${SECRET_TOKEN}` that you have set in your project CI/CD variables)

3. **dynamic variables** set by the template:

    * `${appname}`: the application target name to use in this environment (ex: `myproject-review-fix-bug-12` or `myproject-staging`)

    * `${env}`: the environment type (`review`, `integration`, `staging` or `production`)

    * `${hostname}`: the environment hostname, extracted from `${CI_ENVIRONMENT_URL}` (has to be explicitly declared as [`environment:url`](https://docs.gitlab.com/ee/ci/yaml/#environmenturl) in your `.gitlab-ci.yml` file)

    * `${gcp_project_id}`: the current Google Cloud project ID associated to your environment

### Environments URL management

#### Static vs. Dynamic environment URLs

The GCP template supports two ways of providing your environments url:

The Google Cloud template supports two ways of defining your environments url:

* a **static way**: when you know your environments url in advance, probably because you're exposing your routes through a DNS you manage,

* a **static way**: when the environments url can be determined in advance, probably because you're exposing your routes through a DNS you manage,

* a [**dynamic way**](https://docs.gitlab.com/ee/ci/environments/#set-dynamic-environment-urls-after-a-job-finishes): when the url cannot be known before the

  deployment job is executed.

The static way can be implemented simply by setting the appropriate configuration variables depending on the environments (see environments configuration chapters below):

The **static way** can be implemented simply by setting the appropriate configuration variable(s) depending on the environment (see environments configuration chapters):

* `$GCP_ENVIRONMENT_URL` to define a default url pattern for all your envs,

* `$GCP_REVIEW_ENVIRONMENT_URL`, `$GCP_INTEG_ENVIRONMENT_URL`, `$GCP_STAGING_ENVIRONMENT_URL` and `$GCP_PROD_ENVIRONMENT_URL` to override the default.

* `$GCP_REVIEW_ENVIRONMENT_SCHEME` and`$GCP_REVIEW_ENVIRONMENT_DOMAIN` for the review environments, 

* `$GCP_INTEG_ENVIRONMENT_URL`, `$GCP_STAGING_ENVIRONMENT_URL` and `$GCP_PROD_ENVIRONMENT_URL` for others.

> :information_source: Each of those variables support a **late variable expansion mechanism** with the `%{somevar}` syntax, 

> allowing you to use any dynamically evaluated variables such as `${environment_name}`.

>

> Example:

>

> ```yaml

> variables:

>   GCP_BASE_APP_NAME: "wonderapp"

>   # global url for all environments

>   GCP_ENVIRONMENT_URL: "https://%{environment_name}.nonprod.acme.domain"

>   # override for prod (late expansion of $GCP_BASE_APP_NAME not needed here)

>   GCP_PROD_ENVIRONMENT_URL: "https://$GCP_BASE_APP_NAME.acme.domain"

>   # override for review (using separate resource paths)

>   GCP_REVIEW_ENVIRONMENT_URL: "https://wonderapp-review.nonprod.acme.domain/%{environment_name}"

> ```

To implement the dynamic way, your deployment script shall simply generate a `environment_url.txt` file, containing only

the dynamically generated url.

To implement the **dynamic way**, your deployment script shall simply generate a `environment_url.txt` file in the working directory, containing only

the dynamically generated url. When detected by the template, it will use it as the newly deployed environment url.

#### Deployment output variables

### Deployment output variables

Each deployment job produces _output variables_ that are propagated to downstream jobs (using [dotenv artifacts](https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#artifactsreportsdotenv)):

* `$environment_type`: set to the type of environment (`review`, `integration`, `staging` or `production`),

* `$environment_name`: the application name (see below),

* `$environment_url`: set to `$CI_ENVIRONMENT_URL`.

* `$environment_url`: set to the environment URL (whether determined statically or dynamically).

Those variables may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).

### Environments configuration

## Confuguration reference

As seen above, the Google Cloud template may support up to 4 environments (`review`, `integration`, `staging` and `production`).

### Secrets management

Here are configuration details for each environment.

Here are some advices about your **secrets** (variables marked with a :lock:): 

#### Review environments

1. Manage them as [project or group CI/CD variables](https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui):

    * [**masked**](https://docs.gitlab.com/ee/ci/variables/#mask-a-custom-variable) to prevent them from being inadvertently 

      displayed in your job logs,

    * [**protected**](https://docs.gitlab.com/ee/ci/variables/#protect-a-custom-variable) if you want to secure some secrets 

      you don't want everyone in the project to have access to (for instance production secrets).

2. In case a secret contains [characters that prevent it from being masked](https://docs.gitlab.com/ee/ci/variables/#masked-variable-requirements), 

  simply define its value as the [Base64](https://en.wikipedia.org/wiki/Base64) encoded value prefixed with `@b64@`: 

  it will then be possible to mask it and the template will automatically decode it prior to using it.

3. Don't forget to escape special characters (ex: `$` -> `$$`).

### Global configuration

The Google Cloud template uses some global configuration used throughout all jobs.

| Name                     | description                            | default value     |

| ------------------------ | -------------------------------------- | ----------------- |

| `GCP_CLI_IMAGE`          | the Docker image used to run Google Cloud CLI commands| `google/cloud-sdk:latest` |

| :lock: `GCP_KEY_FILE`    | Default [Service Account key file](https://cloud.google.com/bigquery/docs/authentication/service-account-file) | _none_ |

| `GCP_OIDC_PROVIDER`      | Default Workload Identity Provider associated with GitLab to [authenticate with OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/) | none|

| `GCP_OIDC_ACCOUNT`       | Default Service Account to which impersonate with OpenID Connect authentication | none  |

| `GCP_BASE_APP_NAME`      | Base application name                  | `$CI_PROJECT_NAME` ([see GitLab doc](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)) |

| `GCP_ENVIRONMENT_URL`    | Default environments url _(only define for static environment URLs declaration)_<br/>_supports late variable expansion (ex: `https://%{environment_name}.gcloud.acme.com`)_ | _none_ |

| `GCP_SCRIPTS_DIR`        | Directory where Google Cloud scripts (deploy & cleanup) are located | `.` _(root project dir)_ |

### Review environments configuration

Review environments are dynamic and ephemeral environments to deploy your _ongoing developments_ (a.k.a. _feature_ or 

_topic_ branches).

| `GCP_REVIEW_PROJECT`     | Google Cloud project ID for `review` env | _none_ (disabled) |

| :lock: `GCP_REVIEW_KEY_FILE`| [Service Account key file](https://cloud.google.com/bigquery/docs/authentication/service-account-file) to authenticate on `review` env  _(only define if different from default)_    | `$GCP_KEY_FILE` |

| `GCP_REVIEW_APP_NAME`    | Application name for `review` env      | `"${GCP_BASE_APP_NAME}-${CI_ENVIRONMENT_SLUG}"` (ex: `myproject-review-fix-bug-12`) |

| `GCP_REVIEW_ENVIRONMENT_SCHEME`| The review environment protocol scheme.<br/>_For static environment URLs declaration_ | `https` |

| `GCP_REVIEW_ENVIRONMENT_DOMAIN`| The review environment domain.<br/>_For static environment URLs declaration_ | _none_ |

| `GCP_REVIEW_ENVIRONMENT_URL`| The review environments url _(only define for static environment URLs declaration and if different from default)_ | `$GCP_ENVIRONMENT_URL` |

| `GCP_REVIEW_OIDC_PROVIDER` | Workload Identity Provider associated with GitLab to [authenticate with OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/) on `review` environment | none|

| `GCP_REVIEW_OIDC_ACCOUNT`  | Service Account to which impersonate with OpenID Connect authentication on `review` environment | none  |

Note: If you're managing your environment URLs statically, review environment URLs will be built as `${AWS_REVIEW_ENVIRONMENT_SCHEME}://${$CI_PROJECT_NAME}-${CI_ENVIRONMENT_SLUG}.${AWS_REVIEW_ENVIRONMENT_DOMAIN}`

#### Integration environment

### Integration environment configuration

The integration environment is the environment associated to your integration branch (`develop` by default).

| `GCP_INTEG_PROJECT`      | Google Cloud project ID for `integration` env | _none_ (disabled) |

| :lock: `GCP_INTEG_KEY_FILE`|[Service Account key file](https://cloud.google.com/bigquery/docs/authentication/service-account-file) to authenticate on `integration` env  _(only define if different from default)_    | `$GCP_KEY_FILE` |

| `GCP_INTEG_APP_NAME`     | Application name for `integration` env | `${GCP_BASE_APP_NAME}-integration` |

| `GCP_INTEG_ENVIRONMENT_URL`| The integration environment url (ex: `https://my-application-integration.nonpublic.domain.com`).<br/>_For static environment URLs declaration_ | _none_ |

| `GCP_INTEG_ENVIRONMENT_URL`| The integration environment url _(only define for static environment URLs declaration and if different from default)_ | `$GCP_ENVIRONMENT_URL` |

| `GCP_INTEG_OIDC_PROVIDER` | Workload Identity Provider associated with GitLab to [authenticate with OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/) on `integration` environment | none|

| `GCP_INTEG_OIDC_ACCOUNT`  | Service Account to which impersonate with OpenID Connect authentication on `integration` environment | none  |

#### Staging environment

### Staging environment configuration

The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production 

branch (`master` by default).

| `GCP_STAGING_PROJECT`    | Google Cloud project ID for `staging` env | _none_ (disabled) |

| :lock: `GCP_STAGING_KEY_FILE`|[Service Account key file](https://cloud.google.com/bigquery/docs/authentication/service-account-file) to authenticate on `staging` env  _(only define if different from default)_    | `$GCP_KEY_FILE` |

| `GCP_STAGING_APP_NAME`   | Application name for `staging` env     | `${GCP_BASE_APP_NAME}-staging` |

| `GCP_STAGING_ENVIRONMENT_URL` | The staging environment url (ex: `https://my-application-staging.nonpublic.domain.com`).<br/>_For static environment URLs declaration_ | _none_ |

| `GCP_STAGING_ENVIRONMENT_URL`| The staging environment url _(only define for static environment URLs declaration and if different from default)_ | `$GCP_ENVIRONMENT_URL` |

| `GCP_STAGING_OIDC_PROVIDER` | Workload Identity Provider associated with GitLab to [authenticate with OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/) on `staging` environment | none|

| `GCP_STAGING_OIDC_ACCOUNT`  | Service Account to which impersonate with OpenID Connect authentication on `staging` environment | none  |

#### Production environment

### Production environment configuration

The production environment is the final deployment environment associated with your production branch (`master` by default).

| `GCP_PROD_PROJECT`        | Google Cloud project ID for `production` env | _none_ (disabled) |

| :lock: `GCP_PROD_KEY_FILE`|[Service Account key file](https://cloud.google.com/bigquery/docs/authentication/service-account-file) to authenticate on `production` env  _(only define if different from default)_    | `$GCP_KEY_FILE` |

| `GCP_PROD_APP_NAME`       | Application name for `production` env  | `$GCP_BASE_APP_NAME` |

| `GCP_PROD_ENVIRONMENT_URL`| The production environment url (ex: `https://my-application.public.domain.com`).<br/>_For static environment URLs declaration_ | _none_ |

| `GCP_PROD_ENVIRONMENT_URL`| The production environment url _(only define for static environment URLs declaration and if different from default)_ | `$GCP_ENVIRONMENT_URL` |

| `AUTODEPLOY_TO_PROD`      | Set this variable to auto-deploy to production. If not set deployment to production will be `manual` (default behaviour). | _none_ (disabled) |

| `GCP_PROD_OIDC_PROVIDER` | Workload Identity Provider associated with GitLab to [authenticate with OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/) on `production ` environment | none|

| `GCP_PROD_OIDC_ACCOUNT`  | Service Account to which impersonate with OpenID Connect authentication on `production ` environment | none  |

## Examples

### Google AppEngine application