This project implements a generic GitLab CI template for[Amazon Web Services](https://aws.amazon.com/) environments.
This project implements a generic GitLab CI template to deploy to[Amazon Web Services](https://aws.amazon.com/).
## Overview: managed environments
## Usage
In order to include this template in your project, add the following to your `gitlab-ci.yml`:
```yaml
include:
-project:'to-be-continuous/aws'
ref:'2.2.0'
file:'/templates/gitlab-ci-aws.yml'
```
## Understand
This chapter introduces key notions and principle to understand how this template works.
### Managed deployment environments
This template implements continuous delivery/continuous deployment for projects hosted on AWS.
@@ -13,7 +28,7 @@ workflow, by reusing/extending the base (hidden) jobs. This is advanced usage an
The following chapters present the managed predefined environments and their associated Git workflow.
### Review environments
#### Review environments
The template supports **review** environments: those are dynamic and ephemeral environments to deploy your
_ongoing developments_ (a.k.a. _feature_ or _topic_ branches).
@@ -25,7 +40,7 @@ It is a strict equivalent of GitLab's [Review Apps](https://docs.gitlab.com/ee/c
It also comes with a _cleanup_ job (accessible either from the _environments_ page, or from the pipeline view).
### Integration environment
#### Integration environment
If you're using a Git Workflow with an integration branch (such as [Gitflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow)),
the template supports an **integration** environment.
@@ -33,7 +48,7 @@ the template supports an **integration** environment.
When enabled, it deploys the result from upstream build stages to a dedicated environment.
It is only active for your integration branch (`develop` by default).
### Production environments
#### Production environments
Lastly, the template supports 2 environments associated to your production branch (`master` or `main` by default):
@@ -45,89 +60,65 @@ You're free to enable whichever or both, and you can also choose your deployment
***continuous deployment**: automatic deployment to production (when the upstream pipeline is successful),
***continuous delivery**: deployment to production can be triggered manually (when the upstream pipeline is successful).
## Usage
### Include
### Supported authentication methods
In order to include this template in your project, add the following to your `gitlab-ci.yml`:
```yaml
include:
-project:'to-be-continuous/aws'
ref:'2.2.0'
file:'/templates/gitlab-ci-aws.yml'
```
### Global configuration
The AWS template supports two kinds of authentication:
The AWS template uses some global configuration used throughout all jobs.
1. basic authentication with AWS access key ID & secret access key,
2. or [federated authentication using OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/aws/).
| `AWS_SCRIPTS_DIR` | Directory where AWS scripts (deploy & cleanup) are located | `.` _(root project dir)_ |
| `AWS_OIDC_ROLE_ARN` | Default IAM Role ARN associated with GitLab to [authenticate using OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/aws/) | _none_ (disabled) |
#### Basic authentication
### Secrets management
When using basic authentication (AWS access key ID & secret access key), this template **does not manage AWS authentication**.
Here are some advices about your **secrets** (variables marked with a :lock:):
That means you'll have to manage AWS authentication by yourself, according to the `aws` CLI configuration options (configuration file, CLI options, environment variables).
1. Manage them as [project or group CI/CD variables](https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui):
* [**masked**](https://docs.gitlab.com/ee/ci/variables/#mask-a-custom-variable) to prevent them from being inadvertently
displayed in your job logs,
* [**protected**](https://docs.gitlab.com/ee/ci/variables/#protect-a-custom-variable) if you want to secure some secrets
you don't want everyone in the project to have access to (for instance production secrets).
2. In case a secret contains [characters that prevent it from being masked](https://docs.gitlab.com/ee/ci/variables/#masked-variable-requirements),
simply define its value as the [Base64](https://en.wikipedia.org/wiki/Base64) encoded value prefixed with `@b64@`:
it will then be possible to mask it and the template will automatically decode it prior to using it.
3. Don't forget to escape special characters (ex: `$` -> `$$`).
For credentials management, we strongly advise to use [environment variables configuration](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list), managed as GitLab CI secret variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, possibly `AWS_ROLE_ARN`).
### Deployment and cleanup scripts
If you have to manage different set of authentication credentials depending on managed environments, you shall either use [GitLab scoped variables](https://docs.gitlab.com/ee/ci/environments/index.html#scoping-environments-with-specs) or our [scoped variables syntax](https://to-be-continuous.gitlab.io/doc/usage/#scoped-variables) to limit/override some variables values, using `$CI_ENVIRONMENT_NAME` as the conditional variable.
The AWS template requires you to provide a shell script that fully implements your application
deployment and cleanup using the [`aws` CLI](https://aws.amazon.com/cli/) and all other tools available in the selected Docker image.
Example: different credentials for production (declared as project variables)
1. look for a specific `aws-deploy-$environment_type.sh` in the `$AWS_SCRIPTS_DIR` directory in your project (e.g. `aws-deploy-staging.sh` for staging environment),
2. if not found: look for a default `aws-deploy.sh` in the `$AWS_SCRIPTS_DIR` directory in your project,
3. if not found: the deployment job will fail.
#### Federated authentication using OpenID Connect
The cleanup script is searched as follows:
If you wish to use this authentication mode, please follow carefully [the GitLab guide](https://docs.gitlab.com/ee/ci/cloud_services/aws/),
then configure appropriately the related variables:
1. look for a specific `aws-cleanup-$environment_type.sh` in the `$AWS_SCRIPTS_DIR` directory in your project (e.g. `aws-cleanup-staging.sh` for staging environment),
2. if not found: look for a default `aws-cleanup.sh` in the `$AWS_SCRIPTS_DIR` directory in your project,
3. if not found: the cleanup job will fail.
*`AWS_OIDC_ROLE_ARN` for any global/common access,
*`AWS_REVIEW_OIDC_ROLE_ARN` and/or `AWS_INTEG_OIDC_ROLE_ARN` and/or `AWS_STAGING_OIDC_ROLE_ARN` and/or `AWS_PROD_OIDC_ROLE_ARN` if you wish to use a separate role with any of your environments.
#### Dynamic Variables
### Deployment context variables
You have to be aware that your deployment (and cleanup) scripts have to be able to cope with various environments
(`review`, `integration`, `staging` and `production`), each with different application names, exposed routes, settings, ...
In order to manage the various deployment environments, this template provides a couple of **dynamic variables**
that you might use in your hook scripts, deployment manifests and other deployment resources:
Part of this complexity can be handled by the lookup policies described above (ex: one resource per env).
*`${environment_type}`: the current deployment environment type (`review`, `integration`, `staging` or `production`)
*`${environment_name}`: a generated application name to use for the current deployment environment (ex: `myapp-review-fix-bug-12` or `myapp-staging`) - _details below_
In order to be able to implement some **genericity** in your scripts and templates, you should use available environment variables:
#### Generated environment name
1. any [GitLab CI variable](https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables)
(ex: `${CI_ENVIRONMENT_URL}` to retrieve the actual environment exposed route)
2. any [custom variable](https://docs.gitlab.com/ee/ci/variables/#custom-environment-variables)
(ex: `${SECRET_TOKEN}` that you have set in your project CI/CD variables)
3. **dynamic variables** set by the template:
* `${environment_type}`: the current environment type (`review`, `integration`, `staging` or `production`)
* `${environment_name}`: the application name to use for the current environment (ex: `myproject-review-fix-bug-12` or `myproject-staging`) - see below
* `${hostname}`: the environment hostname, extracted from the current environment url (after late variable expansion - see below)
The `${environment_name}` variable is generated to designate each deployment environment with a unique and meaningful application name.
By construction, it is suitable for inclusion in DNS, URLs, Kubernetes labels...
It is built from:
##### `environment_name` generation
* the application _base name_ (defaults to `$CI_PROJECT_NAME` but can be overridden globally and/or per deployment environment - _see configuration variables_)
Each environment is given an _environmentname_, generated programmatically by the template, using:
The `${environment_name}` variable is then evaluated as:
* `AWS_BASE_APP_NAME` defines the application _base_ name (defaults to `$CI_PROJECT_NAME`)
* `environment_name=$AWS_BASE_APP_NAME-$CI_ENVIRONMENT_SLUG` except for the production environment, where `environment_name` is directly `AWS_BASE_APP_NAME` (no suffix)
* each environmentname can be overriden per environment with the appropriate `AWS_XXX_APP_NAME` variable
*`<app base name>` for the production environment
*`<app base name>-$CI_ENVIRONMENT_SLUG` for all other deployment environments
*:bulb: `${environment_name}` can also be overriden per environment with the appropriate configuration variable
Examples (with an application's base name `myapp`):
@@ -138,57 +129,49 @@ Examples (with an application's base name `myapp`):
The AWS template supports two kinds of authentication:
1. basic authentication with AWS access key ID & secret access key,
2. or [federated authentication using OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/aws/).
##### Basic authentication
When using basic authentication (AWS access key ID & secret access key), this template **does not manage AWS authentication**.
That means you'll have to manage AWS authentication by yourself, according to the `aws` CLI configuration options (configuration file, CLI options, environment variables).
For credentials management, we strongly advise to use [environment variables configuration](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list), managed as GitLab CI secret variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, possibly `AWS_ROLE_ARN`).
If you have to manage different set of authentication credentials depending on managed environments, you shall either use [GitLab scoped variables](https://docs.gitlab.com/ee/ci/environments/index.html#scoping-environments-with-specs) or our [scoped variables syntax](https://to-be-continuous.gitlab.io/doc/usage/#scoped-variables) to limit/override some variables values, using `$CI_ENVIRONMENT_NAME` as the conditional variable.
### Deployment and cleanup scripts
Example: different credentials for production (declared as project variables)
The AWS template requires you to provide a shell script that fully implements your application
deployment and cleanup using the [`aws` CLI](https://aws.amazon.com/cli/) and all other tools available in the selected Docker image.
1. look for a specific `aws-deploy-$environment_type.sh` in the `$AWS_SCRIPTS_DIR` directory in your project (e.g. `aws-deploy-staging.sh` for staging environment),
2. if not found: look for a default `aws-deploy.sh` in the `$AWS_SCRIPTS_DIR` directory in your project,
3. if not found: the deployment job will fail.
##### Federated authentication using OpenID Connect
The cleanup script is searched as follows:
If you wish to use this authentication mode, please follow carefully [the GitLab guide](https://docs.gitlab.com/ee/ci/cloud_services/aws/),
then configure appropriately the related variables:
1. look for a specific `aws-cleanup-$environment_type.sh` in the `$AWS_SCRIPTS_DIR` directory in your project (e.g. `aws-cleanup-staging.sh` for staging environment),
2. if not found: look for a default `aws-cleanup.sh` in the `$AWS_SCRIPTS_DIR` directory in your project,
3. if not found: the cleanup job will fail.
* `AWS_OIDC_ROLE_ARN` for any global/common access,
* `AWS_REVIEW_OIDC_ROLE_ARN` and/or `AWS_INTEG_OIDC_ROLE_ARN` and/or `AWS_STAGING_OIDC_ROLE_ARN` and/or `AWS_PROD_OIDC_ROLE_ARN` if you wish to use a separate role with any of your environments.
> :information_source: Your deployment (and cleanup) scripts have to be able to cope with various environments, each with different application names, exposed routes, settings, ...
> Part of this complexity can be handled by the lookup policies described above (ex: one script per env) and also by using available environment variables:
>
> 1. [deployment context variables](#deployment-context-variables) provided by the template:
> * `${environment_type}`: the current environment type (`review`, `integration`, `staging` or `production`)
> * `${environment_name}`: the application name to use for the current environment (ex: `myproject-review-fix-bug-12` or `myproject-staging`)
> * `${hostname}`: the environment hostname, extracted from the current environment url (after late variable expansion - see below)
> 2. any [GitLab CI variable](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)
> 3. any [custom variable](https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project)
> (ex: `${SECRET_TOKEN}` that you have set in your project CI/CD variables)
#### Static vs. Dynamic environment URLs
### Environments URL management
The AWS template supports two ways of defining your environments url:
The AWS template supports two ways of providing your environments url:
* a **static way**: when you know your environments url in advance, probably because you're exposing your routes through a DNS you manage,
* a **static way**: when the environments url can be determined in advance, probably because you're exposing your routes through a DNS you manage,
* a [**dynamic way**](https://docs.gitlab.com/ee/ci/environments/#set-dynamic-environment-urls-after-a-job-finishes): when the url cannot be known before the
deployment job is executed.
The **static way** can be implemented simply by setting the appropriate configuration variable(s) depending on the environment (see environments configuration chapters below):
The **static way** can be implemented simply by setting the appropriate configuration variable(s) depending on the environment (see environments configuration chapters):
*`$AWS_ENVIRONMENT_URL` to define a default url pattern for all your envs,
*`$AWS_REVIEW_ENVIRONMENT_URL`, `$AWS_INTEG_ENVIRONMENT_URL`, `$AWS_STAGING_ENVIRONMENT_URL` and `$AWS_PROD_ENVIRONMENT_URL` to override the default.
> :information_source: Each of those variables support a late variable expansion mechanism with `%{somevar}` syntax, allowing you to use any dynamically evaluated variables such as `$environment_name`.
> :information_source: Each of those variables support a **late variable expansion mechanism** with the `%{somevar}` syntax,
> allowing you to use any dynamically evaluated variables such as `${environment_name}`.
>
> Example:
>
@@ -197,30 +180,52 @@ The **static way** can be implemented simply by setting the appropriate configur
To implement the **dynamic way**, your deployment script shall simply generate a `environment_url.txt` file, containing only
the dynamically generated url.
To implement the **dynamic way**, your deployment script shall simply generate a `environment_url.txt` file in the working directory, containing only
the dynamically generated url. When detected by the template, it will use it as the newly deployed environment url.
#### Deployment output variables
### Deployment output variables
Each deployment job produces _output variables_ that are propagated to downstream jobs (using [dotenv artifacts](https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#artifactsreportsdotenv)):
*`$environment_type`: set to the type of environment (`review`, `integration`, `staging` or `production`),
*`$environment_name`: the application name (see below),
* `$environment_url`: set to `$CI_ENVIRONMENT_URL`.
*`$environment_url`: set to the environment URL (whether determined statically or dynamically).
Those variables may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).
### Environments configuration
## Configuration reference
As seen above, the AWS template may support up to 4 predefined environments (`review`, `integration`, `staging` and `production`).
### Secrets management
Here are configuration details for each environment.
Here are some advices about your **secrets** (variables marked with a :lock:):
#### Review environments
1. Manage them as [project or group CI/CD variables](https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui):
*[**masked**](https://docs.gitlab.com/ee/ci/variables/#mask-a-custom-variable) to prevent them from being inadvertently
displayed in your job logs,
*[**protected**](https://docs.gitlab.com/ee/ci/variables/#protect-a-custom-variable) if you want to secure some secrets
you don't want everyone in the project to have access to (for instance production secrets).
2. In case a secret contains [characters that prevent it from being masked](https://docs.gitlab.com/ee/ci/variables/#masked-variable-requirements),
simply define its value as the [Base64](https://en.wikipedia.org/wiki/Base64) encoded value prefixed with `@b64@`:
it will then be possible to mask it and the template will automatically decode it prior to using it.
3. Don't forget to escape special characters (ex: `$` -> `$$`).
### Global configuration
The AWS template uses some global configuration used throughout all jobs and environments.
| `AWS_SCRIPTS_DIR` | Directory where AWS scripts (deploy & cleanup) are located | `.` _(root project dir)_ |
| `AWS_OIDC_ROLE_ARN` | Default IAM Role ARN associated with GitLab to [authenticate using OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/aws/) | _none_ (disabled) |
### Review environments configuration
Review environments are dynamic and ephemeral environments to deploy your _ongoing developments_ (a.k.a. _feature_ or
_topic_ branches).
@@ -236,7 +241,7 @@ Here are variables supported to configure review environments:
| `AWS_REVIEW_APP_NAME` | Application name for `review` env | `"${AWS_BASE_APP_NAME}-${CI_ENVIRONMENT_SLUG}"` (ex: `myproject-review-fix-bug-12`) |
| `AWS_REVIEW_ENVIRONMENT_URL`| The review environments url _(only define for static environment URLs declaration and if different from default)_ | `$AWS_ENVIRONMENT_URL` |
#### Integration environment
### Integration environment configuration
The integration environment is the environment associated to your integration branch (`develop` by default).
@@ -251,7 +256,7 @@ Here are variables supported to configure the integration environment:
| `AWS_INTEG_APP_NAME` | Application name for `integration` env | `${AWS_BASE_APP_NAME}-integration` |
| `AWS_INTEG_ENVIRONMENT_URL`| The integration environment url _(only define for static environment URLs declaration and if different from default)_ | `$AWS_ENVIRONMENT_URL` |
#### Staging environment
### Staging environment configuration
The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production
branch (`master` by default).
@@ -267,7 +272,7 @@ Here are variables supported to configure the staging environment:
| `AWS_STAGING_APP_NAME` | Application name for `staging` env | `${AWS_BASE_APP_NAME}-staging` |
| `AWS_STAGING_ENVIRONMENT_URL`| The staging environment url _(only define for static environment URLs declaration and if different from default)_ | `$AWS_ENVIRONMENT_URL` |
#### Production environment
### Production environment configuration
The production environment is the final deployment environment associated with your production branch (`master` by default).
@@ -324,9 +329,9 @@ stages:
##### `aws-deploy.sh`
This script is executed by the template to perform the application(s) deployment based on `aws` CLI, and uses [dynamic variables](#dynamic-variables) provided by the template (`${environment_name}` is used as the CloudFormation stack name).
This script is executed by the template to perform the application(s) deployment based on `aws` CLI, and uses [deployment context variables](#deployment-context-variables) provided by the template (`${environment_name}` is used as the CloudFormation stack name).
It implements [dynamic environment URLs](#static-vs-dynamic-environment-urls), by generating the `environment_url.txt` file, containing
It implements [dynamic environment URLs](#environments-url-management), by generating the `environment_url.txt` file, containing
the dynamically generated url at the end of the deployment script.
```bash
@@ -426,9 +431,9 @@ stages:
##### `aws-deploy.sh`
This script is executed by the template to perform the application(s) deployment based on `sam`and `aws` CLI, and uses [dynamic variables](#dynamic-variables) provided by the template (`${environment_name}` is used as the SAM/CloudFormation stack name).
This script is executed by the template to perform the application(s) deployment based on `sam`and `aws` CLI, and uses [deployment context variables](#deployment-context-variables) provided by the template (`${environment_name}` is used as the SAM/CloudFormation stack name).
It implements [dynamic environment URLs](#static-vs-dynamic-environment-urls), by generating the `environment_url.txt` file, containing
It implements [dynamic environment URLs](#environments-url-management), by generating the `environment_url.txt` file, containing
the dynamically generated url at the end of the deployment script.
