# GitLab CI template for Amazon Web Services Platform
This is a skeleton project for starting a new _to be continuous_ template.
This project implements a generic GitLab CI template for [Amazon Web Services](https://aws.amazon.com/) environments.
You shall fork it when you want to start developing a new template.
## Overview
Based on the kind of template (build, analyse, hosting, acceptance, ...), you should start working from one of the available `initial-xxx` branches, that each implement basic stuff.
This template implements continuous delivery/continuous deployment for projects hosted on AWS platforms.
It provides several features, usable in different modes.
### Review environments
The template supports **review** environments: those are dynamic and ephemeral environments to deploy your
_ongoing developments_ (a.k.a. _feature_ or _topic_ branches).
When enabled, it deploys the result from upstream build stages to a dedicated and temporary environment.
It is only active for non-production, non-integration branches.
It is a strict equivalent of GitLab's [Review Apps](https://docs.gitlab.com/ee/ci/review_apps/) feature.
It also comes with a _cleanup_ job (accessible either from the _environments_ page, or from the pipeline view).
### Integration environment
If you're using a Git Workflow with an integration branch (such as [Gitflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow)),
the template supports an **integration** environment.
When enabled, it deploys the result from upstream build stages to a dedicated environment.
It is only active for your integration branch (`develop` by default).
### Production environments
Lastly, the template supports 2 environments associated to your production branch (`master` by default):
* a **staging** environment (an iso-prod environment meant for testing and validation purpose),
* the **production** environment.
You're free to enable whichever or both, and you can also choose your deployment-to-production policy:
***continuous deployment**: automatic deployment to production (when the upstream pipeline is successful),
***continuous delivery**: deployment to production can be triggered manually (when the upstream pipeline is successful).
## Usage
### Include
In order to include this template in your project, add the following to your `gitlab-ci.yml`:
```yaml
include:
-project:'to-be-continuous/aws'
ref:'1.0.0'
file:'/templates/gitlab-ci-aws.yml'
```
### Global configuration
The AWS template uses some global configuration used throughout all jobs.
| `AWS_CLI_IMAGE` | the Docker image used to run AWS CLI commands| `amazon/aws-cli:latest`**it is highly recommended to set the CLI version compatible with your AWS server** |
| `AWS_BASE_APP_NAME` | Base application name | `$CI_PROJECT_NAME` ([see GitLab doc](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)) |
| `AWS_SCRIPTS_DIR` | Directory where AWS scripts (deploy & cleanup) are located | `.` _(root project dir)_ |
### Secrets management
Here are some advices about your **secrets** (variables marked with a :lock:):
1. Manage them as [project or group CI/CD variables](https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui):
*[**masked**](https://docs.gitlab.com/ee/ci/variables/#mask-a-custom-variable) to prevent them from being inadvertently
displayed in your job logs,
*[**protected**](https://docs.gitlab.com/ee/ci/variables/#protect-a-custom-variable) if you want to secure some secrets
you don't want everyone in the project to have access to (for instance production secrets).
2. In case a secret contains [characters that prevent it from being masked](https://docs.gitlab.com/ee/ci/variables/#masked-variable-requirements),
simply define its value as the [Base64](https://en.wikipedia.org/wiki/Base64) encoded value prefixed with `@b64@`:
it will then be possible to mask it and the template will automatically decode it prior to using it.
3. Don't forget to escape special characters (ex: `$` -> `$$`).
### Deployment and cleanup scripts
The GitLab CI template for AWS requires you to provide a shell script that fully implements your application
deployment and cleanup using the [`aws` CLI](https://aws.amazon.com/cli/) and all other tools available in the selected Docker image.
#### Lookup policy
The deployment script is searched as follows:
1. look for a specific `aws-deploy-$env.sh` in the `$AWS_SCRIPTS_DIR` directory in your project (e.g. `aws-deploy-staging.sh` for staging environment),
2. if not found: look for a default `aws-deploy.sh` in the `$AWS_SCRIPTS_DIR` directory in your project,
3. if not found: the deployment job will fail.
The cleanup script is searched as follows:
1. look for a specific `aws-cleanup-$env.sh` in the `$AWS_SCRIPTS_DIR` directory in your project (e.g. `aws-cleanup-staging.sh` for staging environment),
2. if not found: look for a default `aws-cleanup.sh` in the `$AWS_SCRIPTS_DIR` directory in your project,
3. if not found: the cleanup job will fail.
#### Dynamic Variables
You have to be aware that your deployment (and cleanup) scripts have to be able to cope with various environments
(`review`, `integration`, `staging` and `production`), each with different application names, exposed routes, settings, ...
Part of this complexity can be handled by the lookup policies described above (ex: one resource per env).
In order to be able to implement some **genericity** in your scripts and templates, you should use available environment variables:
1. any [GitLab CI variable](https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables)
(ex: `${CI_ENVIRONMENT_URL}` to retrieve the actual environment exposed route)
2. any [custom variable](https://docs.gitlab.com/ee/ci/variables/#custom-environment-variables)
(ex: `${SECRET_TOKEN}` that you have set in your project CI/CD variables)
3.**dynamic variables** set by the template:
*`${appname}`: the application target name to use in this environment (ex: `myproject-review-fix-bug-12` or `myproject-staging`)
*`${env}`: the environment type (`review`, `integration`, `staging` or `production`)
*`${hostname}`: the environment hostname, extracted from `${CI_ENVIRONMENT_URL}` (has to be explicitly declared as [`environment:url`](https://docs.gitlab.com/ee/ci/yaml/#environmenturl) in your `.gitlab-ci.yml` file)
#### AWS authentication
The AWS template **does not manage AWS authentication**.
That means you'll have to manage AWS authentication by yourself, according to the `aws` CLI configuration options (configuration file, CLI options, environment variables).
For credentials management, we strongly advise to use [environment variables configuration](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list), managed as GitLab CI secret variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, possibly `AWS_ROLE_ARN`).
If you have to manage different set of authentication credentials depending on managed environments, you shall either use [GitLab scoped variables](https://docs.gitlab.com/ee/ci/environments/index.html#scoping-environments-with-specs) or our [scoped variables syntax](https://to-be-continuous.gitlab.io/doc/usage/#scoped-variables) to limit/override some variables values, using `$CI_ENVIRONMENT_NAME` as the conditional variable.
Example: different credentials for production (declared as project variables)
The AWS template supports two ways of defining your environments url:
* a **static way**: when you know your environments url in advance, probably because you're exposing your routes through a DNS you manage,
* a [**dynamic way**](https://docs.gitlab.com/ee/ci/environments/#set-dynamic-environment-urls-after-a-job-finishes): when the url cannot be known before the
deployment job is executed.
The static way can be implemented simply by setting the appropriate configuration variables depending on the environments (see environments configuration chapters below):
*`$AWS_REVIEW_ENVIRONMENT_SCHEME` and`$AWS_REVIEW_ENVIRONMENT_DOMAIN` for the review environments,
*`$AWS_INTEG_ENVIRONMENT_URL`, `$AWS_STAGING_ENVIRONMENT_URL` and `$AWS_PROD_ENVIRONMENT_URL` for others.
To implement the dynamic way, your deployment script shall simply generate a `environment_url.txt` file, containing only
the dynamically generated url.
### Environments configuration
As seen above, the AWS template may support up to 4 environments (`review`, `integration`, `staging` and `production`).
Each deployment job produces _output variables_ that are propagated to downstream jobs (using [dotenv artifacts](https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#artifactsreportsdotenv)):
*`environment_type`: set to the type of environment (`review`, `integration`, `staging` or `production`),
*`environment_name`: the application name (see below),
*`environment_url`: set to `$CI_ENVIRONMENT_URL`.
They may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).
Here are configuration details for each environment.
#### Review environments
Review environments are dynamic and ephemeral environments to deploy your _ongoing developments_ (a.k.a. _feature_ or
_topic_ branches).
They are **disabled by default** and can be enabled by setting the `AWS_REVIEW_ENABLED` variable (see below).
Here are variables supported to configure review environments:
| `AWS_REVIEW_ENABLED` | AWS project ID for `review` env | _none_ (disabled) |
| `AWS_REVIEW_APP_NAME` | Application name for `review` env | `"${AWS_BASE_APP_NAME}-${CI_ENVIRONMENT_SLUG}"` (ex: `myproject-review-fix-bug-12`) |
| `AWS_REVIEW_ENVIRONMENT_SCHEME`| The review environment protocol scheme | `https` |
| `AWS_REVIEW_ENVIRONMENT_DOMAIN`| The review environment domain. | _none_ |
Note: By default, review `environment.url` will be built as `${AWS_REVIEW_ENVIRONMENT_SCHEME}://${$CI_PROJECT_NAME}-${CI_ENVIRONMENT_SLUG}.${AWS_REVIEW_ENVIRONMENT_DOMAIN}`
#### Integration environment
The integration environment is the environment associated to your integration branch (`develop` by default).
It is **disabled by default** and can be enabled by setting the `AWS_INTEG_ENABLED` variable (see below).
Here are variables supported to configure the integration environment:
| `AWS_INTEG_ENABLED` | AWS project ID for `integration` env | _none_ (disabled) |
| `AWS_INTEG_APP_NAME` | Application name for `integration` env | `${AWS_BASE_APP_NAME}-integration` |
| `AWS_INTEG_ENVIRONMENT_URL`| The integration environment url **including scheme** (ex: `https://my-application-integration.compute-1.amazonaws.com`). Do not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that. | _none_ |
#### Staging environment
The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production
branch (`master` by default).
It is **disabled by default** and can be enabled by setting the `AWS_STAGING_ENABLED` variable (see below).
Here are variables supported to configure the staging environment:
| `AWS_STAGING_ENABLED` | AWS project ID for `staging` env | _none_ (disabled) |
| `AWS_STAGING_APP_NAME` | Application name for `staging` env | `${AWS_BASE_APP_NAME}-staging` |
| `AWS_STAGING_ENVIRONMENT_URL` | The staging environment url **including scheme** (ex: `https://my-application-staging.compute-1.amazonaws.com`). Do not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that. | _none_ |
#### Production environment
The production environment is the final deployment environment associated with your production branch (`master` by default).
It is **disabled by default** and can be enabled by setting the `AWS_PROD_ENABLED` variable (see below).
Here are variables supported to configure the production environment:
| `AWS_PROD_ENABLED` | AWS project ID for `production` env | _none_ (disabled) |
| `AWS_PROD_APP_NAME` | Application name for `production` env | `$AWS_BASE_APP_NAME` |
| `AWS_PROD_ENVIRONMENT_URL`| The production environment url **including scheme** (ex: `https://my-application.compute-1.amazonaws.com`) Do not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that. | _none_ |
| `AUTODEPLOY_TO_PROD` | Set this variable to auto-deploy to production. If not set deployment to production will be `manual` (default behaviour). | _none_ (disabled) |
## Examples
### AWS CloudFormation stack
This chapter gives some implementation hints to implement continuous deployment based on [AWS CloudFormation](https://aws.amazon.com/cloudformation/).
It enables review, staging and production environments.
#### `.gitlab-ci.yml`
```yaml
include:
# Include AWS template
-project:'to-be-continuous/aws'
ref:'1.0.0'
file:'/templates/gitlab-ci-aws.yml'
...
# Global variables
variables:
# AWS
# AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY defined as secret CI/CD variable
AWS_REVIEW_ENABLED:"true"# enable review env
AWS_STAGING_ENABLED:"true"# enable staging env
AWS_PROD_ENABLED:"true"# enable production env
...
# Pipeline steps
stages:
-build
-test
-deploy
-acceptance
-publish
-production
```
#### hook scripts
##### `aws-deploy.sh`
This script is executed by the template to perform the application(s) deployment based on `aws` CLI, and uses dynamic variables provided by the template.
It implements [dynamic environment URLs](#static-vs-dynamic-environment-urls), by generating the `environment_url.txt` file, containing
the dynamically generated url at the end of the deployment script.
```bash
#!/usr/bin/env bash
echo"[aws-deploy] Deploying $appname..."
template_file=file://MyStack.template
# retrieve environment type ($env) from template
# retrieve $AWS_KEYPAIR_NAME from project secret variables
