feat: promote (pseudo-standard) $environment_type & $environment_name variables

It uses [s3cmd](https://github.com/s3tools/s3cmd) to control the S3 API endpoint and uploading objects.

## Overview

## Usage

In order to include this template in your project, add the following to your `gitlab-ci.yml`:

```yaml

include:

  - project: 'to-be-continuous/s3'

    ref: '5.1.0'

    file: '/templates/gitlab-ci-s3.yml'

```

## Understand

This chapter introduces key notions and principle to understand how this template works.

### Managed deployment environments

This template implements continuous delivery/continuous deployment for projects hosted on S3 platforms.

It provides several features, usable in different modes.

It allows you to manage automatic deployment & cleanup of standard predefined environments. Each environment can be enabled/disabled by configuration. If you're not satisfied with predefined environments and/or their associated Git workflow, you may implement you own environments and workflow, by reusing/extending the base (hidden) jobs. This is advanced usage and will not be covered by this documentation.

The following chapters present the managed predefined environments and their associated Git workflow.

### Review environments

#### Review environments

The template supports **review** environments: those are dynamic and ephemeral environments to deploy your

_ongoing developments_ (a.k.a. _feature_ or _topic_ branches).

It also comes with a _cleanup_ job (accessible either from the _environments_ page, or from the pipeline view).

### Integration environment

#### Integration environment

If you're using a Git Workflow with an integration branch (such as [Gitflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow)),

the template supports an **integration** environment.

When enabled, it deploys the result from upstream build stages to a dedicated environment.

It is only active for your integration branch (`develop` by default).

### Production environments

#### Production environments

Lastly, the template supports 2 environments associated to your production branch (`master` by default):

* **continuous deployment**: automatic deployment to production (when the upstream pipeline is successful),

* **continuous delivery**: deployment to production can be triggered manually (when the upstream pipeline is successful).

## Usage

### Using other S3-compatible systems than AWS

### Include

The template might be used with other storage systems provided they are implementing a compatible API. 

In that case, you'll have to override the default `$S3_ENDPOINT_HOST` and `$S3_WEBSITE_ENDPOINT`variables.

In order to include this template in your project, add the following to your `gitlab-ci.yml`:

| Provider               | `S3_ENDPOINT_HOST`   | `S3_WEBSITE_ENDPOINT` |

| ---------------------- | -------------------- | --------------------- |

| [Google Cloud Platform](https://cloud.google.com/storage/docs/interoperability) | `storage.googleapis.com` | _website hosting in GCP not supported by `s3cmd`_ |

| Microsoft Azure | requires using [Minio](https://min.io/)<br/>Read [this article](https://cloudblogs.microsoft.com/opensource/2017/11/09/s3cmd-amazon-s3-compatible-apps-azure-storage/) for further information | N/A |

| [Flexible Engine (Orange Business Services)](https://docs.prod-cloud-ocb.orange-business.com/endpoint/) | `oss.<region>.prod-cloud-ocb.orange-business.com` (:warning: `<region>` must be set) | `https://%(bucket)s.oss-website.%(location)s.prod-cloud-ocb.orange-business.com` |

### Buckets namespacing

In its default configuration, the template manages (create/sync/delete) one S3 bucket per environment.

But you may also configure it to implement other policies. Here are several examples of alternate policies.

#### A single bucket with separate prefixes for each env

Here the `.gitlab-ci.yml` configuration for one shared bucket for all envs, each separated by prefix:

```yaml

include:

  - project: 'to-be-continuous/s3'

    ref: '5.1.0'

    file: '/templates/gitlab-ci-s3.yml'

variables:

  # use same bucket for all envs

  S3_REVIEW_BUCKET_NAME: "acme-bucket-shared"

  S3_INTEG_BUCKET_NAME: "acme-bucket-shared"

  S3_STAGING_BUCKET_NAME: "acme-bucket-shared"

  S3_PROD_BUCKET_NAME: "acme-bucket-shared"

  # segregate envs with prefixes

  S3_PREFIX: "$CI_ENVIRONMENT_SLUG"

```

#### Hybrid policy

Here the `.gitlab-ci.yml` configuration for one shared bucket for review envs and separate buckets for others:

```yaml

variables:

  # use same bucket for all review envs

  S3_REVIEW_BUCKET_NAME: "acme-bucket-review"

  S3_INTEG_BUCKET_NAME: "acme-bucket-integ"

  S3_STAGING_BUCKET_NAME: "acme-bucket-staging"

  S3_PROD_BUCKET_NAME: "acme-bucket-prod"

  # segregate review envs with prefixes

  S3_REVIEW_PREFIX: "$CI_ENVIRONMENT_SLUG"

```

### Deployment output variables

As seen above, the S3 template may support up to 4 environments (`review`, `integration`, `staging` and `production`).

Each deployment job produces _output variables_ that are propagated to downstream jobs (using [dotenv artifacts](https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#artifactsreportsdotenv)):

* `environment_type`: set to the type of environment (`review`, `integration`, `staging` or `production`),

* `environment_name`: the application name (see below),

* `environment_url`: set to `$CI_ENVIRONMENT_URL`.

They may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).

## Configuration reference

### Secrets management

Here are some advices about your **secrets** (variables marked with a :lock:): 

1. Manage them as [project or group CI/CD variables](https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project):

    * [**masked**](https://docs.gitlab.com/ee/ci/variables/#mask-a-cicd-variable) to prevent them from being inadvertently

      displayed in your job logs,

    * [**protected**](https://docs.gitlab.com/ee/ci/variables/#protected-cicd-variables) if you want to secure some secrets

      you don't want everyone in the project to have access to (for instance production secrets).

2. In case a secret contains [characters that prevent it from being masked](https://docs.gitlab.com/ee/ci/variables/#mask-a-cicd-variable),

  simply define its value as the [Base64](https://en.wikipedia.org/wiki/Base64) encoded value prefixed with `@b64@`:

  it will then be possible to mask it and the template will automatically decode it prior to using it.

3. Don't forget to escape special characters (ex: `$` -> `$$`).

### Global configuration

The S3 template uses some global configuration used throughout all jobs.

| `S3_PREFIX`            | Default S3 prefix to use as a root destination to upload objects in the S3 bucket | _none_ |

| `S3_SCRIPTS_DIR`       | Directory where S3 hook scripts are located   | `.`|

#### Using other S3-compatible systems than AWS

### Deployment jobs

The template might be used with other storage systems provided they are implementing a compatible API. 

In that case, you'll have to override the default `$S3_ENDPOINT_HOST` and `$S3_WEBSITE_ENDPOINT`variables.

Each environment has its own deployment job (associated with the right branch).

| Provider               | `S3_ENDPOINT_HOST`   | `S3_WEBSITE_ENDPOINT` |

| ---------------------- | -------------------- | --------------------- |

| [Google Cloud Platform](https://cloud.google.com/storage/docs/interoperability) | `storage.googleapis.com` | _website hosting in GCP not supported by `s3cmd`_ |

| Microsoft Azure | requires using [Minio](https://min.io/)<br/>Read [this article](https://cloudblogs.microsoft.com/opensource/2017/11/09/s3cmd-amazon-s3-compatible-apps-azure-storage/) for further information | N/A |

| [Flexible Engine (Orange Business Services)](https://docs.prod-cloud-ocb.orange-business.com/endpoint/) | `oss.<region>.prod-cloud-ocb.orange-business.com` (:warning: `<region>` must be set) | `https://%(bucket)s.oss-website.%(location)s.prod-cloud-ocb.orange-business.com` |

It uses the following variables:

### Secrets management

| Name                   | description                                   | default value     |

| ---------------------- | --------------------------------------------- | ----------------- |

| `S3_DEPLOY_ARGS`       | [s3cmd](https://s3tools.org/usage) command and options to deploy files to the bucket | `sync --recursive --delete-removed --acl-public --no-mime-magic --guess-mime-type` |

| `S3_DEPLOY_FILES`      | Pattern(s) of files to deploy to the S3 bucket| `public/` _(all files from `public` directory)_ |

| `S3_WEBSITE_DISABLED`  | Set to `true` to disable WebSite hosting by your S3 bucket    | _none_ (enabled by default) |

| `S3_WEBSITE_ARGS`      | [s3cmd](https://s3tools.org/usage) command and options to enable WebSite hosting on the bucket | `ws-create --ws-index=index.html --ws-error=404.html` |

| `S3_WEBSITE_ENDPOINT`  | Default WebSite endpoint url pattern (supports `%(bucket)s` and `%(location)s` placeholders).<br/>_only required when website hosting is not disabled_ | `http://%(bucket)s.s3-website.%(location)s.amazonaws.com` |

Here are some advices about your **secrets** (variables marked with a :lock:): 

If need be you could add your own hook script `s3-pre-deploy.sh` that will be triggered right before deploying files to

the S3 bucket.

1. Manage them as [project or group CI/CD variables](https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project):

    * [**masked**](https://docs.gitlab.com/ee/ci/variables/#mask-a-cicd-variable) to prevent them from being inadvertently

      displayed in your job logs,

    * [**protected**](https://docs.gitlab.com/ee/ci/variables/#protected-cicd-variables) if you want to secure some secrets

      you don't want everyone in the project to have access to (for instance production secrets).

2. In case a secret contains [characters that prevent it from being masked](https://docs.gitlab.com/ee/ci/variables/#mask-a-cicd-variable),

  simply define its value as the [Base64](https://en.wikipedia.org/wiki/Base64) encoded value prefixed with `@b64@`:

  it will then be possible to mask it and the template will automatically decode it prior to using it.

3. Don't forget to escape special characters (ex: `$` -> `$$`).

If the target bucket doesn't appear to exist, the template tries to create it.

### Environments configuration

#### `s3-cleanup-review` job

As seen above, the S3 template may support up to 4 environments (`review`, `integration`, `staging` and `production`).

This job allows destroying each review environment. Simply deletes the associated objects in the bucket.

After objects removal, if the bucket appears to be empty, also tries to delete the bucket.

Each deployment job produces _output variables_ that are propagated to downstream jobs (using [dotenv artifacts](https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#artifactsreportsdotenv)):

#### `s3-cleanup-all-review` job

* `environment_type`: set to the type of environment (`review`, `integration`, `staging` or `production`),

* `environment_name`: the application name (see below),

* `environment_url`: set to `$CI_ENVIRONMENT_URL`.

This job allows destroying all review environments at once (in order to save cloud resources).

They may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).

It is **disabled by default** and can be controlled using the `$CLEANUP_ALL_REVIEW` variable:

Here are configuration details for each environment.

1. automatically executed if `$CLEANUP_ALL_REVIEW` set to `force`,

2. manual job enabled from any `master` branch pipeline if `$CLEANUP_ALL_REVIEW` set to `true` (or any other value),

#### Review environments

The first value `force` can be used in conjunction with a [scheduled](https://docs.gitlab.com/ee/ci/pipelines/schedules.html)

pipeline to cleanup cloud resources for instance everyday at 6pm or on friday evening.

The second one simply enables the (manual) cleanup job on the `master` branch pipeline.

Anyway destroyed review environments will be automatically re-created the next time a developer pushes a new commit on a

feature branch.

:warning: in case of scheduling the cleanup, you'll probably have to create an almost empty branch without any other

template (no need to build/test/analyse your code if your only goal is to cleanup environments).

### Review environments configuration

Review environments are dynamic and ephemeral environments to deploy your _ongoing developments_ (a.k.a. _feature_ or 

_topic_ branches).

Note: By default, review `environment.url` will be built as `${S3_REVIEW_ENVIRONMENT_SCHEME}://${$CI_PROJECT_NAME}-${CI_ENVIRONMENT_SLUG}.${S3_REVIEW_ENVIRONMENT_DOMAIN}`

#### Integration environment

### Integration environment configuration

The integration environment is the environment associated to your integration branch (`develop` by default).

| `S3_INTEG_BUCKET_NAME`   | Bucket name for `integration` env | `${S3_BASE_BUCKET_NAME}-integration` |

| `S3_INTEG_PREFIX`  |  S3 prefix to use for `integration` env _(only define if different from default)_ | `S3_PREFIX` |

#### Staging environment

### Staging environment configuration

The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production 

branch (`master` by default).

| `S3_STAGING_BUCKET_NAME` | Bucket name for `staging` env     | `${S3_BASE_BUCKET_NAME}-staging` |

| `S3_STAGING_PREFIX`  |  S3 prefix to use for `staging` env _(only define if different from default)_ | `S3_PREFIX` |

#### Production environment

### Production environment configuration

The production environment is the final deployment environment associated with your production branch (`master` by default).

| `S3_PROD_DEPLOY_STRATEGY`| Defines the deployment to production strategy. One of `manual` (i.e. _one-click_) or `auto`. | `manual` |

| `S3_PROD_PREFIX`  |  S3 prefix to use for `production` env _(only define if different from default)_ | `S3_PREFIX` |

### Buckets namespacing

In its default configuration, the template manages (create/sync/delete) one S3 bucket per environment.

But you may also configure it to implement other policies. Here are several examples of alternate policies.

#### A single bucket with separate prefixes for each env

Here the `.gitlab-ci.yml` configuration for one shared bucket for all envs, each separated by prefix:

```yaml

variables:

  # use same bucket for all envs

  S3_REVIEW_BUCKET_NAME: "acme-bucket-shared"

  S3_INTEG_BUCKET_NAME: "acme-bucket-shared"

  S3_STAGING_BUCKET_NAME: "acme-bucket-shared"

  S3_PROD_BUCKET_NAME: "acme-bucket-shared"

  # segregate envs with prefixes

  S3_PREFIX: "$CI_ENVIRONMENT_SLUG"

```

#### Hybrid policy

Here the `.gitlab-ci.yml` configuration for one shared bucket for review envs and separate buckets for others:

```yaml

variables:

  # use same bucket for all review envs

  S3_REVIEW_BUCKET_NAME: "acme-bucket-review"

  S3_INTEG_BUCKET_NAME: "acme-bucket-integ"

  S3_STAGING_BUCKET_NAME: "acme-bucket-staging"

  S3_PROD_BUCKET_NAME: "acme-bucket-prod"

  # segregate review envs with prefixes

  S3_REVIEW_PREFIX: "$CI_ENVIRONMENT_SLUG"

```

### Deployment jobs

Each environment has its own deployment job (associated with the right branch).

It uses the following variables:

| Name                   | description                                   | default value     |

| ---------------------- | --------------------------------------------- | ----------------- |

| `S3_DEPLOY_ARGS`       | [s3cmd](https://s3tools.org/usage) command and options to deploy files to the bucket | `sync --recursive --delete-removed --acl-public --no-mime-magic --guess-mime-type` |

| `S3_DEPLOY_FILES`      | Pattern(s) of files to deploy to the S3 bucket| `public/` _(all files from `public` directory)_ |

| `S3_WEBSITE_DISABLED`  | Set to `true` to disable WebSite hosting by your S3 bucket    | _none_ (enabled by default) |

| `S3_WEBSITE_ARGS`      | [s3cmd](https://s3tools.org/usage) command and options to enable WebSite hosting on the bucket | `ws-create --ws-index=index.html --ws-error=404.html` |

| `S3_WEBSITE_ENDPOINT`  | Default WebSite endpoint url pattern (supports `%(bucket)s` and `%(location)s` placeholders).<br/>_only required when website hosting is not disabled_ | `http://%(bucket)s.s3-website.%(location)s.amazonaws.com` |

If need be you could add your own hook script `s3-pre-deploy.sh` that will be triggered right before deploying files to

the S3 bucket.

If the target bucket doesn't appear to exist, the template tries to create it.

### `s3-cleanup-review` job

This job allows destroying each review environment. Simply deletes the associated objects in the bucket.

After objects removal, if the bucket appears to be empty, also tries to delete the bucket.

### `s3-cleanup-all-review` job

This job allows destroying all review environments at once (in order to save cloud resources).

It is **disabled by default** and can be controlled using the `$CLEANUP_ALL_REVIEW` variable:

1. automatically executed if `$CLEANUP_ALL_REVIEW` set to `force`,

2. manual job enabled from any `master` branch pipeline if `$CLEANUP_ALL_REVIEW` set to `true` (or any other value),

The first value `force` can be used in conjunction with a [scheduled](https://docs.gitlab.com/ee/ci/pipelines/schedules.html)

pipeline to cleanup cloud resources for instance everyday at 6pm or on friday evening.

The second one simply enables the (manual) cleanup job on the `master` branch pipeline.

Anyway destroyed review environments will be automatically re-created the next time a developer pushes a new commit on a

feature branch.

:warning: in case of scheduling the cleanup, you'll probably have to create an almost empty branch without any other

template (no need to build/test/analyse your code if your only goal is to cleanup environments).

## Variants

### Vault variant

  # upload/sync files to bucket

  function deploy() {

    export env=$1

    export environment_type=$1

    # backward compatibility

    export env=$environment_type

    bucket=$2

    prefix=$3

    # normalize prefix with leading slash

    fi

    region=$4

    log_info "--- \\e[32mdeploy\\e[0m (env: \\e[33;1m${env}\\e[0m)"

    log_info "--- bucket: \\e[33;1m${bucket}\\e[0m"

    log_info "--- region: \\e[33;1m${region}\\e[0m"

    log_info "--- prefix: \\e[33;1m${prefix:-(none)}\\e[0m"

    log_info "--- env: \\e[33;1m${env}\\e[0m"

    log_info "--- \\e[32mdeploy\\e[0m"

    log_info "--- \$environment_type: \\e[33;1m${environment_type}\\e[0m"

    log_info "--- \$bucket: \\e[33;1m${bucket}\\e[0m"

    log_info "--- \$region: \\e[33;1m${region}\\e[0m"

    log_info "--- \$prefix: \\e[33;1m${prefix:-(none)}\\e[0m"

    # unset any upstream deployment env & artifacts

    rm -f s3.env

    rm -f environment_url.txt

    # maybe create bucket

    if s3cmd info "s3://${bucket}" >/dev/null 2>&1

      environment_url="$website_url$prefix"

      # finally persist environment url

      echo "$environment_url" > environment_url.txt

      echo -e "environment_type=$env\\nenvironment_name=$bucket\\nenvironment_url=$environment_url" > s3.env

      echo -e "environment_type=$environment_type\\nenvironment_name=$bucket\\nenvironment_url=$environment_url" > s3.env

    fi

  }

  # delete application

  function delete() {

    export env=$1

    export environment_type=$1

    bucket=$2

    prefix=$3

    # normalize prefix with leading slash

      prefix="/$prefix"

    fi

    log_info "--- \\e[32mdelete\\e[0m (env: ${env})"

    log_info "--- \\e[32mdelete\\e"

    log_info "--- \$environment_type: \\e[33;1m${environment_type}\\e[0m"

    log_info "--- bucket: \\e[33;1m${bucket}\\e[0m"

    log_info "--- prefix: \\e[33;1m${prefix:-(none)}\\e[0m"

    log_info "--- env: \\e[33;1m${env}\\e[0m"

    # check if bucket exists

    if ! s3cmd info "s3://${bucket}" >/dev/null

  }

  function delete_all() {

    export env=$1

    appnameproto=$2

    export environment_type=$1

    envnameproto=$2

    prefix=$3

    # make appname regex by replacing $CI_COMMIT_REF_SLUG with .*

    appnameregex=$(echo "$appnameproto" | sed -r "s/$CI_COMMIT_REF_SLUG/.*/g")

    # make environment_name regex by replacing $CI_COMMIT_REF_SLUG with .*

    envnameregex=$(echo "$envnameproto" | sed -r "s/$CI_COMMIT_REF_SLUG/.*/g")

    # list buckets | extract 3rd col | strip 's3://' | filter buckets with "$regex"

    matchingbuckets=$(s3cmd ls | awk '{print $3}' | cut -d'/' -f3 | awk "/$appnameregex/")

    matchingbuckets=$(s3cmd ls | awk '{print $3}' | cut -d'/' -f3 | awk "/$envnameregex/")

    matchcount=$(echo "$matchingbuckets" | wc -w)

    log_info "--- \\e[32mdelete all\\e[0m (env: \\e[33;1m${env}\\e[0m, bucket matcher: \\e[33;1m${appnameregex}\\e[0m, prefix: \\e[33;1m${prefix:-(none)}\\e[0m): \\e[33;1m${matchcount}\\e[0m buckets found"

    log_info "--- \\e[32mdelete all"

    log_info "--- \$environment_type: \\e[33;1m${environment_type}\\e[0m"

    log_info "--- bucket matcher: \\e[33;1m${envnameregex}\\e[0m (\\e[33;1m${matchcount}\\e[0m buckets found)"

    log_info "--- prefix: \\e[33;1m${prefix:-(none)}\\e[0m"

    rc=0

    for bckt in $matchingbuckets

    do

      echo -e "\\e[1;93m-------------------------------------------------------------------------------\\e[0m"

      if ! delete "$env" "$bckt" "$prefix"

      if ! delete "$environment_type" "$bckt" "$prefix"

      then

        log_warn "... failed deleting bucket \\e[33;1m${bckt}\\e[0m (see logs)"

        rc=1