Commit e99528a2 authored by Pierre Smeyers's avatar Pierre Smeyers
Browse files

Merge branch 'initial' into 'main' 

initial docker-compose template

See merge request to-be-continuous/docker-compose!2
parents 86d0d6c7 df084959

README.md

+153 −58
Original line number Diff line number Diff line
@@ -19,9 +19,9 @@ include: 
    inputs:

      # ⚠ this is only an example

      base-app-name: wonderapp

      review-project: "prj-12345" # enable review env

      staging-project: "prj-12345" # enable staging env

      prod-project: "prj-12345" # enable production env

      review-docker-host: "ssh://docker@192.168.64.5:2375" # enable review env

      staging-docker-host: "ssh://docker@192.168.64.6:2375" # enable staging env

      prod-docker-host: "ssh://docker@192.168.64.7:2375" # enable production env

```



### Use as a CI/CD template (legacy)
@@ -39,9 +39,9 @@ variables: 
  # 2: set/override template variables

  # ⚠ this is only an example

  DCMP_BASE_APP_NAME: wonderapp

  DCMP_REVIEW_PROJECT: "prj-12345" # enable review env

  DCMP_STAGING_PROJECT: "prj-12345" # enable staging env

  DCMP_PROD_PROJECT: "prj-12345" # enable production env

  DCMP_REVIEW_DOCKER_HOST: "ssh://docker@192.168.64.5:2375" # enable review env

  DCMP_STAGING_DOCKER_HOST: "ssh://docker@192.168.64.6:2375" # enable staging env

  DCMP_PROD_DOCKER_HOST: "ssh://docker@192.168.64.7:2375" # enable production env

```



## Understand
@@ -93,18 +93,22 @@ You're free to enable whichever or both, and you can also choose your deployment 


### Supported authentication methods



The Docker Compose template supports the following authentication methods:

The Docker Compose template supports [deployment on remote Docker hosts](https://www.docker.com/blog/how-to-deploy-on-remote-docker-hosts-with-docker-compose/), using 

dedicated variables:



* TODO (document)

* `DCMP_REVIEW_DOCKER_HOST`, `DCMP_INTEG_DOCKER_HOST`, `DCMP_STAGING_DOCKER_HOST` and `DCMP_PROD_DOCKER_HOST` to both enable and configure the target Docker host for each environment (ex: `ssh://docker@192.168.64.5:2375`),

* :lock: `DCMP_SSH_PRIVATE_KEY`, :lock: `DCMP_REVIEW_SSH_PRIVATE_KEY`, :lock: `DCMP_INTEG_SSH_PRIVATE_KEY`, :lock: `DCMP_STAGING_SSH_PRIVATE_KEY` and :lock: `DCMP_PROD_SSH_PRIVATE_KEY` to provide the global or per env SSH private key (in case SSH authentication is required).



### Deployment context variables



In order to manage the various deployment environments, this template provides a couple of **dynamic variables**

that you might use in your hook scripts, deployment manifests and other deployment resources:

that you might use in your hook scripts or [Docker Compose files](https://docs.docker.com/compose/compose-application-model/#the-compose-file):



* `${environment_type}`: the current deployment environment type (`review`, `integration`, `staging` or `production`)

* `${environment_name}`: a generated application name to use for the current deployment environment (ex: `myapp-review-fix-bug-12` or `myapp-staging`) - _details below_



> :information_source: the `${environment_name}` is used by the Docker Compose template as the [Docker Compose project name](https://docs.docker.com/compose/project-name/).



#### Generated environment name



The `${environment_name}` variable is generated to designate each deployment environment with a unique and meaningful application name.
@@ -131,38 +135,116 @@ Examples (with an application's base name `myapp`): 


### Deployment and cleanup



> TODO: explain here the supported techniques to deploy and cleanup the environments.

The Docker Compose template requires you to provide:



* a _required_ [Compose file](https://docs.docker.com/compose/compose-application-model/#the-compose-file) that implements your application deployment and cleanup,

* _optional_ [`.override.` Compose files](https://docs.docker.com/compose/multiple-compose-files/merge/) defining common and/or per-env override,

* _optional_ [dotenv files](https://docs.docker.com/compose/environment-variables/env-file/) defining common and/or per-env configuration,

* _optional_ hook scripts (shell) to implement logic that can't be implemented with Docker Compose.



#### Compose files lookup strategy



Unless you have explicitly set the [`COMPOSE_FILES`](https://docs.docker.com/compose/environment-variables/envvars/#compose_file) variable, the Docker Compose template will handle it and 

implement the following [Compose file(s)](https://docs.docker.com/compose/compose-application-model/#the-compose-file) lookup strategy:



| Lookup order | Compose file | Additional `.override.` files (optional)                                                                                           |

|:------------:|--------------|------------------------------------------------------------------------------------------------------------------------------------------|

|      1.      | **environment specific** (`compose-${environment_type}.yaml`) | **environment specific** override (`compose-${environment_type}.override.yaml`)                                                          |

|      2.      | **default** (`compose.yaml`) | 1. **default** override (`compose.override.yaml`)<br/>2. **environment specific** override (`compose-${environment_type}.override.yaml`) |



> :information_source: Important:

>

> You should also explained clearly what is expected from the template user and what is the lookup policy in case the template

> implements one.

> * Compose file base name `docker-compose` is also supported as an alternative to `compose`,

> * Compose file extension `.yml` is also supported as an alternative to `.yaml`,

> * `.override.` files must match the same base name (`compose` or `docker-compose`) and extension (`yaml` or `yml`) as the found Compose file.



Examples with different combinations of files:



<table>

  <thead>

    <tr>

      <th>Files in your project</th>

      <th>Compose files for Review</th>

      <th>Compose files for Integration</th>

      <th>Compose files for Staging</th>

      <th>Compose files for Production</th>

    </tr>

  </thead>

  <tfoot>

    <tr>

      <td>- <code>compose.yaml</code></td>

      <td colspan="4" align=center><code>compose.yaml</code></td>

    </tr>

    <tr>

      <td>- <code>compose.yaml</code><br/>- <code>compose-production.yaml<code></td>

      <td colspan="3" align=center><code>compose.yaml</code></td>

      <td align=center><code>compose-production.yaml</code></td>

    </tr>

    <tr>

      <td>- <code>docker-compose.yml</code><br/>- <code>docker-compose-production.override.yml<code></td>

      <td colspan="3" align=center><code>docker-compose.yml</code></td>

      <td align=center><code>docker-compose.yml</code> + <code>docker-compose-production.override.yml</code> (merged)</td>

    </tr>

    <tr>

      <td>- <code>compose.yml</code><br/>- <code>compose.override.yml</code><br/>- <code>compose-review.override.yml</code><br/>- <code>compose-production.yml</code></td>

      <td align=center><code>compose.yml</code> + <code>compose.override.yml</code> + <code>compose-review.override.yml</code> (merged)</td>

      <td colspan="2" align=center><code>compose.yml</code>+ <code>compose.override.yml</code> (merged)</td>

      <td align=center><code>compose-production.yml</code></td>

    </tr>

  </tfoot>

  <tbody>

  </tbody>

</table>



#### Dotenv files lookup strategy



Unless you have explicitly set the [`COMPOSE_ENV_FILES`](https://docs.docker.com/compose/environment-variables/envvars/#compose_env_files) variable, the Docker Compose template will handle it 

and implement the following [dotenv files](https://docs.docker.com/compose/environment-variables/env-file/) lookup strategy:



  1. a `.env` file defining defaults for all environments,

  2. a `${environment_type}.env` file that might redefine or override defaults for a specific environment (e.g. `staging.env`).



#### Deployment



The **deployment** is processed as follows by the template:



1. _optionally_ executes the `pre-compose-up.sh` script found in your project to perform pre-deployment stuff (for e.g. create required services),

2. runs [`docker-compose up`](https://docs.docker.com/reference/cli/docker/compose/up/),

3. _optionally_ executes the `post-compose-up.sh` script found in your project to perform post-deployment stuff,



> :information_source: Important:

>

> Example:

> * Compose files, dotenv files and hook scripts are searched in the `$DCMP_SCRIPTS_DIR` directory (configurable),

> * Hook scripts need to be executable; you can add the execution flag with `git update-index --chmod=+x pre-compose-up.sh`.



The Docker Compose template requires you to provide a shell script that fully implements your application

deployment and cleanup using the `docker-compose` CLI and all other tools available in the selected Docker image.

#### Cleanup



The deployment script is searched as follows:

The **cleanup** is processed as follows by the template:



1. look for a specific `dcmp-deploy-$environment_type.sh` in the `$DCMP_SCRIPTS_DIR` directory in your project (e.g. `dcmp-deploy-staging.sh` for staging environment),

2. if not found: look for a default `dcmp-deploy.sh` in the `$DCMP_SCRIPTS_DIR` directory in your project,

3. if not found: the deployment job will fail.

1. _optionally_ executes the `pre-compose-down.sh` script found in your project to perform pre-cleanup stuff,

2. runs [`docker-compose down`](https://docs.docker.com/reference/cli/docker/compose/down/),

3. _optionally_ executes the `post-compose-down.sh` script found in your project to perform post-cleanup stuff,



The cleanup script is searched as follows:

> :information_source:  Important:

>

> * Compose files, dotenv files and hook scripts are searched in the `$DCMP_SCRIPTS_DIR` directory (configurable),

> * Hook scripts need to be executable; you can add the execution flag with `git update-index --chmod=+x pre-compose-up.sh`.



1. look for a specific `dcmp-cleanup-$environment_type.sh` in the `$DCMP_SCRIPTS_DIR` directory in your project (e.g. `dcmp-cleanup-staging.sh` for staging environment),

2. if not found: look for a default `dcmp-cleanup.sh` in the `$DCMP_SCRIPTS_DIR` directory in your project,

3. if not found: the cleanup job will fail.

#### Using Variables



> :information_source: Your deployment (and cleanup) scripts have to be able to cope with various environments, each with different application names, exposed routes, settings, ...

> Part of this complexity can be handled by the lookup policies described above (ex: one script per env) and also by using available environment variables:

>

> 1. [deployment context variables](#deployment-context-variables) provided by the template:

>     * `${environment_type}`: the current environment type (`review`, `integration`, `staging` or `production`)

>     * `${environment_name}`: the application name to use for the current environment (ex: `myproject-review-fix-bug-12` or `myproject-staging`)

>     * `${hostname}`: the environment hostname, extracted from the current environment url (after late variable expansion - see below)

> 2. any [GitLab CI variable](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)

> 3. any [custom variable](https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project)

>     (ex: `${SECRET_TOKEN}` that you have set in your project CI/CD variables)

Your deployment (and cleanup) scripts have to be able to cope with various environments, each with different application names, exposed routes, settings, ...

Part of this complexity can be handled by the lookup strategies described above (ex: one file per env) and also by using available environment variables:



1. [deployment context variables](#deployment-context-variables) provided by the template:

   * `${environment_type}`: the current environment type (`review`, `integration`, `staging` or `production`)

   * `${environment_name}`: the application name to use for the current environment (ex: `myproject-review-fix-bug-12` or `myproject-staging`)

   * `${hostname}`: the environment hostname, extracted from the current environment url (after late variable expansion - see below)

2. any [GitLab CI variable](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)

3. any [custom variable](https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-docker-host)

   (ex: `${SECRET_TOKEN}` that you have set in your project CI/CD variables)



Be aware that environment variables may be freely used and substitued in [dotenv files](https://docs.docker.com/compose/environment-variables/env-file/) 

using the appropriate [interpolation syntax](https://docs.docker.com/compose/environment-variables/env-file/#interpolation).



### Environments URL management
@@ -193,7 +275,7 @@ The **static way** can be implemented simply by setting the appropriate configur 
>   DCMP_REVIEW_ENVIRONMENT_URL: "https://wonderapp-review.nonprod.acme.domain/%{environment_name}"

> ```



To implement the **dynamic way**, your deployment script shall simply generate a `environment_url.txt` file in the working directory, containing only

To implement the **dynamic way**, your post deployment hook script shall simply generate a `environment_url.txt` file in the working directory, containing only

the dynamically generated url. When detected by the template, it will use it as the newly deployed environment url.



### Deployment output variables
@@ -206,7 +288,7 @@ Each deployment job produces _output variables_ that are propagated to downstrea 


Those variables may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).



You may also add and propagate your own custom variables, by pushing them to the `docker-compose.env` file in your [deployment scripts or hooks](#deployment-and-cleanup).

You may also add and propagate your own custom variables, by pushing them to the `docker-compose.out.env` file in your [deployment scripts or hooks](#deployment-and-cleanup).



## Configuration reference
@@ -214,7 +296,7 @@ You may also add and propagate your own custom variables, by pushing them to the 


Here are some advices about your **secrets** (variables marked with a :lock:): 



1. Manage them as [project or group CI/CD variables](https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project):

1. Manage them as [project or group CI/CD variables](https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-docker-host):

    * [**masked**](https://docs.gitlab.com/ee/ci/variables/#mask-a-cicd-variable) to prevent them from being inadvertently 

      displayed in your job logs,

    * [**protected**](https://docs.gitlab.com/ee/ci/variables/#protected-cicd-variables) if you want to secure some secrets
@@ -230,28 +312,30 @@ The Docker Compose template uses some global configuration used throughout all j 


| Input / Variable         | Description                            | Default value     |

| ------------------------ | -------------------------------------- | ----------------- |

| `image` / `DCMP_IMAGE`              | the Docker image used to run Docker Compose CLI commands | `registry.hub.docker.com/docker-compose:latest` |

| `image` / `DCMP_IMAGE` | the Docker image used to run Docker Compose CLI commands | `registry.hub.docker.com/library/docker:latest` |

| `cmd` / `DCMP_CMD`     | The docker compose command (`docker compose` or `docker-compose`) | _none_ (auto) |

| `base-app-name` / `DCMP_BASE_APP_NAME`| Base application name                  | `$CI_PROJECT_NAME` ([see GitLab doc](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)) |

| `api-url` / `DCMP_API_URL`            | Default Docker Compose API url | _none_ |

| :lock: `DCMP_API_TOKEN`          | Default Docker Compose API token | _none_ |

| `environment-url` / `DCMP_ENVIRONMENT_URL`| Default environments url _(only define for static environment URLs declaration)_<br/>_supports late variable expansion (ex: `https://%{environment_name}.docker-compose.acme.com`)_ | _none_ |

| `scripts-dir` / `DCMP_SCRIPTS_DIR`        | Directory where deploy & cleanup scripts are located | `.` _(root project dir)_ |

| `scripts-dir` / `DCMP_SCRIPTS_DIR`| Directory where Compose files, dotenv files and hook scripts are located | `.` _(root project dir)_ |

| `up-opts` / `DCMP_UP_OPTS` | [`compose up` options](https://docs.docker.com/reference/cli/docker/compose/up/#options) | `--no-build --remove-orphans --wait --wait-timeout 180` |

| `down-opts`/ `DCMP_DOWN_OPTS` | [`compose down` options](https://docs.docker.com/reference/cli/docker/compose/down/#options) | `--volumes --remove-orphans --rmi all` |

| :lock: `DCMP_SSH_PRIVATE_KEY` | Default SSH key to use when connecting to Docker hosts over SSH (can be overridden per env) | _none_ |

| `ssh-known-hosts` / `DCMP_SSH_KNOWN_HOSTS` | SSH `known_hosts` (file or text variable) | _none_ |



### Review environments configuration



Review environments are dynamic and ephemeral environments to deploy your _ongoing developments_ (a.k.a. _feature_ or 

_topic_ branches).



They are **disabled by default** and can be enabled by setting the `DCMP_REVIEW_PROJECT` variable (see below).

They are **disabled by default** and can be enabled by setting the `DCMP_REVIEW_DOCKER_HOST` variable (see below).



Here are variables supported to configure review environments:



| Input / Variable         | Description                            | Default value     |

| ------------------------ | -------------------------------------- | ----------------- |

| `review-project` / `DCMP_REVIEW_PROJECT`     | Project ID for `review` env | _none_ (disabled) |

| `review-docker-host` / `DCMP_REVIEW_DOCKER_HOST`     | Docker Host for `review` env (ex: `ssh://docker@docker-host-for-review:2375`) | _none_ (disabled) |

| :lock: `DCMP_REVIEW_SSH_PRIVATE_KEY` | `review` env specific SSH key to use when connecting to Docker Host over SSH | `$DCMP_SSH_PRIVATE_KEY` |

| `review-app-name` / `DCMP_REVIEW_APP_NAME`    | Application name for `review` env      | `"${DCMP_BASE_APP_NAME}-${CI_ENVIRONMENT_SLUG}"` (ex: `myproject-review-fix-bug-12`) |

| `review-api-url` / `DCMP_REVIEW_API_URL`     | API url for `review` env _(only define to override default)_ | `$DCMP_API_URL` |

| :lock: `DCMP_REVIEW_API_TOKEN`   | API token for `review` env _(only define to override default)_ | `$DCMP_API_TOKEN` |

| `review-environment-url` / `DCMP_REVIEW_ENVIRONMENT_URL`| The review environments url _(only define for static environment URLs declaration and if different from default)_ | `$DCMP_ENVIRONMENT_URL` |

| `review-autostop-duration` / `DCMP_REVIEW_AUTOSTOP_DURATION`| The amount of time before GitLab will automatically stop `review` environments | `4 hours` |
@@ -259,16 +343,15 @@ Here are variables supported to configure review environments: 


The integration environment is the environment associated to your integration branch (`develop` by default).



It is **disabled by default** and can be enabled by setting the `DCMP_INTEG_PROJECT` variable (see below).

It is **disabled by default** and can be enabled by setting the `DCMP_INTEG_DOCKER_HOST` variable (see below).



Here are variables supported to configure the integration environment:



| Input / Variable         | Description                            | Default value     |

| ------------------------ | -------------------------------------- | ----------------- |

| `integ-project` / `DCMP_INTEG_PROJECT`      | Project ID for `integration` env | _none_ (disabled) |

| `integ-docker-host` / `DCMP_INTEG_DOCKER_HOST`      | Docker Host for `integration` env (ex: `ssh://docker@docker-host-for-integ:2375`) | _none_ (disabled) |

| :lock: `DCMP_INTEG_SSH_PRIVATE_KEY` | `integration` env specific SSH key to use when connecting to Docker Host over SSH | `$DCMP_SSH_PRIVATE_KEY` |

| `integ-app-name` / `DCMP_INTEG_APP_NAME`     | Application name for `integration` env | `${DCMP_BASE_APP_NAME}-integration` |

| `integ-api-url` / `DCMP_INTEG_API_URL`      | API url for `integration` env _(only define to override default)_ | `$DCMP_API_URL` |

| :lock: `DCMP_INTEG_API_TOKEN`    | API token for `integration` env _(only define to override default)_ | `$DCMP_API_TOKEN` |

| `integ-environment-url` / `DCMP_INTEG_ENVIRONMENT_URL`| The integration environment url _(only define for static environment URLs declaration and if different from default)_ | `$DCMP_ENVIRONMENT_URL` |



### Staging environment configuration
@@ -276,31 +359,43 @@ Here are variables supported to configure the integration environment: 
The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production 

branch (`main` or `master` by default).



It is **disabled by default** and can be enabled by setting the `DCMP_STAGING_PROJECT` variable (see below).

It is **disabled by default** and can be enabled by setting the `DCMP_STAGING_DOCKER_HOST` variable (see below).



Here are variables supported to configure the staging environment:



| Input / Variable         | Description                            | Default value     |

| ------------------------ | -------------------------------------- | ----------------- |

| `staging-project` / `DCMP_STAGING_PROJECT`    | Project ID for `staging` env | _none_ (disabled) |

| `staging-docker-host` / `DCMP_STAGING_DOCKER_HOST`    | Docker Host for `staging` env (ex: `ssh://docker@docker-host-for-staging:2375`) | _none_ (disabled) |

| :lock: `DCMP_STAGING_SSH_PRIVATE_KEY` | `staging` env specific SSH key to use when connecting to Docker Host over SSH | `$DCMP_SSH_PRIVATE_KEY` |

| `staging-app-name` / `DCMP_STAGING_APP_NAME`   | Application name for `staging` env     | `${DCMP_BASE_APP_NAME}-staging` |

| `staging-api-url` / `DCMP_STAGING_API_URL`    | API url for `staging` env _(only define to override default)_ | `$DCMP_API_URL` |

| :lock: `DCMP_STAGING_API_TOKEN`  | API token for `staging` env _(only define to override default)_ | `$DCMP_API_TOKEN` |

| `staging-environment-url` / `DCMP_STAGING_ENVIRONMENT_URL`| The staging environment url _(only define for static environment URLs declaration and if different from default)_ | `$DCMP_ENVIRONMENT_URL` |



### Production environment configuration



The production environment is the final deployment environment associated with your production branch (`main` or `master` by default).



It is **disabled by default** and can be enabled by setting the `DCMP_PROD_PROJECT` variable (see below).

It is **disabled by default** and can be enabled by setting the `DCMP_PROD_DOCKER_HOST` variable (see below).



Here are variables supported to configure the production environment:



| Input / Variable          | Description                            | Default value     |

| ------------------------- | -------------------------------------- | ----------------- |

| `prod-project` / `DCMP_PROD_PROJECT`        | Project ID for `production` env | _none_ (disabled) |

| `prod-docker-host` / `DCMP_PROD_DOCKER_HOST`        | Docker Host for `production` env (ex: `ssh://docker@docker-host-for-prod:2375`) | _none_ (disabled) |

| :lock: `DCMP_PROD_SSH_PRIVATE_KEY` | `production` env specific SSH key to use when connecting to Docker Host over SSH | `$DCMP_SSH_PRIVATE_KEY` |

| `prod-app-name` / `DCMP_PROD_APP_NAME`       | Application name for `production` env  | `$DCMP_BASE_APP_NAME` |

| `prod-api-url` / `DCMP_PROD_API_URL`        | API url for `production` env _(only define to override default)_ | `$DCMP_API_URL` |

| :lock: `DCMP_PROD_API_TOKEN`      | API token for `production` env _(only define to override default)_ | `$DCMP_API_TOKEN` |

| `prod-environment-url` / `DCMP_PROD_ENVIRONMENT_URL`| The production environment url _(only define for static environment URLs declaration and if different from default)_ | `$DCMP_ENVIRONMENT_URL` |

| `prod-deploy-strategy` / `DCMP_PROD_DEPLOY_STRATEGY`| Defines the deployment to production strategy. One of `manual` (i.e. _one-click_) or `auto`. | `manual` |



### Compose Config job



The Docker Compose template enables running [Compose Config](https://docs.docker.com/reference/cli/docker/compose/config/), thus enabling detection of syntax errors in your Compose or dotenv files.



This job is mapped to the `package-test` stage and is **active** by default.



Here are its parameters:



| Input / Variable | Description                               | Default value                 |

| ----------------------- | ----------------------------------------- | ----------------------------- |

| `config-disabled` / `DCMP_CONFIG_DISABLED` | Set to `true` to disable `compose config`  | _none_ (enabled) |

| `config-opts` / `DCMP_CONFIG_OPTS` | [`compose config` options](https://docs.docker.com/reference/cli/docker/compose/config/#options) | `--quiet` _(to avoid displaying secrets inadvertently)_ |

kicker.json

+68 −63

File changed.

Preview size limit exceeded, changes collapsed.

templates/gitlab-ci-docker-compose.yml

+414 −124

File changed.

Preview size limit exceeded, changes collapsed.