Merge branch 'feat/support-parallel-matrix' into 'main' (1fb5f369) · Commits · to be continuous... / S3

README.md

+61 −0

Original line number	Diff line number	Diff line
		@@ -147,6 +147,56 @@ variables:
		S3_REVIEW_PREFIX: "$CI_ENVIRONMENT_SLUG"
		```

		### Multiple environments support

		In some situations, you might want to deploy multiple instances of your application to the same environment
		(for instance in a monorepo context, or to deploy the same application with different configurations).

		The S3 template supports this use case by leveraging GitLab's [`parallel:matrix`](https://docs.gitlab.com/ci/yaml/#parallelmatrix)
		feature combined with the `S3_ENVIRONMENT_NAMESPACE` variable.

		The `.s3-env-base` hidden job (extended by both `.s3-deploy` and `.s3-cleanup`) supports
		`parallel:matrix` with the `S3_ENVIRONMENT_NAMESPACE` variable (that shall begin with `/`).

		Here is an example that deploys both a front-end and a back-end to S3:

		The S3 template allows deploying multiple environments in parallel. Use cases of this include:

		- monorepo, where a single Git repository might host several separate deployable components or apps,
		- multi-instances deployment of the same application.

		This feature can be enabled using the [parallel matrix jobs](https://docs.gitlab.com/ee/ci/yaml/#parallelmatrix)
		pattern at the `.s3-env-base` job level (this is the top parent job of all deployment jobs).
		Environments namespacing is ensured by the `S3_ENVIRONMENT_NAMESPACE` variable (must start with a slash `/`).

		Here is the example of the `.gitlab-ci.yml` file for a project deploying 2 static websites (_doc_ and _dev-guide_):

		```yaml
		.s3-env-base:
		parallel:
		matrix:
		- S3_ENVIRONMENT_NAMESPACE: /doc
		S3_SCRIPTS_DIR: doc
		S3_PREFIX: doc
		S3_DEPLOY_FILES: doc/public/
		- S3_ENVIRONMENT_NAMESPACE: /dev-guide
		S3_SCRIPTS_DIR: dev-guide
		S3_PREFIX: dev-guide
		S3_DEPLOY_FILES: dev-guide/public/

		# ⚠ on_stop must be unset when defining parallel:matrix environments
		# see: https://gitlab.com/gitlab-org/gitlab/-/issues/332247
		s3-review:
		environment:
		on_stop: null
		```

		The above configuration will deploy 2 environments on each pipeline:

		- on feature branches: `review/doc/$CI_COMMIT_REF_NAME` and `review/dev-guide/$CI_COMMIT_REF_NAME`
		- on the integration branch: `integration/doc` and `integration/dev-guide`
		- on the production branch: `staging/doc` and `staging/dev-guide` (and finally `production/doc` and `production/dev-guide`)

		### Deployment output variables

		As seen above, the S3 template may support up to 4 environments (`review`, `integration`, `staging` and `production`).
		@@ -159,6 +209,16 @@ Each deployment job produces _output variables_ that are propagated to downstrea

		They may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).

		> [!important]
		> If [multiple environments](#multiple-environments-support) are configured, the output variables are prefixed with a
		> sluggified value of the `S3_ENVIRONMENT_NAMESPACE` variable (stripped of punctuation characters and converted to lowercase):
		>
		> * `<namespace_slug>_environment_type`: set to the type of environment (`review`, `integration`, `staging` or `production`),
		> * `<namespace_slug>_environment_name`: the application name (see below),
		> * `<namespace_slug>_environment_url`: set to the environment URL (whether determined statically or dynamically).
		>
		> The output dotenv file will be `s3.env.<namespace_slug>` instead, and the dynamic variable `${environment_namespace}` can be used in your scripts and manifests to access the contextual value of `<namespace_slug>`.

		## Configuration reference

		### Secrets management
		@@ -189,6 +249,7 @@ The S3 template uses some global configuration used throughout all jobs.
		\| :lock: `S3_SECRET_KEY` \| Default S3 service Secret Key \| has to be defined \|
		\| `base-bucket-name` / `S3_BASE_BUCKET_NAME` \| Base bucket name \| `$CI_PROJECT_NAME` ([see GitLab doc](https://docs.gitlab.com/ci/variables/predefined_variables/)) \|
		\| `prefix` / `S3_PREFIX` \| Default S3 prefix to use as a root destination to upload objects in the S3 bucket \| _none_ \|
		\| `environment-namespace` / `S3_ENVIRONMENT_NAMESPACE` \| Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying [multiple environments](#multiple-environments-support))_<br/>:warning: must start with a slash `/` \| _none_ \|
		\| `scripts-dir` / `S3_SCRIPTS_DIR` \| Directory where S3 hook scripts are located \| `.`\|

		### Deployment jobs

kicker.json

+5 −0

Original line number	Diff line number	Diff line
		@@ -77,6 +77,11 @@
		"name": "S3_PREFIX",
		"description": "Default S3 prefix to use as a root destination to upload objects in the S3 bucket"
		},
		{
		"name": "S3_ENVIRONMENT_NAMESPACE",
		"description": "Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying multiple environments)_\n\n:warning: must start with a slash `/`",
		"advanced": true
		},
		{
		"name": "S3_SCRIPTS_DIR",
		"description": "Directory where S3 hook scripts are located",

templates/gitlab-ci-s3.yml

+46 −33

Original line number	Diff line number	Diff line
		@@ -50,6 +50,12 @@ spec:
		prefix:
		description: Default S3 prefix to use as a root destination to upload objects in the S3 bucket
		default: ''
		environment-namespace:
		description: \|-
		Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying multiple environments)_

		:warning: must start with a slash `/`
		default: ''
		scripts-dir:
		description: Directory where S3 hook scripts are located
		default: .
		@@ -183,6 +189,7 @@ variables:
		S3_ENDPOINT_HOST: $[[ inputs.endpoint-host ]]
		S3_HOST_BUCKET: $[[ inputs.host-bucket ]]
		S3_BASE_BUCKET_NAME: $[[ inputs.base-bucket-name ]]
		S3_ENVIRONMENT_NAMESPACE: $[[ inputs.environment-namespace ]]
		S3_SCRIPTS_DIR: $[[ inputs.scripts-dir ]]
		S3_BASE_TEMPLATE_NAME: s3
		S3_REVIEW_AUTOSTOP_DURATION: $[[ inputs.review-autostop-duration ]]
		@@ -487,6 +494,8 @@ stages:
		# upload/sync files to bucket
		function s3_deploy() {
		export environment_type=$ENV_TYPE
		environment_namespace=$(echo "$S3_ENVIRONMENT_NAMESPACE" \| tr -d '[:punct:]' \| tr '[:upper:]' '[:lower:]')
		export environment_namespace
		bucket=${ENV_BUCKET_NAME:-${S3_BASE_BUCKET_NAME}${ENV_APP_SUFFIX}}
		prefix=${ENV_PREFIX:-${S3_PREFIX}}
		region=${ENV_REGION:-${S3_REGION}}
		@@ -505,7 +514,7 @@ stages:
		log_info "--- \$prefix: \\e[33;1m${prefix:-(none)}\\e[0m"

		# unset any upstream deployment env & artifacts
		rm -f s3.env
		rm -f s3.env*
		rm -f environment_url.txt

		# maybe create bucket
		@@ -553,8 +562,15 @@ stages:
		environment_url="$website_url$prefix"
		# finally persist environment url
		echo "$environment_url" > environment_url.txt
		echo -e "environment_type=$environment_type\\nenvironment_name=$bucket\\nenvironment_url=$environment_url" > s3.env
		chmod 644 environment_url.txt s3.env
		# var prefix ('_' if namespace)
		prefix="${environment_namespace:+${environment_namespace}_}"
		dotenvfile="s3.env${environment_namespace:+.${environment_namespace}}"
		{
		echo "${prefix}environment_type=${environment_type}"
		echo "${prefix}environment_name=${bucket}"
		echo "${prefix}environment_url=${environment_url}"
		} >> "$dotenvfile"
		chmod 644 environment_url.txt "$dotenvfile"
		fi
		}

		@@ -647,9 +663,8 @@ stages:
		before_script:
		- !reference [.s3-scripts]
		- install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}"
		- s3_login

		# Deploy job prototype
		# Env management job prototype
		# Can be extended to define a concrete environment
		#
		# @arg ENV_TYPE : environment type
		@@ -657,11 +672,23 @@ stages:
		# @arg ENV_REGION : env-specific region
		# @arg ENV_PREFIX : env-specific S3 prefix
		# @arg ENV_APP_SUFFIX : env-specific application suffix
		.s3-deploy:
		.s3-env-base:
		extends: .s3-base
		stage: deploy
		variables:
		ENV_APP_SUFFIX: "-$CI_ENVIRONMENT_SLUG"
		before_script:
		- !reference [.s3-base, before_script]
		- s3_login
		environment:
		name: ${ENV_TYPE}${S3_ENVIRONMENT_NAMESPACE}
		resource_group: ${ENV_TYPE}${S3_ENVIRONMENT_NAMESPACE}

		# Deploy job prototype
		# Can be extended for each deployable environment
		# @arg ENV_URL : env-specific application url
		.s3-deploy:
		extends: .s3-env-base
		script:
		- s3_deploy
		artifacts:
		@@ -669,26 +696,19 @@ stages:
		paths:
		- environment_url.txt
		reports:
		dotenv: s3.env
		dotenv: s3.env*
		environment:
		url: "$environment_url" # can be either static or dynamic

		# Cleanup job prototype
		# Can be extended for each deletable environment
		#
		# @arg ENV_TYPE : environment type
		# @arg ENV_BUCKET_NAME : env-specific S3 bucket name
		# @arg ENV_PREFIX : env-specific S3 prefix
		# @arg ENV_APP_SUFFIX : env-specific application suffix
		.s3-cleanup:
		extends: .s3-base
		stage: deploy
		extends: .s3-env-base
		# force no dependencies
		dependencies: []
		# no need to clone repository
		variables:
		GIT_STRATEGY: none
		ENV_APP_SUFFIX: "-$CI_ENVIRONMENT_SLUG"
		script:
		- s3_delete
		environment:
		@@ -706,10 +726,12 @@ s3-review:
		ENV_ACCESS_KEY: "$S3_REVIEW_ACCESS_KEY"
		ENV_SECRET_KEY: "$S3_REVIEW_SECRET_KEY"
		environment:
		name: review/$CI_COMMIT_REF_NAME
		name: ${ENV_TYPE}${S3_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}
		# ⚠ on_stop must be unset when defining parallel:matrix environments
		# see: https://gitlab.com/gitlab-org/gitlab/-/issues/332247
		on_stop: s3-cleanup-review
		auto_stop_in: "$S3_REVIEW_AUTOSTOP_DURATION"
		resource_group: review/$CI_COMMIT_REF_NAME
		resource_group: ${ENV_TYPE}${S3_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}
		rules:
		# exclude tags
		- if: $CI_COMMIT_TAG
		@@ -734,9 +756,9 @@ s3-cleanup-review:
		ENV_ACCESS_KEY: "$S3_REVIEW_ACCESS_KEY"
		ENV_SECRET_KEY: "$S3_REVIEW_SECRET_KEY"
		environment:
		name: review/$CI_COMMIT_REF_NAME
		name: ${ENV_TYPE}${S3_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}
		action: stop
		resource_group: review/$CI_COMMIT_REF_NAME
		resource_group: ${ENV_TYPE}${S3_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}
		rules:
		# exclude tags
		- if: $CI_COMMIT_TAG
		@@ -794,9 +816,6 @@ s3-integration:
		ENV_ENDPOINT_HOST: "$S3_INTEG_ENDPOINT_HOST"
		ENV_ACCESS_KEY: "$S3_INTEG_ACCESS_KEY"
		ENV_SECRET_KEY: "$S3_INTEG_SECRET_KEY"
		environment:
		name: integration
		resource_group: integration
		rules:
		# exclude if $S3_INTEG_DISABLED set
		- if: '$S3_INTEG_DISABLED == "true"'
		@@ -816,9 +835,6 @@ s3-staging:
		ENV_ENDPOINT_HOST: "$S3_STAGING_ENDPOINT_HOST"
		ENV_ACCESS_KEY: "$S3_STAGING_ACCESS_KEY"
		ENV_SECRET_KEY: "$S3_STAGING_SECRET_KEY"
		environment:
		name: staging
		resource_group: staging
		rules:
		# exclude if $S3_STAGING_DISABLED set
		- if: '$S3_STAGING_DISABLED == "true"'
		@@ -839,9 +855,6 @@ s3-production:
		ENV_ENDPOINT_HOST: "$S3_PROD_ENDPOINT_HOST"
		ENV_ACCESS_KEY: "$S3_PROD_ACCESS_KEY"
		ENV_SECRET_KEY: "$S3_PROD_SECRET_KEY"
		environment:
		name: production
		resource_group: production
		rules:
		# exclude non-production branches
		- if: '$CI_COMMIT_REF_NAME !~ $PROD_REF'