feat: add parallel:matrix jobs support (1b4953c4) · Commits · to be continuous... / Kubernetes

README.md

+48 −0

Original line number	Diff line number	Diff line
		@@ -341,6 +341,43 @@ The static way can be implemented simply by setting the appropriate configur
		To implement the dynamic way, your deployment script shall simply generate a `environment_url.txt` file in the working directory, containing only
		the dynamically generated url. When detected by the template, it will use it as the newly deployed environment url.

		### Multiple environments support

		The Kubernetes template allows deploying multiple environments in parallel. Use cases of this include:

		- monorepo, where a single Git repository might host several separate deployable components or apps,
		- multi-instances deployment of the same application.

		This feature can be enabled using the [parallel matrix jobs](https://docs.gitlab.com/ee/ci/yaml/#parallelmatrix)
		pattern at the `.k8s-env-base` job level (this is the top parent job of all deployment jobs).
		Environments namespacing is ensured by the `K8S_ENVIRONMENT_NAMESPACE` variable (must start with a slash `/`).

		Here is the example of the `.gitlab-ci.yml` file for a project deploying both a frontend and a backend applications:

		```yaml
		.k8s-env-base:
		parallel:
		matrix:
		- K8S_ENVIRONMENT_NAMESPACE: "/front"
		# Kubernetes deployment scripts are located in the ./front/ directory
		K8S_SCRIPTS_DIR: "front"
		- K8S_ENVIRONMENT_NAMESPACE: "/back"
		# Kubernetes deployment scripts are located in the ./back/ directory
		K8S_SCRIPTS_DIR: "back"

		# ⚠ on_stop must be unset when defining parallel:matrix environments
		# see: https://gitlab.com/gitlab-org/gitlab/-/issues/332247
		k8s-review:
		environment:
		on_stop: null
		```

		The above configuration will deploy 2 environments on each pipeline:

		- on feature branches: `review/front/$CI_COMMIT_REF_NAME` and `review/back/$CI_COMMIT_REF_NAME`
		- on the integration branch: `integration/front` and `integration/back`
		- on the production branch: `staging/front` and `staging/back` (and finally `production/front` and `production/back`)

		### Deployment output variables

		Each deployment job produces _output variables_ that are propagated to downstream jobs (using [dotenv artifacts](https://docs.gitlab.com/ci/yaml/artifacts_reports/#artifactsreportsdotenv)):
		@@ -353,6 +390,16 @@ Those variables may be freely used in downstream jobs (for instance to run accep

		You may also add and propagate your own custom variables, by pushing them to the `kubernetes.env` file in your [deployment script or hook](#supported-deployment-methods).

		> [!important]
		> If [multiple environments](#multiple-environments-support) are configured, the output variables are prefixed with a
		> sluggified value of the `K8S_ENVIRONMENT_NAMESPACE` variable (stripped of punctuation characters and converted to lowercase):
		>
		> * `<namespace_slug>_environment_type`: set to the type of environment (`review`, `integration`, `staging` or `production`),
		> * `<namespace_slug>_environment_name`: the application name (see below),
		> * `<namespace_slug>_environment_url`: set to the environment URL (whether determined statically or dynamically).
		>
		> The output dotenv file will be `kubernetes.env.<namespace_slug>` instead, and the dynamic variable `${environment_namespace}` can be used in your scripts and manifests to access the contextual value of `<namespace_slug>`.

		## Configuration reference

		### Secrets management
		@@ -378,6 +425,7 @@ The Kubernetes template uses some global configuration used throughout all jobs.
		\| `kubectl-image` / `K8S_KUBECTL_IMAGE` \| the Docker image used to run Kubernetes `kubectl` commands <br/>:warning: set the version required by your Kubernetes server \| `docker.io/alpine/k8s:MUST_SET_VERSION` <br/>[![Trivy Badge](https://to-be-continuous.gitlab.io/doc/secu/trivy-badge-K8S_KUBECTL_IMAGE.svg)](https://to-be-continuous.gitlab.io/doc/secu/trivy-K8S_KUBECTL_IMAGE) \|
		\| `base-app-name` / `K8S_BASE_APP_NAME` \| Default application name \| `$CI_PROJECT_NAME` ([see GitLab doc](https://docs.gitlab.com/ci/variables/predefined_variables/)) \|
		\| `environment-url` / `K8S_ENVIRONMENT_URL` \| Default environments url _(only define for static environment URLs declaration)_<br/>_supports late variable expansion (ex: `https://%{environment_name}.k8s.acme.com`)_ \| _none_ \|
		\| `environment-namespace` / `K8S_ENVIRONMENT_NAMESPACE` \| Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying [multiple environments](#multiple-environments-support))_<br/>:warning: must start with a slash `/` \| _none_ \|
		\| `KUBE_CONTEXT` \| Defines the context to be used in `KUBECONFIG`. When using [GitLab agents with the CI/CD workflow](https://docs.gitlab.com/user/clusters/agent/ci_cd_workflow/), the value should be like `path/to/agent/project:agent-name`. To use different agents per environment, define an [environment-scoped CI/CD variable](https://docs.gitlab.com/ci/environments/#limit-the-environment-scope-of-a-cicd-variable) for each agent. \| _none_ \|
		\| :lock: `K8S_DEFAULT_KUBE_CONFIG`\| The default kubeconfig to use (either content or file variable) \| required if not using exploded kubeconfig parameters \|
		\| `url` / `K8S_URL` \| the Kubernetes API url \| required if using exploded kubeconfig parameters \|

kicker.json

+5 −0

Original line number	Diff line number	Diff line
		@@ -42,6 +42,11 @@
		"type": "url",
		"description": "The default environments url _(only define for static environment URLs declaration)_\n\n_supports late variable expansion (ex: `https://%{environment_name}.k8s.acme.com`)_"
		},
		{
		"name": "K8S_ENVIRONMENT_NAMESPACE",
		"description": "Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying multiple environments)_\n\n:warning: must start with a slash `/`",
		"advanced": true
		},
		{
		"name": "K8S_SCRIPTS_DIR",
		"description": "directory where Kubernetes scripts (templates, hook scripts) are located",

templates/gitlab-ci-k8s-aws.yml

+1 −11

Original line number	Diff line number	Diff line
		@@ -56,17 +56,7 @@ variables:
		export AWS_ROLE_SESSION_NAME="GitLabRunner-${CI_PROJECT_ID}-${CI_PIPELINE_ID}"
		fi

		.k8s-deploy:
		id_tokens:
		AWS_JWT:
		aud: "$AWS_OIDC_AUD"
		before_script:
		- !reference [.k8s-scripts]
		- !reference [.k8s-aws-sts]
		- install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}"
		- k8s_login

		.k8s-cleanup:
		.k8s-env-base:
		id_tokens:
		AWS_JWT:
		aud: "$AWS_OIDC_AUD"

templates/gitlab-ci-k8s-gcp.yml

+1 −11

Original line number	Diff line number	Diff line
		@@ -98,17 +98,7 @@ variables:
		echo '[WARN] $GCP_JWT is not set: cannot setup Application Default Credentials (ADC) authentication'
		fi

		.k8s-deploy:
		id_tokens:
		GCP_JWT:
		aud: "$GCP_OIDC_AUD"
		before_script:
		- !reference [.k8s-scripts]
		- !reference [.k8s-gcp-adc]
		- install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}"
		- k8s_login

		.k8s-cleanup:
		.k8s-env-base:
		id_tokens:
		GCP_JWT:
		aud: "$GCP_OIDC_AUD"

templates/gitlab-ci-k8s.yml

+39 −38

Original line number	Diff line number	Diff line
		@@ -30,6 +30,12 @@ spec:

		_supports late variable expansion (ex: `https://%{environment_name}.k8s.acme.com`)_
		default: ''
		environment-namespace:
		description: \|-
		Extra [GitLab environments](https://docs.gitlab.com/ci/environments/) namespace _(only required when deploying multiple environments)_

		:warning: must start with a slash `/`
		default: ''
		scripts-dir:
		description: directory where Kubernetes scripts (templates, hook scripts) are located
		default: .
		@@ -198,6 +204,7 @@ variables:

		K8S_URL: $[[ inputs.url ]]
		K8S_ENVIRONMENT_URL: $[[ inputs.environment-url ]]
		K8S_ENVIRONMENT_NAMESPACE: $[[ inputs.environment-namespace ]]
		K8S_KUSTOMIZE_ENABLED: $[[ inputs.kustomize-enabled ]]
		K8S_KUSTOMIZE_ARGS: $[[ inputs.kustomize-args ]]
		K8S_CREATE_NAMESPACE_ENABLED: $[[ inputs.create-namespace-enabled ]]
		@@ -650,6 +657,8 @@ stages:
		export environment_type=$ENV_TYPE
		export environment_name=${ENV_APP_NAME:-${K8S_BASE_APP_NAME}${ENV_APP_SUFFIX}}
		environment_url=${ENV_URL:-$K8S_ENVIRONMENT_URL}
		environment_namespace=$(echo "$K8S_ENVIRONMENT_NAMESPACE" \| tr -d '[:punct:]' \| tr '[:upper:]' '[:lower:]')
		export environment_namespace
		# also export environment_name in SCREAMING_SNAKE_CASE format (may be useful with Kubernetes env variables)
		environment_name_ssc=$(to_ssc "$environment_name")
		export environment_name_ssc
		@@ -674,7 +683,7 @@ stages:
		log_info "--- \$hostname: \\e[33;1m${hostname}\\e[0m"

		# unset any upstream deployment env & artifacts
		rm -f kubernetes.env
		rm -f kubernetes.env*
		rm -f environment_url.txt

		# maybe execute deploy script
		@@ -716,8 +725,15 @@ stages:
		else
		echo "$environment_url" > environment_url.txt
		fi
		echo -e "environment_type=$environment_type\\nenvironment_name=$environment_name\\nenvironment_url=$environment_url" >> kubernetes.env
		chmod 644 environment_url.txt kubernetes.env
		# var prefix ('_' if namespace)
		prefix="${environment_namespace:+${environment_namespace}_}"
		dotenvfile="kubernetes.env${environment_namespace:+.${environment_namespace}}"
		{
		echo "${prefix}environment_type=${environment_type}"
		echo "${prefix}environment_name=${environment_name}"
		echo "${prefix}environment_url=${environment_url}"
		} >> "$dotenvfile"
		chmod 644 environment_url.txt "$dotenvfile"

		# maybe execute readiness check script
		readycheck="$K8S_SCRIPTS_DIR/k8s-readiness-check.sh"
		@@ -979,7 +995,7 @@ k8s-score-production:
		when: never
		- !reference [.test-policy, rules]

		# Deploy job prototype
		# Env management job prototype
		# Can be extended to define a concrete environment
		#
		# @arg ENV_TYPE : environment type
		@@ -990,15 +1006,23 @@ k8s-score-production:
		# @arg ENV_TOKEN : env-specific Kubernetes API token
		# @arg ENV_CA_CERT : env-specific Kubernetes CA certificate
		# @arg ENV_KUBE_CONFIG: env-specific Kubeconfig
		.k8s-deploy:
		.k8s-env-base:
		extends: .k8s-base
		stage: deploy
		variables:
		ENV_APP_SUFFIX: "-$CI_ENVIRONMENT_SLUG"
		before_script:
		- !reference [.k8s-scripts]
		- install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}"
		- !reference [.k8s-base, before_script]
		- k8s_login
		environment:
		name: ${ENV_TYPE}${K8S_ENVIRONMENT_NAMESPACE}
		resource_group: ${ENV_TYPE}${K8S_ENVIRONMENT_NAMESPACE}

		# Deploy job prototype
		# Can be extended for each deployable environment
		# @arg ENV_URL : env-specific application url
		.k8s-deploy:
		extends: .k8s-env-base
		script:
		- k8s_deploy
		artifacts:
		@@ -1007,32 +1031,16 @@ k8s-score-production:
		paths:
		- environment_url.txt
		reports:
		dotenv: kubernetes.env
		dotenv: kubernetes.env*
		environment:
		url: "$environment_url" # can be either static or dynamic

		# Cleanup job prototype
		# Can be extended for each deletable environment
		#
		# @arg ENV_TYPE : environment type
		# @arg ENV_APP_NAME : env-specific application name
		# @arg ENV_APP_SUFFIX : env-specific application suffix
		# @arg ENV_API_URL : env-specific Kubernetes API url
		# @arg ENV_SPACE : env-specific Kubernetes namespace
		# @arg ENV_TOKEN : env-specific Kubernetes API token
		# @arg ENV_CA_CERT : env-specific Kubernetes CA certificate
		# @arg ENV_KUBE_CONFIG: env-specific Kubeconfig
		.k8s-cleanup:
		extends: .k8s-base
		stage: deploy
		extends: .k8s-env-base
		# force no dependencies
		dependencies: []
		variables:
		ENV_APP_SUFFIX: "-$CI_ENVIRONMENT_SLUG"
		before_script:
		- !reference [.k8s-scripts]
		- install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}"
		- k8s_login
		script:
		- k8s_cleanup
		environment:
		@@ -1050,10 +1058,12 @@ k8s-review:
		ENV_KUBE_CONFIG: "$K8S_REVIEW_KUBE_CONFIG"
		ENV_URL: "${K8S_REVIEW_ENVIRONMENT_URL}"
		environment:
		name: review/$CI_COMMIT_REF_NAME
		name: ${ENV_TYPE}${K8S_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}
		# ⚠ on_stop must be unset when defining parallel:matrix environments
		# see: https://gitlab.com/gitlab-org/gitlab/-/issues/332247
		on_stop: k8s-cleanup-review
		auto_stop_in: "$K8S_REVIEW_AUTOSTOP_DURATION"
		resource_group: review/$CI_COMMIT_REF_NAME
		resource_group: ${ENV_TYPE}${K8S_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}
		rules:
		# exclude tags
		- if: $CI_COMMIT_TAG
		@@ -1073,9 +1083,9 @@ k8s-cleanup-review:
		ENV_CA_CERT: "$K8S_REVIEW_CA_CERT"
		ENV_KUBE_CONFIG: "$K8S_REVIEW_KUBE_CONFIG"
		environment:
		name: review/$CI_COMMIT_REF_NAME
		name: ${ENV_TYPE}${K8S_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}
		action: stop
		resource_group: review/$CI_COMMIT_REF_NAME
		resource_group: ${ENV_TYPE}${K8S_ENVIRONMENT_NAMESPACE}/${CI_COMMIT_REF_NAME}
		rules:
		# exclude tags
		- if: $CI_COMMIT_TAG
		@@ -1096,9 +1106,6 @@ k8s-integration:
		ENV_CA_CERT: "$K8S_INTEG_CA_CERT"
		ENV_KUBE_CONFIG: "$K8S_INTEG_KUBE_CONFIG"
		ENV_URL: "${K8S_INTEG_ENVIRONMENT_URL}"
		environment:
		name: integration
		resource_group: integration
		rules:
		# only on integration branch(es), with $K8S_INTEG_SPACE set
		- if: '$K8S_INTEG_SPACE != "" && $CI_COMMIT_REF_NAME =~ $INTEG_REF'
		@@ -1121,9 +1128,6 @@ k8s-staging:
		ENV_CA_CERT: "$K8S_STAGING_CA_CERT"
		ENV_KUBE_CONFIG: "$K8S_STAGING_KUBE_CONFIG"
		ENV_URL: "${K8S_STAGING_ENVIRONMENT_URL}"
		environment:
		name: staging
		resource_group: staging
		rules:
		# only on production branch(es), with $K8S_STAGING_SPACE set
		- if: '$K8S_STAGING_SPACE != "" && $CI_COMMIT_REF_NAME =~ $PROD_REF'
		@@ -1141,9 +1145,6 @@ k8s-production:
		ENV_CA_CERT: "$K8S_PROD_CA_CERT"
		ENV_KUBE_CONFIG: "$K8S_PROD_KUBE_CONFIG"
		ENV_URL: "${K8S_PROD_ENVIRONMENT_URL}"
		environment:
		name: production
		resource_group: production
		rules:
		# exclude non-production branches
		- if: '$CI_COMMIT_REF_NAME !~ $PROD_REF'