Merge branch 'feat/component' into 'master' (7728be7b) · Commits · to be continuous... / dbt

.gitlab-ci.yml

+1 −1

Original line number	Diff line number	Diff line
		@@ -10,7 +10,7 @@ include:
		file: '/templates/validation.yml'
		- project: 'to-be-continuous/bash'
		ref: '3.3'
		file: 'templates/gitlab-ci-bash.yml'
		file: '/templates/gitlab-ci-bash.yml'
		- project: 'to-be-continuous/semantic-release'
		ref: '3.7'
		file: '/templates/gitlab-ci-semrel.yml'

README.md

+73 −56

Original line number	Diff line number	Diff line
		@@ -34,13 +34,40 @@ You're free to enable whichever or both, and you can also choose your deployment

		## Usage

		In order to include this template in your project, add the following to your `gitlab-ci.yml`:
		This template can be used both as a [CI/CD component](https://docs.gitlab.com/ee/ci/components/#use-a-component-in-a-cicd-configuration)
		or using the legacy [`include:project`](https://docs.gitlab.com/ee/ci/yaml/index.html#includeproject) syntax.

		### Use as a CI/CD component

		Add the following to your `gitlab-ci.yml`:

		```yaml
		include:
		# 1: include the component
		- component: gitlab.com/to-be-continuous/dbt/gitlab-ci-dbt@3.1.1
		# 2: set/override component inputs
		inputs:
		# ⚠ this is only an example
		staging-target: "nonprod"
		prod-target: "prod"
		```

		### Use as a CI/CD template (legacy)

		Add the following to your `gitlab-ci.yml`:

		```yaml
		include:
		# 1: include the template
		- project: 'to-be-continuous/dbt'
		ref: '3.1.1'
		file: '/templates/gitlab-ci-dbt.yml'

		variables:
		# 2: set/override template variables
		# ⚠ this is only an example
		DBT_STAGING_TARGET: "nonprod"
		DBT_PROD_TARGET: "prod"
		```

		You can find a sample of dbt project here : https://gitlab.com/to-be-continuous/samples/dbt-sample/
		@@ -49,14 +76,14 @@ You can find a sample of dbt project here : https://gitlab.com/to-be-continuous/

		The dbt template uses some global configuration used throughout all jobs.

		\| Name \| Description \| Default value \|
		\| Input / Variable \| Description \| Default value \|
		\| --------------------- \| -------------------------------------- \| ----------------- \|
		\| `DBT_IMAGE` \| The Docker image used to run dbt \| `registry.hub.docker.com/library/python:latest` \|
		\| `DBT_PROJECT_DIR` \| The [dbt_project.yml](https://docs.getdbt.com/reference/dbt_project.yml) dir \| `.` \|
		\| `DBT_PROFILES_DIR` \| The dbt [profile](https://docs.getdbt.com/dbt-cli/configure-your-profile) location \| `.` \|
		\| `DBT_ADAPTER` \| The dbt [adapter](https://docs.getdbt.com/docs/available-adapters) to use \| __none__ (required) \|
		\| `DBT_TARGET` \| The dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) to use \| __none__ (required) \|
		\| `DBT_BUILD_ARGS` \| Arguments used by [`dbt cli`](https://docs.getdbt.com/reference/global-configs#command-line-flags) \| __none__ \|
		\| `image` / `DBT_IMAGE` \| The Docker image used to run dbt \| `registry.hub.docker.com/library/python:latest` \|
		\| `project-dir` / `DBT_PROJECT_DIR` \| The [dbt_project.yml](https://docs.getdbt.com/reference/dbt_project.yml) dir \| `.` \|
		\| `profiles-dir` / `DBT_PROFILES_DIR` \| The dbt [profile](https://docs.getdbt.com/dbt-cli/configure-your-profile) location \| `.` \|
		\| `adapter` / `DBT_ADAPTER` \| The dbt [adapter](https://docs.getdbt.com/docs/available-adapters) to use \| __none__ (required) \|
		\| `target` / `DBT_TARGET` \| The dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) to use \| __none__ (required) \|
		\| `build-args` / `DBT_BUILD_ARGS` \| Arguments used by [`dbt cli`](https://docs.getdbt.com/reference/global-configs#command-line-flags) \| __none__ \|

		### Environments configuration

		@@ -79,9 +106,9 @@ They are disabled by default and can be enabled by setting the `DBT_REVIEW_T

		Here are variables supported to configure review environments:

		\| Name \| Description \| Default value \|
		\| Input / Variable \| Description \| Default value \|
		\| ------------------------ \| -------------------------------------- \| ----------------- \|
		\| `DBT_REVIEW_TARGET` \| dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) for `review` env \| _none_ (disabled) \|
		\| `review-target` / `DBT_REVIEW_TARGET` \| dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) for `review` env \| _none_ (disabled) \|


		#### Integration environment
		@@ -92,9 +119,9 @@ It is disabled by default and can be enabled by setting the `DBT_INTEG_TARGE

		Here are variables supported to configure the integration environment:

		\| Name \| Description \| Default value \|
		\| Input / Variable \| Description \| Default value \|
		\| ------------------------ \| -------------------------------------- \| ----------------- \|
		\| `DBT_INTEG_TARGET` \| dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) for `integration` env \| _none_ (disabled) \|
		\| `integ-target` / `DBT_INTEG_TARGET` \| dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) for `integration` env \| _none_ (disabled) \|

		#### Staging environment

		@@ -104,9 +131,9 @@ It is disabled by default and can be enabled by setting the `DBT_STAGING_TAR

		Here are variables supported to configure the staging environment:

		\| Name \| Description \| Default value \|
		\| Input / Variable \| Description \| Default value \|
		\| ------------------------ \| -------------------------------------- \| ----------------- \|
		\| `DBT_STAGING_TARGET` \| dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) for `staging` env \| _none_ (disabled) \|
		\| `staging-target` / `DBT_STAGING_TARGET` \| dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) for `staging` env \| _none_ (disabled) \|

		#### Production environment

		@@ -116,10 +143,10 @@ It is disabled by default and can be enabled by setting the `DBT_PROD_TARGET

		Here are variables supported to configure the production environment:

		\| Name \| Description \| Default value \|
		\| Input / Variable \| Description \| Default value \|
		\| ------------------------ \| -------------------------------------- \| ----------------- \|
		\| `DBT_PROD_TARGET` \| dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) for `production` env \| _none_ (disabled) \|
		\| `DBT_PROD_DEPLOY_STRATEGY`\| Defines the deployment to production strategy. One of `manual` (i.e. _one-click_) or `auto`. \| `manual` \|
		\| `prod-target` / `DBT_PROD_TARGET` \| dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) for `production` env \| _none_ (disabled) \|
		\| `prod-deploy-strategy` / `DBT_PROD_DEPLOY_STRATEGY` \| Defines the deployment to production strategy. One of `manual` (i.e. _one-click_) or `auto`. \| `manual` \|


		## Jobs
		@@ -137,10 +164,10 @@ This job performs SQL Lint.

		`dbt-sqlfluff-lint` execute [sqlfluff linter with dbt plugin](https://pypi.org/project/sqlfluff-templater-dbt/) to lint SQL and uses the following variables:

		\| Name \| Description \| Default value \|
		\| Input / Variable \| Description \| Default value \|
		\| ------------------------ \| -------------------------------------- \| ----------------- \|
		\| `DBT_SQLFLUFF_ENABLED` \| set to `true` to enable SQLFluff lint \| _none_ (disabled) \|
		\| `DBT_SQLFLUFF_LINT_ARGS` \| Lint [options and arguments](https://docs.sqlfluff.com/en/stable/cli.html#sqlfluff-lint) \| _none_ \|
		\| `sqlfluff-enabled` / `DBT_SQLFLUFF_ENABLED` \| set to `true` to enable SQLFluff lint \| _none_ (disabled) \|
		\| `sqlfluff-lint-args` / `DBT_SQLFLUFF_LINT_ARGS` \| Lint [options and arguments](https://docs.sqlfluff.com/en/stable/cli.html#sqlfluff-lint) \| _none_ \|

		:warning: this jobs read [SQLFluff configuration files](https://docs.sqlfluff.com/en/stable/configuration.html) in `DBT_PROJECT_DIR` directory.

		@@ -150,9 +177,9 @@ This job performs deployment.

		`dbt-deploy` execute generated SQL from models on target and uses the following variables:

		\| Name \| Description \| Default value \|
		\| Input / Variable \| Description \| Default value \|
		\| ------------------------ \| -------------------------------------- \| ----------------- \|
		\| `DBT_DEPLOY_ENABLED` \| set to `true` to enable deployment \| _none_ (disabled) \|
		\| `deploy-enabled` / `DBT_DEPLOY_ENABLED` \| set to `true` to enable deployment \| _none_ (disabled) \|

		### Secrets management

		@@ -178,15 +205,10 @@ Basically it copies the content of the dbt generated site folder (`target` by de
		If you wish to use it, add the following to your `gitlab-ci.yml`:

		```yaml
		include:
		# main template
		- project: 'to-be-continuous/dbt'
		ref: '3.1.1'
		file: '/templates/gitlab-ci-dbt.yml'
		- component: gitlab.com/to-be-continuous/dbt/gitlab-ci-dbt@3.1.1
		# GitLab pages variant
		- project: 'to-be-continuous/dbt'
		ref: '3.1.1'
		file: '/templates/gitlab-ci-dbt-pages.yml'
		- component: gitlab.com/to-be-continuous/dbt/gitlab-ci-dbt-pages@3.1.1
		```

		### Google Cloud variant
		@@ -199,42 +221,37 @@ Provided you successfully configured the [federated authentication using OpenID

		The variant requires the additional configuration parameters:

		\| Name \| Description \| Default value \|
		\| Input / Variable \| Description \| Default value \|
		\| ----------------- \| -------------------------------------- \| ----------------- \|
		\| `TBC_GCP_PROVIDER_IMAGE` \| The [GCP Auth Provider](https://gitlab.com/to-be-continuous/tools/gcp-auth-provider) image to use (can be overridden) \| `registry.gitlab.com/to-be-continuous/tools/gcp-auth-provider:main` \|
		\| `GCP_OIDC_PROVIDER` \| Default Workload Identity Provider associated with GitLab to [authenticate with OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/) \| _none_ \|
		\| `GCP_OIDC_ACCOUNT` \| Default Service Account to which impersonate with OpenID Connect authentication \| _none_ \|
		\| `GCP_REVIEW_OIDC_PROVIDER` \| Workload Identity Provider associated with GitLab to [authenticate with OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/) on `review` environment _(only define if different from default)_ \| _none_ \|
		\| `GCP_REVIEW_OIDC_ACCOUNT` \| Service Account to which impersonate with OpenID Connect authentication on `review` environment _(only define if different from default)_ \| _none_ \|
		\| `GCP_INTEG_OIDC_PROVIDER` \| Workload Identity Provider associated with GitLab to [authenticate with OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/) on `integration` environment _(only define if different from default)_ \| _none_ \|
		\| `GCP_INTEG_OIDC_ACCOUNT` \| Service Account to which impersonate with OpenID Connect authentication on `integration` environment _(only define if different from default)_ \| _none_ \|
		\| `GCP_STAGING_OIDC_PROVIDER` \| Workload Identity Provider associated with GitLab to [authenticate with OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/) on `staging` environment _(only define if different from default)_ \| _none_ \|
		\| `GCP_STAGING_OIDC_ACCOUNT` \| Service Account to which impersonate with OpenID Connect authentication on `staging` environment _(only define if different from default)_ \| _none_ \|
		\| `GCP_PROD_OIDC_PROVIDER` \| Workload Identity Provider associated with GitLab to [authenticate with OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/) on `production` environment _(only define if different from default)_ \| _none_ \|
		\| `GCP_PROD_OIDC_ACCOUNT` \| Service Account to which impersonate with OpenID Connect authentication on `production` environment _(only define if different from default)_ \| _none_ \|
		\| `gcp-oidc-aud` / `GCP_OIDC_AUD` \| The `aud` claim for the JWT token \| `$CI_SERVER_URL` \|
		\| `gcp-oidc-provider` / `GCP_OIDC_PROVIDER` \| Default Workload Identity Provider associated with GitLab to [authenticate with OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/) \| _none_ \|
		\| `gcp-oidc-account` / `GCP_OIDC_ACCOUNT` \| Default Service Account to which impersonate with OpenID Connect authentication \| _none_ \|
		\| `gcp-review-oidc-provider` / `GCP_REVIEW_OIDC_PROVIDER` \| Workload Identity Provider associated with GitLab to [authenticate with OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/) on `review` environment _(only define to override default)_ \| _none_ \|
		\| `gcp-review-oidc-account` / `GCP_REVIEW_OIDC_ACCOUNT` \| Service Account to which impersonate with OpenID Connect authentication on `review` environment _(only define to override default)_ \| _none_ \|
		\| `gcp-integ-oidc-provider` / `GCP_INTEG_OIDC_PROVIDER` \| Workload Identity Provider associated with GitLab to [authenticate with OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/) on `integration` environment _(only define to override default)_ \| _none_ \|
		\| `gcp-integ-oidc-account` / `GCP_INTEG_OIDC_ACCOUNT` \| Service Account to which impersonate with OpenID Connect authentication on `integration` environment _(only define to override default)_ \| _none_ \|
		\| `gcp-staging-oidc-provider` / `GCP_STAGING_OIDC_PROVIDER` \| Workload Identity Provider associated with GitLab to [authenticate with OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/) on `staging` environment _(only define to override default)_ \| _none_ \|
		\| `gcp-staging-oidc-account` / `GCP_STAGING_OIDC_ACCOUNT` \| Service Account to which impersonate with OpenID Connect authentication on `staging` environment _(only define to override default)_ \| _none_ \|
		\| `gcp-prod-oidc-provider` / `GCP_PROD_OIDC_PROVIDER` \| Workload Identity Provider associated with GitLab to [authenticate with OpenID Connect](https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/) on `production` environment _(only define to override default)_ \| _none_ \|
		\| `gcp-prod-oidc-account` / `GCP_PROD_OIDC_ACCOUNT` \| Service Account to which impersonate with OpenID Connect authentication on `production` environment _(only define to override default)_ \| _none_ \|

		#### Example

		With a common default `GCP_OIDC_PROVIDER` and `GCP_OIDC_ACCOUNT` configuration for non-prod environments, and a specific one for production:

		```yaml
		include:
		# main template
		- project: 'to-be-continuous/dbt'
		ref: '3.1.1'
		file: '/templates/gitlab-ci-dbt.yml'
		# `Google Cloud` variant
		- project: 'to-be-continuous/dbt'
		ref: '3.1.1'
		file: '/templates/gitlab-ci-dbt-gcp.yml'

		variables:
		- component: gitlab.com/to-be-continuous/dbt/gitlab-ci-dbt@3.1.1
		# Google Cloud variant
		- component: gitlab.com/to-be-continuous/dbt/gitlab-ci-dbt-gcp@3.1.1
		inputs:
		# common OIDC config for non-prod envs
		GCP_OIDC_PROVIDER: "projects/<gcp_nonprod_proj_id>/locations/global/workloadIdentityPools/<pool_id>/providers/<provider_id>"
		GCP_OIDC_ACCOUNT: "<name>@$<gcp_nonprod_proj_id>.iam.gserviceaccount.com"
		gcp-oidc-provider: "projects/<gcp_nonprod_proj_id>/locations/global/workloadIdentityPools/<pool_id>/providers/<provider_id>"
		gcp-oidc-account: "<name>@$<gcp_nonprod_proj_id>.iam.gserviceaccount.com"
		# specific OIDC config for prod
		GCP_PROD_OIDC_PROVIDER: "projects/<gcp_prod_proj_id>/locations/global/workloadIdentityPools/<pool_id>/providers/<provider_id>"
		GCP_PROD_OIDC_ACCOUNT: "<name>@$<gcp_prod_proj_id>.iam.gserviceaccount.com"
		gcp-prod-oidc-provider: "projects/<gcp_prod_proj_id>/locations/global/workloadIdentityPools/<pool_id>/providers/<provider_id>"
		gcp-prod-oidc-account: "<name>@$<gcp_prod_proj_id>.iam.gserviceaccount.com"
		```

		To setup dbt to use `GOOGLE_OAUTH_ACCESS_TOKEN`, apply this configuration in dbt `profiles.yml`:

bumpversion.sh

+2 −2

Original line number	Diff line number	Diff line
		@@ -27,13 +27,13 @@ if [[ "$curVer" ]]; then
		log_info "Bump version from \\e[33;1m${curVer}\\e[0m to \\e[33;1m${nextVer}\\e[0m (release type: $relType)..."

		# replace in README
		sed -e "s/ref: '$curVer'/ref: '$nextVer'/" README.md > README.md.next
		sed -e "s/ref: '$curVer'/ref: '$nextVer'/" -e "s/ref: \"$curVer\”/ref: \”$nextVer\”/" -e "s/component: $.$@$curVer/component: \1@$nextVer/" README.md > README.md.next
		mv -f README.md.next README.md

		# replace in template and variants
		for tmpl in templates/*.yml
		do
		sed -e "s/\"$curVer\"/\"$nextVer\"/" "$tmpl" > "$tmpl.next"
		sed -e "s/command: \[\"--service\", \"$.$\", \"$curVer\"\]/command: [\"--service\", \"\1\", \"$nextVer\"]/" "$tmpl" > "$tmpl.next"
		mv -f "$tmpl.next" "$tmpl"
		done
		else

kicker.json

+8 −0

Original line number	Diff line number	Diff line
		@@ -3,6 +3,8 @@
		"description": "Continuously integrate and deploy your data with [dbt](https://www.getdbt.com/)",
		"template_path": "templates/gitlab-ci-dbt.yml",
		"kind": "build",
		"prefix": "dbt",
		"is_component": true,
		"variables": [
		{
		"name": "DBT_IMAGE",
		@@ -135,6 +137,12 @@
		"default": "registry.gitlab.com/to-be-continuous/tools/vault-secrets-provider:master",
		"advanced": true
		},
		{
		"name": "GCP_OIDC_AUD",
		"description": "The `aud` claim for the JWT token _(only required for [OIDC authentication](https://docs.gitlab.com/ee/ci/cloud_services/aws/))_",
		"default": "$CI_SERVER_URL",
		"advanced": true
		},
		{
		"name": "GCP_OIDC_ACCOUNT",
		"description": "Default Service Account to which impersonate with OpenID Connect authentication"

logo.png

+4.48 KiB (13.6 KiB)

Loading image diff...