Commit d75f35e5 authored by Cédric OLIVIER's avatar Cédric OLIVIER
Browse files

Merge branch 'initial-build' into 'master' 

Initialize dbt template

See merge request to-be-continuous/dbt!1
parents b04f27e3 6a10488a

.gitlab-ci.yml

+4 −4
Original line number Diff line number Diff line
@@ -9,10 +9,10 @@ include: 
    ref: 'master'

    file: '/templates/validation.yml'

  - project: 'to-be-continuous/bash'

    ref: '2.0.0'

    ref: '3.0.1'

    file: 'templates/gitlab-ci-bash.yml'

  - project: 'to-be-continuous/semantic-release'

    ref: '2.0.2'

    ref: '3.0.0'

    file: '/templates/gitlab-ci-semrel.yml'



stages:
@@ -20,7 +20,7 @@ stages: 
  - publish



variables:

  GITLAB_CI_FILES: "templates/gitlab-ci-xxx.yml"

  GITLAB_CI_FILES: "templates/gitlab-ci-dbt.yml"

  BASH_SHELLCHECK_FILES: "*.sh"



semantic-release:

README.md

+123 −57
Original line number Diff line number Diff line 
# GitLab CI template for XXX

# GitLab CI template for dbt



This project implements a generic GitLab CI template for [XXX](https://link.to.tool.com/).

This project implements a generic GitLab CI template for [dbt](https://www.getdbt.com/).



It provides several features, usable in different modes (by configuration).



### Review environments



The template supports **review** environments: those are dynamic and ephemeral environments to deploy your

_ongoing developments_ (a.k.a. _feature_ or _topic_ branches).



When enabled, it deploys the result from upstream build stages to a dedicated environment.

It is only active for non-production, non-integration branches.



It is a strict equivalent of GitLab's [Review Apps](https://docs.gitlab.com/ee/ci/review_apps/) feature.



### Integration environment



If you're using a Git Workflow with an integration branch (such as [Gitflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow)),

the template supports an **integration** environment.



When enabled, it deploys the result from upstream build stages to a dedicated environment.

It is only active for your integration branch (`develop` by default).



### Production environments



Lastly, the template supports 2 environments associated to your production branch (`master` by default):



* a **staging** environment (an iso-prod environment meant for testing and validation purpose),

* the **production** environment.



You're free to enable whichever or both, and you can also choose your deployment-to-production policy:



* **continuous deployment**: automatic deployment to production (when the upstream pipeline is successful),

* **continuous delivery**: deployment to production can be triggered manually (when the upstream pipeline is successful).



## Usage



In order to include this template in your project, add the following to your `gitlab-ci.yml`:



```yaml

include:

  - project: 'to-be-continuous/xxx'

  - project: 'to-be-continuous/dbt'

    ref: '1.0.0'

    file: '/templates/gitlab-ci-xxx.yml'

    file: '/templates/gitlab-ci-dbt.yml'

```



You can find a sample of dbt project here : https://gitlab.com/to-be-continuous/samples/dbt-sample/



## Global configuration



The XXX template uses some global configuration used throughout all jobs.

The dbt template uses some global configuration used throughout all jobs.



| Name                  | description                            | default value     |

| --------------------- | -------------------------------------- | ----------------- |

| `XXX_IMAGE`           | The Docker image used to run XXX       | `xxx:latest` |

| `DBT_IMAGE`           | The Docker image used to run dbt       | `python:3.11`   |

| `DBT_PROJECT_DIR`     | The [dbt_project.yml](https://docs.getdbt.com/reference/dbt_project.yml) dir       | `.`   |

| `DBT_PROFILES_DIR`    | The dbt [profile](https://docs.getdbt.com/dbt-cli/configure-your-profile) location       | `.`   |

| `DBT_ADAPTER`         | The dbt [adapter](https://docs.getdbt.com/docs/available-adapters) to use       | __none__ (required)  |

| `DBT_TARGET`          | The dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) to use  |  __none__ (required)  |

| `DBT_BUILD_ARGS`      | Arguments used by [`dbt cli`](https://docs.getdbt.com/reference/global-configs#command-line-flags)      | __none__          |



## Jobs

### Environments configuration



As seen above, the dbt template may support up to 4 environments (`review`, `integration`, `staging` and `production`).



Each deployment job produces _output variables_ that are propagated to downstream jobs (using [dotenv artifacts](https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#artifactsreportsdotenv)):



### `xxx-build` job

* `environment_type`: set to the type of environment (`review`, `integration`, `staging` or `production`),

* `environment_name`: the application name (see below),



This job performs **build and tests** at once.

They may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).



It uses the following variable:

Here are configuration details for each environment.



#### Review environments



Review environments are dynamic and ephemeral environments to deploy your _ongoing developments_ (a.k.a. _feature_ or _topic_ branches).



They are **disabled by default** and can be enabled by setting the `DBT_REVIEW_TARGET` variable (see below).



Here are variables supported to configure review environments:



| Name                     | description                            | default value     |

| --------------------- | ---------------------------------------- | ----------------- |

| `XXX_BUILD_ARGS`      | Arguments used by the build job          | `build --with-default-args` |

| ------------------------ | -------------------------------------- | ----------------- |

| `DBT_REVIEW_TARGET`      | dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) for `review` env | _none_ (disabled) |



### SonarQube analysis



If you're using the SonarQube template to analyse your XXX code, here are 2 sample `sonar-project.properties` files.

#### Integration environment



```properties

# see: https://docs.sonarqube.org/latest/analysis/languages/xxx/

# set your source directory(ies) here (relative to the sonar-project.properties file)

sonar.sources=.

# exclude unwanted directories and files from being analysed

sonar.exclusions=output/**,**/*_test.xxx

The integration environment is the environment associated to your integration branch (`develop` by default).



# set your tests directory(ies) here (relative to the sonar-project.properties file)

sonar.tests=.

sonar.test.inclusions=**/*_test.xxx

It is **disabled by default** and can be enabled by setting the `DBT_INTEG_TARGET` variable (see below).



# tests report (TODO)

sonar.xxx.testExecutionReportPaths=reports/sonar_test_report.xml

# coverage report (TODO)

sonar.xxx.coverage.reportPaths=reports/coverage.cov

```

Here are variables supported to configure the integration environment:



More info:

| Name                     | description                            | default value     |

| ------------------------ | -------------------------------------- | ----------------- |

| `DBT_INTEG_TARGET`       | dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) for `integration` env | _none_ (disabled) |



* [XXX language support](https://docs.sonarqube.org/latest/analysis/languages/xxx/)

* [test coverage & execution parameters](https://docs.sonarqube.org/latest/analysis/coverage/)

* [third-party issues](https://docs.sonarqube.org/latest/analysis/external-issues/)

#### Staging environment



### `xxx-lint` job

The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production branch (`master` by default).



This job performs a [lint](link-to-the-tool) analysis of your code, mapped to the `build` stage.

It is **disabled by default** and can be enabled by setting the `DBT_STAGING_TARGET` variable (see below).



It uses the following variables:

Here are variables supported to configure the staging environment:



| Name                     | description                            | default value     |

| --------------------- | ------------------------------------------ | ----------------- |

| `XXX_LINT_IMAGE`      | The Docker image used to run the lint tool | `xxx-lint:latest` |

| `XXX_LINT_ARGS`       | Lint [options and arguments](link-to-the-cli-options) | `--serevity=medium` |

| ------------------------ | -------------------------------------- | ----------------- |

| `DBT_STAGING_TARGET`     | dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) for `staging` env | _none_ (disabled) |



#### Production environment



### `xxx-depcheck` job

The production environment is the final deployment environment associated with your production branch (`master` by default).



This job enables a manual [dependency check](link-to-the-tool) analysis of your code, mapped to the `test` stage.

It is **disabled by default** and can be enabled by setting the `DBT_PROD_TARGET` variable (see below).



It uses the following variables:

Here are variables supported to configure the production environment:



| Name                     | description                            | default value     |

| --------------------- | ------------------------------------------ | ----------------- |

| `XXX_DEPCHECK_IMAGE`  | The Docker image used to run the dependency check tool | `xxx-depcheck:latest` |

| `XXX_DEPCHECK_ARGS`   | Dependency check [options and arguments](link-to-the-cli-options) | _none_ |

| ------------------------ | -------------------------------------- | ----------------- |

| `DBT_PROD_TARGET`        | dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) for `production` env | _none_ (disabled) |





## Jobs



### `dbt-build` job



### `xxx-publish` job

This job performs **build, doc generation** and documentation **coverage**.



This job is **disabled by default** and performs a publish of your built binaries.

`dbt-build` generates executable SQL from source model, test, and analysis files



It uses the following variables:



### `dbt-deploy` job



This job performs **deployment**.



`dbt-deploy` execute generated SQL from models on target and uses the following variables:



| Name                     | description                            | default value     |

| --------------------- | -------------------------------------- | ----------------- |

| `XXX_PUBLISH_ENABLED` | Variable to enable the publish job     | _none_ (disabled) |

| `XXX_PUBLISH_ARGS`    | Arguments used by the publish job      | `publish --with-default-args` |

| :lock: `XXX_PUBLISH_LOGIN` | Login to use to publish           | **has to be defined** |

| :lock: `XXX_PUBLISH_PASSWORD` | Password to use to publish     | **has to be defined** |

| ------------------------ | -------------------------------------- | ----------------- |

| `DBT_DEPLOY_ENABLED`     | set to `true` to enable deployment     | _none_ (disabled) |



### Secrets management
@@ -110,3 +155,24 @@ Here are some advices about your **secrets** (variables marked with a :lock:): 
  simply define its value as the [Base64](https://en.wikipedia.org/wiki/Base64) encoded value prefixed with `@b64@`:

  it will then be possible to mask it and the template will automatically decode it prior to using it.

3. Don't forget to escape special characters (ex: `$` -> `$$`).





## Variants



### GitLab Pages variant



Basically it copies the content of the dbt generated site folder (`target` by default) to the `public` folder which is published by [GitLab pages](https://docs.gitlab.com/ee/user/project/pages/#how-it-works).



If you wish to use it, add the following to your `gitlab-ci.yml`:



```yaml

include:

  # main template

  - project: 'to-be-continuous/dbt'

    ref: '1.0.0'

    file: '/templates/gitlab-ci-dbt.yml'

  # GitLab pages variant

  - project: 'to-be-continuous/dbt'

    ref: '1.0.0'

    file: '/templates/gitlab-ci-dbt-pages.yml'

```
 
 No newline at end of file

kicker.json

+71 −49
Original line number Diff line number Diff line 
{

  "name": "XXX",

  "description": "GitLab CI template for XXX",

  "template_path": "templates/gitlab-ci-xxx.yml",

  "name": "dbt",

  "description": "GitLab CI template for dbt",

  "template_path": "templates/gitlab-ci-dbt.yml",

  "kind": "build",

  "variables": [

    {

      "name": "XXX_IMAGE",

      "description": "The Docker image used to run XXX",

      "default": "xxx:1.2.3"

      "name": "DBT_IMAGE",

      "description": "The Docker image used to run dbt",

      "default": "python:3.11"

    },

    {

      "name": "XXX_BUILD_ARGS",

      "description": "Arguments used by the build job",

      "default": "build --with-default-args",

      "name": "DBT_PROJECT_DIR",

      "description": "The [dbt_project.yml](https://docs.getdbt.com/reference/dbt_project.yml) dir",

      "default": "."

    },

    {

      "name": "DBT_PROFILES_DIR",

      "description": "The dbt [profile](https://docs.getdbt.com/dbt-cli/configure-your-profile) location",

      "default": "."

    },

    {

      "name": "DBT_ADAPTER",

      "description": "The dbt [adapter](https://docs.getdbt.com/docs/available-adapters) to use ",

      "mandatory": true

    },

    {

      "name": "DBT_TARGET",

      "description": "The dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) to use "

    },

    {

      "name": "DBT_BUILD_ARGS",

      "description": "Arguments used by [dbt cli](https://docs.getdbt.com/reference/global-configs#command-line-flags) ",

      "advanced": true

    }

  ],

  "features": [

    {

      "id": "lint",

      "name": "XXX lint",

      "description": "[XXX lint](link-to-the-tool) analysis",

      "variables": [

        {

          "name": "XXX_LINT_IMAGE",

          "description": "The Docker image used to run the lint tool",

          "default": "xxx-lint:latest"

      "id": "dbt-deploy",

      "name": "dbt deploy",

      "description": "execute generated SQL from models on target",

      "enable_with": "DBT_DEPLOY_ENABLED"

    },

    {

          "name": "XXX_LINT_ARGS",

          "description": "Lint [options and arguments](link-to-the-cli-options)",

          "default": "--serevity=medium",

          "advanced": true

      "id": "review",

      "name": "Review",

      "description": "Dynamic review environments for your topic branches (see GitLab [Review Apps](https://docs.gitlab.com/ee/ci/review_apps/))",

      "variables": [

        {

          "name": "DBT_REVIEW_TARGET",

          "description": "dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) for `review` env",

          "mandatory": true

        }

      ]

    },

    {

      "id": "depcheck",

      "name": "XXX dependency check",

      "description": "[XXX dependency check](link-to-the-tool) analysis",

      "id": "integration",

      "name": "Integration",

      "description": "A continuous-integration environment associated to your integration branch (`develop` by default)",

      "variables": [

        {

          "name": "XXX_DEPCHECK_IMAGE",

          "description": "The Docker image used to run the dependency check tool",

          "default": "xxx-depcheck:latest"

        },

        {

          "name": "XXX_DEPCHECK_ARGS",

          "description": "Dependency check [options and arguments](link-to-the-cli-options)",

          "advanced": true

          "name": "DBT_INTEG_TARGET",

          "description": " dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) for `integration` env",

          "mandatory": true

        }

      ]

    },

    {

      "id": "publish",

      "name": "Publish",

      "description": "Publish your package to a repository",

      "enable_with": "XXX_PUBLISH_ENABLED",

      "id": "staging",

      "name": "Staging",

      "description": "An iso-prod environment meant for testing and validation purpose on your production branch (`master` by default)",

      "variables": [

        {

          "name": "XXX_PUBLISH_ARGS",

          "description": "Arguments used by the publish job",

          "default": "publish --with-default-args",

          "advanced": true

          "name": "DBT_STAGING_TARGET",

          "description": "dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) for `staging` env",

          "mandatory": true

        }

      ]

    },

    {

          "name": "XXX_PUBLISH_LOGIN",

          "description": "Login to use to publish",

          "secret": true

        },

      "id": "prod",

      "name": "Production",

      "description": "The production environment",

      "variables": [

        {

          "name": "XXX_PUBLISH_PASSWORD",

          "description": "Password to use to publish",

          "secret": true

          "name": "DBT_PROD_TARGET",

          "description": "dbt [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) for `production` env ",

          "mandatory": true

        }

      ]

    }

  ],

  "variants": [

    {

      "id": "pages",

      "name": "GitLab Pages",

      "description": "Adds a job to publish the generated site to GitLab pages",

      "template_path": "templates/gitlab-ci-dbt-pages.yml"

    }

  ]



}

logo.png

−1.9 KiB (9.07 KiB)
Loading image diff...

templates/gitlab-ci-dbt-pages.yml

0 → 100644
+53 −0
Original line number Diff line number Diff line 
# ==================================================

# Variables definition

# ==================================================

variables:

  # default production ref name (pattern)

  PROD_REF: '/^(master|main)$/'



# ==================================================

# Stages definition

# ==================================================

stages:

  - production



.dbt-scripts: &dbt-scripts |

  # BEGSCRIPT

  set -e



  function log_info() {

      echo -e "[\\e[1;94mINFO\\e[0m] $*"

  }



  function log_warn() {

      echo -e "[\\e[1;93mWARN\\e[0m] $*"

  }



  function log_error() {

      echo -e "[\\e[1;91mERROR\\e[0m] $*"

  }



  # ENDSCRIPT







pages:

  stage: production

  before_script:

    - *dbt-scripts

  script:

    - target="public/"

    - |

        log_info "copying dbt docs generated site from \\e[33;1m${DBT_PROJECT_DIR}/target\\e[0m to pages expected folder \\e[33;1m${target}\\e[0m"

        mkdir -p public

        cp -r ${DBT_PROJECT_DIR}/target/* ${target}

  artifacts:

    paths:

      - public

    expire_in: 1 days

  environment:

    name: pages

    url: ${CI_PAGES_URL}

  rules:

    # on production branch(es)

    - if: '$CI_COMMIT_REF_NAME =~ $PROD_REF'
 
 No newline at end of file