Commit 0166bd43 authored by Cédric OLIVIER's avatar Cédric OLIVIER Committed by Pierre Smeyers
Browse files

feat: support CI/CD component design 

See: https://docs.gitlab.com/ee/ci/components/
parent b767b8f9

README.md

+76 −54
Original line number Diff line number Diff line
@@ -4,30 +4,55 @@ This project implements a GitLab CI/CD template to build, test and analyse your 


## Usage



In order to include this template in your project, add the following to your `gitlab-ci.yml`:

This template can be used both as a [CI/CD component](https://docs.gitlab.com/ee/ci/components/#use-a-component-in-a-cicd-configuration) 

or using the legacy [`include:project`](https://docs.gitlab.com/ee/ci/yaml/index.html#includeproject) syntax.



### Use as a CI/CD component



Add the following to your `gitlab-ci.yml`:



```yaml

include:

  # 1: include the component

  - component: gitlab.com/to-be-continuous/python/gitlab-ci-python@6.4.1

    # 2: set/override component inputs

    inputs:

      image: registry.hub.docker.com/library/python:3.10

      pytest-enabled: true

```



### Use as a CI/CD template (legacy)



Add the following to your `gitlab-ci.yml`:



```yaml

include:

  # 1: include the template

  - project: 'to-be-continuous/python'

    ref: '6.4.1'

    file: '/templates/gitlab-ci-python.yml'



variables:

  # 2: set/override template variables

  PYTHON_IMAGE: registry.hub.docker.com/library/python:3.10

  PYTEST_ENABLED: "true"

```



## Global configuration



The Python template uses some global configuration used throughout all jobs.



| Name                 | description                                                                           | default value      |

| Input / Variable     | Description                                                                           | Default value      |

| -------------------- | ------------------------------------------------------------------------------------- | ------------------ |

| `PYTHON_IMAGE`       | The Docker image used to run Python <br/>:warning: **set the version required by your project** | `registry.hub.docker.com/library/python:3` |

| `PYTHON_PROJECT_DIR` | Python project root directory                                                         | `.`                |

| `PYTHON_BUILD_SYSTEM`| Python build-system to use to install dependencies, build and package the project (see below) | _none_ (auto-detect) |

| `image` / `PYTHON_IMAGE` | The Docker image used to run Python <br/>:warning: **set the version required by your project** | `registry.hub.docker.com/library/python:3` |

| `project-dir` / `PYTHON_PROJECT_DIR` | Python project root directory                                                         | `.`                |

| `build-system` / `PYTHON_BUILD_SYSTEM`| Python build-system to use to install dependencies, build and package the project (see below) | _none_ (auto-detect) |

| `PIP_INDEX_URL`      | Python repository url                                                                 | _none_             |

| `PIP_EXTRA_INDEX_URL` | Exra Python repository url                                                           | _none_             |

| `PIP_OPTS`           | pip [extra options](https://pip.pypa.io/en/stable/cli/pip/#general-options)     | _none_             |

| `PYTHON_EXTRA_DEPS`  | Python extra sets of dependencies to install<br/>For [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies) or [Poetry](https://python-poetry.org/docs/pyproject/#extras) only | _none_ |

| `PYTHON_REQS_FILE`   | Main requirements file _(relative to `$PYTHON_PROJECT_DIR`)_<br/>For [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only | `requirements.txt` |

| `PYTHON_EXTRA_REQS_FILES` | Extra dev requirements file(s) to install _(relative to `$PYTHON_PROJECT_DIR`)_ | `requirements-dev.txt` |

| `pip-opts` / `PIP_OPTS` | pip [extra options](https://pip.pypa.io/en/stable/cli/pip/#general-options)     | _none_             |

| `extra-deps` / `PYTHON_EXTRA_DEPS` | Python extra sets of dependencies to install<br/>For [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies) or [Poetry](https://python-poetry.org/docs/pyproject/#extras) only | _none_ |

| `reqs-file` / `PYTHON_REQS_FILE` | Main requirements file _(relative to `$PYTHON_PROJECT_DIR`)_<br/>For [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only | `requirements.txt` |

| `extra-reqs-files` / `PYTHON_EXTRA_REQS_FILES` | Extra dev requirements file(s) to install _(relative to `$PYTHON_PROJECT_DIR`)_ | `requirements-dev.txt` |



The cache policy also makes the necessary to manage pip cache (not to download Python dependencies over and over again).
@@ -66,10 +91,10 @@ It is activated by setting `$PYLINT_ENABLED` to `true`. 


It is bound to the `build` stage, and uses the following variables:



| Name                     | description                        | default value     |

| Input / Variable         | Description                        | Default value     |

| ------------------------ | ---------------------------------- | ----------------- |

| `PYLINT_ARGS`            | Additional [pylint CLI options](http://pylint.pycqa.org/en/latest/user_guide/run.html#command-line-options) |  _none_           |

| `PYLINT_FILES`           | Files or directories to analyse   | _none_ (by default analyses all found python source files) |

| `pylint-args` / `PYLINT_ARGS` | Additional [pylint CLI options](http://pylint.pycqa.org/en/latest/user_guide/run.html#command-line-options) |  _none_           |

| `pylint-files` / `PYLINT_FILES` | Files or directories to analyse   | _none_ (by default analyses all found python source files) |



In addition to a textual report in the console, this job produces the following reports, kept for one day:
@@ -96,9 +121,9 @@ In order to produce JUnit test reports, the tests are executed with the [xmlrunn 


It is bound to the `build` stage, and uses the following variables:



| Name                     | description                                                          | default value           |

| Input / Variable         | Description                                                          | Default value           |

| ------------------------ | -------------------------------------------------------------------- | ----------------------- |

| `UNITTEST_ARGS`          | Additional xmlrunner/unittest CLI options                            | _none_                  |

| `unittest-args` / `UNITTEST_ARGS` | Additional xmlrunner/unittest CLI options                            | _none_                  |



:information_source: use a `.coveragerc` file at the root of your Python project to control the coverage settings.
@@ -128,9 +153,9 @@ It is activated by setting `$PYTEST_ENABLED` to `true`. 


It is bound to the `build` stage, and uses the following variables:



| Name                     | description                                                                                                                                   | default value           |

| Input / Variable         | Description                                                                                                                                   | Default value           |

| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |

| `PYTEST_ARGS`            | Additional [pytest](https://docs.pytest.org/en/stable/usage.html) or [pytest-cov](https://github.com/pytest-dev/pytest-cov#usage) CLI options | _none_                  |

| `pytest-args` / `PYTEST_ARGS` | Additional [pytest](https://docs.pytest.org/en/stable/usage.html) or [pytest-cov](https://github.com/pytest-dev/pytest-cov#usage) CLI options | _none_                  |



:information_source: use a `.coveragerc` file at the root of your Python project to control the coverage settings.
@@ -160,9 +185,9 @@ It is activated by setting `$NOSETESTS_ENABLED` to `true`. 


It is bound to the `build` stage, and uses the following variables:



| Name                     | description                                                                             | default value           |

| Input / Variable         | Description                                                                             | Default value           |

| ------------------------ | --------------------------------------------------------------------------------------- | ----------------------- |

| `NOSETESTS_ARGS`         | Additional [nose CLI options](https://nose.readthedocs.io/en/latest/usage.html#options) | _none_                  |

| `nosetests-args` / `NOSETESTS_ARGS` | Additional [nose CLI options](https://nose.readthedocs.io/en/latest/usage.html#options) | _none_                  |



By default coverage will be run on all the project directories. You can restrict it to your packages by setting the `$NOSE_COVER_PACKAGE` variable.

More [info](https://nose.readthedocs.io/en/latest/plugins/cover.html)
@@ -183,9 +208,9 @@ are not set), and performs a [`compileall`](https://docs.python.org/3/library/co 


It is bound to the `build` stage, and uses the following variables:



| Name                  | description                                                                   | default value |

| Input / Variable      | Description                                                                   | Default value |

| --------------------- | ----------------------------------------------------------------------------- | ------------- |

| `PYTHON_COMPILE_ARGS` | [`compileall` CLI options](https://docs.python.org/3/library/compileall.html) | `*`           |

| `compile-args` / `PYTHON_COMPILE_ARGS` | [`compileall` CLI options](https://docs.python.org/3/library/compileall.html) | `*`           |



### `py-bandit` job (SAST)
@@ -193,10 +218,10 @@ This job is **disabled by default** and performs a [Bandit](https://pypi.org/pro 


It is bound to the `test` stage, and uses the following variables:



| Name             | description                                                            | default value     |

| Input / Variable | Description                                                            | Default value     |

| ---------------- | ---------------------------------------------------------------------- | ----------------- |

| `BANDIT_ENABLED` | Set to `true` to enable Bandit analysis                                | _none_ (disabled) |

| `BANDIT_ARGS`    | Additional [Bandit CLI options](https://github.com/PyCQA/bandit#usage) | `--recursive .`   |

| `bandit-enabled` / `BANDIT_ENABLED` | Set to `true` to enable Bandit analysis                                | _none_ (disabled) |

| `bandit-args` / `BANDIT_ARGS`    | Additional [Bandit CLI options](https://github.com/PyCQA/bandit#usage) | `--recursive .`   |



In addition to a textual report in the console, this job produces the following reports, kept for one day:
@@ -211,10 +236,10 @@ This job is **disabled by default** and performs a dependency check analysis usi 


It is bound to the `test` stage, and uses the following variables:



| Name             | description                                                             | default value     |

| Input / Variable | Description                                                             | Default value     |

| ---------------- | ----------------------------------------------------------------------- | ----------------- |

| `SAFETY_ENABLED` | Set to `true` to enable Safety job                                      | _none_ (disabled) |

| `SAFETY_ARGS`    | Additional [Safety CLI options](https://github.com/pyupio/safety#usage) | `--full-report`   |

| `safety-enabled` / `SAFETY_ENABLED` | Set to `true` to enable Safety job                                      | _none_ (disabled) |

| `safety-args` / `SAFETY_ARGS`    | Additional [Safety CLI options](https://github.com/pyupio/safety#usage) | `--full-report`   |



### `py-trivy` job (dependency check)
@@ -222,10 +247,10 @@ This job is **disabled by default** and performs a dependency check analysis usi 


It is bound to the `test` stage, and uses the following variables:



| Name             | description                                                             | default value     |

| Input / Variable | Description                                                             | Default value     |

| ---------------- | ----------------------------------------------------------------------- | ----------------- |

| `PYTHON_TRIVY_ENABLED` | Set to `true` to enable Trivy job                                 | _none_ (disabled) |

| `PYTHON_TRIVY_ARGS`    | Additional [Trivy CLI options](https://aquasecurity.github.io/trivy/v0.21.1/getting-started/cli/fs/) | `--vuln-type library`   |

| `trivy-enabled` / `PYTHON_TRIVY_ENABLED` | Set to `true` to enable Trivy job                                 | _none_ (disabled) |

| `trivy-args` / `PYTHON_TRIVY_ARGS`       | Additional [Trivy CLI options](https://aquasecurity.github.io/trivy/v0.21.1/getting-started/cli/fs/) | `--vuln-type library`   |



In addition to a textual report in the console, this job produces the following reports, kept for one day:
@@ -239,12 +264,12 @@ This job generates a [SBOM](https://cyclonedx.org/) file listing all dependencie 


It is bound to the `test` stage, and uses the following variables:



| Name                  | description                            | default value     |

| Input / Variable      | Description                            | Default value     |

| --------------------- | -------------------------------------- | ----------------- |

| `PYTHON_SBOM_DISABLED` | Set to `true` to disable this job | _none_ |

| `PYTHON_SBOM_SYFT_URL` | Url to the `tar.gz` package for `linux_amd64` of Syft to use (ex: `https://github.com/anchore/syft/releases/download/v0.62.3/syft_0.62.3_linux_amd64.tar.gz`)<br/>_When unset, the latest version will be used_ | _none_ |

| `PYTHON_SBOM_NAME` | Component name of the emitted SBOM | `$CI_PROJECT_PATH/$PYTHON_PROJECT_DIR` |

| `PYTHON_SBOM_OPTS` | Options for syft used for SBOM analysis | `--catalogers python-index-cataloger` |

| `sbom-disabled` / `PYTHON_SBOM_DISABLED` | Set to `true` to disable this job | _none_ |

| `sbom-syft-url` / `PYTHON_SBOM_SYFT_URL` | Url to the `tar.gz` package for `linux_amd64` of Syft to use (ex: `https://github.com/anchore/syft/releases/download/v0.62.3/syft_0.62.3_linux_amd64.tar.gz`)<br/>_When unset, the latest version will be used_ | _none_ |

| `sbom-name` / `PYTHON_SBOM_NAME` | Component name of the emitted SBOM | `$CI_PROJECT_PATH/$PYTHON_PROJECT_DIR` |

| `sbom-opts` / `PYTHON_SBOM_OPTS` | Options for syft used for SBOM analysis | `--catalogers python-index-cataloger` |



In addition to logs in the console, this job produces the following reports, kept for one week:
@@ -299,16 +324,16 @@ The Python template supports two packaging systems: 


The release job is bound to the `publish` stage, appears only on production and integration branches and uses the following variables:



| Name                    | description                                                             | default value     |

| Input / Variable        | Description                                                             | Default value     |

| ----------------------- | ----------------------------------------------------------------------- | ----------------- |

| `PYTHON_RELEASE_ENABLED`| Set to `true` to enable the release job                                 | _none_ (disabled) |

| `PYTHON_RELEASE_NEXT`   | The part of the version to increase (one of: `major`, `minor`, `patch`) | `minor`           |

| `PYTHON_SEMREL_RELEASE_DISABLED`| Set to `true` to disable [semantic-release integration](#semantic-release-integration)   | _none_ (disabled) |

| `release-enabled` / `PYTHON_RELEASE_ENABLED`| Set to `true` to enable the release job                                 | _none_ (disabled) |

| `release-next` / `PYTHON_RELEASE_NEXT`   | The part of the version to increase (one of: `major`, `minor`, `patch`) | `minor`           |

| `semrel-release-disabled` / `PYTHON_SEMREL_RELEASE_DISABLED`| Set to `true` to disable [semantic-release integration](#semantic-release-integration)   | _none_ (disabled) |

| `GIT_USERNAME`          | Git username for Git push operations (see below)                        | _none_            |

| :lock: `GIT_PASSWORD`   | Git password for Git push operations (see below)                        | _none_            |

| :lock: `GIT_PRIVATE_KEY`| SSH key for Git push operations (see below)                             | _none_            |

| `PYTHON_RELEASE_COMMIT_MESSAGE`| The Git commit message to use on the release commit. This is templated using the [Python Format String Syntax](http://docs.python.org/2/library/string.html#format-string-syntax). Available in the template context are current_version and new_version. | `chore(python-release): {current_version} → {new_version}` |

| `PYTHON_REPOSITORY_URL`| Target PyPI repository to publish packages                              | _[GitLab project's PyPI packages repository](https://docs.gitlab.com/ee/user/packages/pypi_repository/)_ |

| `release-commit-message` / `PYTHON_RELEASE_COMMIT_MESSAGE`| The Git commit message to use on the release commit. This is templated using the [Python Format String Syntax](http://docs.python.org/2/library/string.html#format-string-syntax). Available in the template context are current_version and new_version. | `chore(python-release): {current_version} → {new_version}` |

| `repository-url` / `PYTHON_REPOSITORY_URL`| Target PyPI repository to publish packages                              | _[GitLab project's PyPI packages repository](https://docs.gitlab.com/ee/user/packages/pypi_repository/)_ |

| `PYTHON_REPOSITORY_USERNAME`| Target PyPI repository username credential                              | `gitlab-ci-token` |

| :lock: `PYTHON_REPOSITORY_PASSWORD`| Target PyPI repository password credential                              | `$CI_JOB_TOKEN` |
@@ -359,7 +384,7 @@ You can either use a SSH key or user/password credentials. 


We recommend you to use a [project deploy key](https://docs.gitlab.com/ee/user/project/deploy_keys/#project-deploy-keys) with write access to your project.



The key should not have a passphrase (see [how to generate a new SSH key pair](https://docs.gitlab.com/ce/ssh/README.html#generating-a-new-ssh-key-pair)).

The key should not have a passphrase (see [how to generate a new SSH key pair](https://docs.gitlab.com/ee/user/ssh.html#generate-an-ssh-key-pair)).



Specify :lock: `$GIT_PRIVATE_KEY` as secret project variable with the private part of the deploy key.
@@ -412,11 +437,11 @@ This variant allows delegating your secrets management to a [Vault](https://www. 


In order to be able to communicate with the Vault server, the variant requires the additional configuration parameters:



| Name              | Description                            | Default value     |

| Input / Variable  | Description                            | Default value     |

| ----------------- | -------------------------------------- | ----------------- |

| `TBC_VAULT_IMAGE` | The [Vault Secrets Provider](https://gitlab.com/to-be-continuous/tools/vault-secrets-provider) image to use (can be overridden) | `$CI_REGISTRY/to-be-continuous/tools/vault-secrets-provider:master` |

| `VAULT_BASE_URL`  | The Vault server base API url          | _none_ |

| `VAULT_OIDC_AUD`  | The `aud` claim for the JWT | `$CI_SERVER_URL` |

| `vault-base-url` / `VAULT_BASE_URL`  | The Vault server base API url          | _none_ |

| `vault-oidc-aud` / `VAULT_OIDC_AUD`  | The `aud` claim for the JWT | `$CI_SERVER_URL` |

| :lock: `VAULT_ROLE_ID`   | The [AppRole](https://www.vaultproject.io/docs/auth/approle) RoleID | **must be defined** |

| :lock: `VAULT_SECRET_ID` | The [AppRole](https://www.vaultproject.io/docs/auth/approle) SecretID | **must be defined** |
@@ -439,22 +464,19 @@ With: 


```yaml

include:

  # main template

  - project: 'to-be-continuous/python'

    ref: '6.4.1'

    file: '/templates/gitlab-ci-python.yml'

  # main component

  - component: gitlab.com/to-be-continuous/python/gitlab-ci-python@6.4.1

  # Vault variant

  - project: 'to-be-continuous/python'

    ref: '6.4.1'

    file: '/templates/gitlab-ci-python-vault.yml'

  - component: gitlab.com/to-be-continuous/python/gitlab-ci-python-vault@6.4.1

    inputs:

      vault-base-url: "https://vault.acme.host/v1"

      # audience claim for JWT

      vault-oidc-aud: "https://vault.acme.host"



variables:

    # audience claim for JWT

    VAULT_OIDC_AUD: "https://vault.acme.host"

    # Secrets managed by Vault

    GIT_PASSWORD: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/git/semantic-release?field=group-access-token"

    GIT_PRIVATE_KEY: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/git/semantic-release?field=private-key"

    PYTHON_REPOSITORY_PASSWORD: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/pip-repo/repository?field=password"

    VAULT_BASE_URL: "https://vault.acme.host/v1"

    # $VAULT_ROLE_ID and $VAULT_SECRET_ID defined as a secret CI/CD variable

```

bumpversion.sh

+1 −1
Original line number Diff line number Diff line
@@ -27,7 +27,7 @@ if [[ "$curVer" ]]; then 
  log_info "Bump version from \\e[33;1m${curVer}\\e[0m to \\e[33;1m${nextVer}\\e[0m (release type: $relType)..."



  # replace in README

  sed -e "s/ref: '$curVer'/ref: '$nextVer'/" README.md > README.md.next

  sed -e "s/ref: '$curVer'/ref: '$nextVer'/" -e "s/@$curVer/@$nextVer/" README.md > README.md.next

  mv -f README.md.next README.md



  # replace in template and variants

kicker.json

+6 −0
Original line number Diff line number Diff line
@@ -148,6 +148,12 @@ 
      "description": "Detect security vulnerabilities with [Trivy](https://github.com/aquasecurity/trivy/) (dependencies analysis)",

      "enable_with": "PYTHON_TRIVY_ENABLED",

      "variables": [

        {

          "name": "PYTHON_TRIVY_IMAGE",

          "description": "The Docker image used to run Trivy",

          "default": "registry.hub.docker.com/aquasec/trivy:latest",

          "advanced": true

        },

        {

          "name": "PYTHON_TRIVY_ARGS",

          "description": "Additional [Trivy CLI options](https://aquasecurity.github.io/trivy/v0.21.1/getting-started/cli/fs/)",

templates/gitlab-ci-python-vault.yml

+12 −2
Original line number Diff line number Diff line 
# =====================================================================================================================

# === Vault template variant

# =====================================================================================================================

spec:

  inputs:

    vault-base-url:

      description: The Vault server base API url

      default: '' # null

    vault-oidc-aud:

      description: The `aud` claim for the JWT

      default: $CI_SERVER_URL

---

variables:

  # variabilized vault-secrets-provider image

  TBC_VAULT_IMAGE: "$CI_REGISTRY/to-be-continuous/tools/vault-secrets-provider:master"

  TBC_VAULT_IMAGE: $CI_REGISTRY/to-be-continuous/tools/vault-secrets-provider:master

  # variables have to be explicitly declared in the YAML to be exported to the service

  VAULT_ROLE_ID: "$VAULT_ROLE_ID"

  VAULT_SECRET_ID: "$VAULT_SECRET_ID"

  VAULT_OIDC_AUD: "$CI_SERVER_URL"

  VAULT_OIDC_AUD: $[[ inputs.vault-oidc-aud ]]

  VAULT_BASE_URL: $[[ inputs.vault-base-url ]]



.python-base:

  services:

templates/gitlab-ci-python.yml

+183 −15

File changed.

Preview size limit exceeded, changes collapsed.