Commit 347297bb authored by Pierre Smeyers's avatar Pierre Smeyers
Browse files

Merge branch 'feat/support-parallel-matrix' into 'main' 

Add parallel:matrix support

See merge request to-be-continuous/docker!190
parents ed44bcb2 3c1a0bb3

README.md

+32 −5
Original line number Diff line number Diff line
@@ -76,6 +76,7 @@ The Docker template uses some global configuration used throughout all jobs. 
| `dind-image` / `DOCKER_DIND_IMAGE`       | The Docker image used to run the Docker daemon (see [full list](https://hub.docker.com/r/library/docker/)) - _for Docker-in-Docker build only_ | `docker.io/library/docker:dind`<br/>[![Trivy Badge](https://to-be-continuous.gitlab.io/doc/secu/trivy-badge-DOCKER_DIND_IMAGE.svg)](https://to-be-continuous.gitlab.io/doc/secu/trivy-DOCKER_DIND_IMAGE)|

| `file` / `DOCKER_FILE`                   | The path to your `Dockerfile`                                                                                                                  | `Dockerfile`                                                                        |

| `context-path` / `DOCKER_CONTEXT_PATH`   | The Docker [context path](https://docs.docker.com/engine/reference/commandline/build/#build-with-path) (working directory)                     | _none_ _only set if you want a context path different from the Dockerfile location_ |

| `artifacts-namespace` / `DOCKER_ARTIFACTS_NAMESPACE` | Output artifacts namespace _(only required when deploying [multiple environments](#multiple-environments-support))_ | _none_ |



In addition to this, the template supports _standard_ Linux proxy variables:
@@ -261,12 +262,13 @@ The signing process can be configured with the following variables: 
| :lock: `DOCKER_COSIGN_PRIVATE_KEY` | Private key used for signing the Docker image and the attestation | _none_ |

| :lock: `COSIGN_PASSWORD`              | Password of the private key | _none_ |



## Multi Dockerfile support

### Multiple images build



This template supports building multiple Docker images from a single Git repository.



You can define the images to build using the [parallel matrix jobs](https://docs.gitlab.com/ci/yaml/#parallelmatrix)

pattern inside the `.docker-base` job (this is the top parent job of all Docker template jobs).

:information: Output variables namespacing can be ensured by using the `DOCKER_ARTIFACTS_NAMESPACE` variable.



Since each job in the template extends this base job, the pipeline will produce one job instance per image to build.

You can independently configure each instance of these jobs by redefining the variables described throughout this
@@ -280,11 +282,13 @@ You can do so by adding a patch to the `.docker-base` job in your `.gitlab-ci.ym 
.docker-base:

  parallel:

    matrix:

      - DOCKER_FILE: "front/Dockerfile"

        DOCKER_SNAPSHOT_IMAGE: "$CI_REGISTRY_IMAGE/front:$CI_COMMIT_REF_SLUG"

      - DOCKER_ARTIFACTS_NAMESPACE: "front"

        DOCKER_FILE: "front/Dockerfile"

        DOCKER_SNAPSHOT_IMAGE: "$CI_REGISTRY_IMAGE/front/snapshot:$CI_COMMIT_REF_SLUG"

        DOCKER_RELEASE_IMAGE: "$CI_REGISTRY_IMAGE/front:$CI_COMMIT_REF_NAME"

      - DOCKER_FILE: "back/Dockerfile"

        DOCKER_SNAPSHOT_IMAGE: "$CI_REGISTRY_IMAGE/back:$CI_COMMIT_REF_SLUG"

      - DOCKER_ARTIFACTS_NAMESPACE: "back"

        DOCKER_FILE: "back/Dockerfile"

        DOCKER_SNAPSHOT_IMAGE: "$CI_REGISTRY_IMAGE/back/snapshot:$CI_COMMIT_REF_SLUG"

        DOCKER_RELEASE_IMAGE: "$CI_REGISTRY_IMAGE/back:$CI_COMMIT_REF_NAME"

```
@@ -295,6 +299,9 @@ variables: 
  DOCKER_BUILD_TOOL: "buildah"

```



> [!tip]

> Setting the `DOCKER_ARTIFACTS_NAMESPACE` variable is only necessary if you need to namespace the build job output variables.



### Secrets management



Here are some advices about your **secrets** (variables marked with a :lock:):
@@ -372,6 +379,16 @@ This job produces _output variables_ that are propagated to downstream jobs (usi 


They may be freely used in downstream jobs (for instance to deploy the upstream built Docker image, whatever the branch or tag).



> [!important]

> If [multiple images build](#multiple-images-build) is configured and `DOCKER_ARTIFACTS_NAMESPACE` is set, the output variables are namespaced with a 

> sluggified value of the `DOCKER_ARTIFACTS_NAMESPACE` variable (stripped of punctuation characters and converted to lowercase):

> 

> * `docker_<namespace_slug>_image`: snapshot image name **with tag**,

> * `docker_<namespace_slug>_image_digest`: snapshot image name **with digest** (no tag),

> * `docker_<namespace_slug>_repository`: snapshot image **bare repository** (no tag nor digest),

> * `docker_<namespace_slug>_tag`: snapshot image tag,

> * `docker_<namespace_slug>_digest`: snapshot image digest.



If you want to use GitLab CI variables or any other variable in your Dockerfile, you can add them to `DOCKER_BUILD_ARGS` like so:



```yaml
@@ -525,6 +542,16 @@ This job produces _output variables_ that are propagated to downstream jobs (usi 


They may be freely used in downstream jobs (for instance to deploy the upstream built Docker image, whatever the branch or tag).



> [!important]

> If [multiple images build](#multiple-images-build) is configured and `DOCKER_ARTIFACTS_NAMESPACE` is set, the output variables are namespaced with a 

> sluggified value of the `DOCKER_ARTIFACTS_NAMESPACE` variable (stripped of punctuation characters and converted to lowercase):

> 

> * `docker_<namespace_slug>_image`: release image name **with tag**,

> * `docker_<namespace_slug>_image_digest`: release image name **with digest** (no tag),

> * `docker_<namespace_slug>_repository`: release image **bare repository** (no tag nor digest),

> * `docker_<namespace_slug>_tag`: release image tag,

> * `docker_<namespace_slug>_digest`: release image digest.



#### Using extra tags



When publishing the _release_ image, the Docker template might publish it again with additional tags (aliases):

kicker.json

+5 −0
Original line number Diff line number Diff line
@@ -125,6 +125,11 @@ 
      "name": "DOCKER_PUSH_ARGS",

      "description": "Additional docker/buildah `push` arguments (executed right after `build`).\n\nEx: `--compression-format zstd --compression-level 20`",

      "advanced": true

    },

    {

      "name": "DOCKER_ARTIFACTS_NAMESPACE",

      "description": "Output artifacts namespace _(only required when building multiple images)_",

      "advanced": true

    }

  ],

  "features": [

templates/gitlab-ci-docker.yml

+40 −23
Original line number Diff line number Diff line
@@ -66,6 +66,9 @@ spec: 
    release-image:

      description: Docker release image

      default: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME

    artifacts-namespace:

      description: Output artifacts namespace _(only required when building multiple images)_

      default: ''

    release-extra-tags-pattern:

      description: |-

        Defines the image tag pattern that `$DOCKER_RELEASE_IMAGE` should match to push extra tags (supports capturing groups)
@@ -298,6 +301,7 @@ variables: 


  DOCKER_FILE: $[[ inputs.file ]]

  DOCKER_CONFIG_FILE: $[[ inputs.config-file ]]

  DOCKER_ARTIFACTS_NAMESPACE: $[[ inputs.artifacts-namespace ]]



  # When testing a Docker Health (test stage), how long (in seconds) wait for the HealthCheck status (https://docs.docker.com/engine/reference/builder/#healthcheck)

  DOCKER_HEALTHCHECK_TIMEOUT: $[[ inputs.healthcheck-timeout ]]
@@ -907,19 +911,21 @@ stages: 
  }



  function create_dotenv_file() {

    export docker_digest=$1

    export docker_image=${2:-$DOCKER_SNAPSHOT_IMAGE}



    export docker_repository=${docker_image%:*}

    export docker_tag=${docker_image##*:}

    docker_digest=$1

    docker_image=${2:-$DOCKER_SNAPSHOT_IMAGE}

    docker_repository=${docker_image%:*}

    docker_tag=${docker_image##*:}

    ans_slug=$(echo "$DOCKER_ARTIFACTS_NAMESPACE" | tr -d '[:punct:]' | tr '[:upper:]' '[:lower:]')

    var_prefix="${ans_slug:+${ans_slug}_}"

    dotenvfile="docker.env${ans_slug:+.${ans_slug}}"

    {

      echo "docker_image=$docker_image"

      echo "docker_image_digest=$docker_repository@$docker_digest"

      echo "docker_repository=$docker_repository"

      echo "docker_tag=$docker_tag"

      echo "docker_digest=$docker_digest"

    } > docker.env

    chmod 644 docker.env

      echo "docker_${var_prefix}image=$docker_image"

      echo "docker_${var_prefix}image_digest=$docker_repository@$docker_digest"

      echo "docker_${var_prefix}repository=$docker_repository"

      echo "docker_${var_prefix}tag=$docker_tag"

      echo "docker_${var_prefix}digest=$docker_digest"

    } > "$dotenvfile"

    chmod 644 "$dotenvfile"

  }



  function configure_cosign_private_key() {
@@ -956,8 +962,19 @@ stages: 
        ;;

    esac



    # shellcheck disable=SC1091

    source ./docker.env

    ans_slug=$(echo "$DOCKER_ARTIFACTS_NAMESPACE" | tr -d '[:punct:]' | tr '[:upper:]' '[:lower:]')

    dotenvfile="docker.env${ans_slug:+.${ans_slug}}"

    # shellcheck disable=SC1090

    source ./"$dotenvfile"

    if [[ "$ans_slug" ]]

    then

      # unbox namespaced vars

      docker_image=$(eval echo "docker_\$${ans_slug}_image")

      docker_image_digest=$(eval echo "docker_\$${ans_slug}_image_digest")

      docker_repository=$(eval echo "docker_\$${ans_slug}_repository")

      docker_tag=$(eval echo "docker_\$${ans_slug}_tag")

      docker_digest=$(eval echo "docker_\$${ans_slug}_digest")

    fi

    install_cosign

    configure_cosign_private_key
@@ -1085,14 +1102,14 @@ docker-hadolint: 
      hadolint -v

    - mkdir -p -m 777 reports

    - log_info "Scanning ${DOCKER_FILE}..."

    - dockerfile_hash=$(echo "$DOCKER_FILE" | md5sum | cut -d" " -f1)

    - basename=$(echo "$DOCKER_FILE" | tr '[:punct:]' '_')

    # Output in Code Climate format (GitLab integration)

    - hadolint --no-fail -f gitlab_codeclimate $DOCKER_HADOLINT_ARGS $hadolint_config_opts "$DOCKER_FILE" > "reports/docker-hadolint-${dockerfile_hash}.codeclimate.json"

    - hadolint --no-fail -f gitlab_codeclimate $DOCKER_HADOLINT_ARGS $hadolint_config_opts "$DOCKER_FILE" > "reports/docker-hadolint-${basename}.codeclimate.json"

    # Output in JSON format

    - |

      if [[ "$DEFECTDOJO_HADOLINT_REPORTS" ]]

      then

        hadolint --no-fail -f json $DOCKER_HADOLINT_ARGS $hadolint_config_opts "$DOCKER_FILE" > "reports/docker-hadolint-${dockerfile_hash}.native.json"

        hadolint --no-fail -f json $DOCKER_HADOLINT_ARGS $hadolint_config_opts "$DOCKER_FILE" > "reports/docker-hadolint-${basename}.native.json"

      fi

    # last run with console output (with failure)

    - hadolint $DOCKER_HADOLINT_ARGS $hadolint_config_opts "$DOCKER_FILE"
@@ -1125,7 +1142,7 @@ docker-kaniko-build: 
  artifacts:

    reports:

      dotenv:

        - docker.env

        - docker.env*

  rules:

    - if: '$DOCKER_BUILD_TOOL == "kaniko" || ($DOCKER_BUILD_TOOL == "default" && $TBC_DEFAULT_DOCKER_BUILD_TOOL == "kaniko")'
@@ -1157,7 +1174,7 @@ docker-dind-build: 
  artifacts:

    reports:

      dotenv:

        - docker.env

        - docker.env*

  rules:

    - if: '$DOCKER_BUILD_TOOL == "dind" || ($DOCKER_BUILD_TOOL == "default" && $TBC_DEFAULT_DOCKER_BUILD_TOOL == "dind")'
@@ -1204,7 +1221,7 @@ docker-buildah-build: 
  artifacts:

    reports:

      dotenv:

        - docker.env

        - docker.env*

  rules:

    - if: '$DOCKER_BUILD_TOOL == "buildah" || ($DOCKER_BUILD_TOOL == "default" && $TBC_DEFAULT_DOCKER_BUILD_TOOL == "buildah")'
@@ -1297,7 +1314,7 @@ docker-trivy: 
  - |

    export TRIVY_USERNAME=${DOCKER_REGISTRY_SNAPSHOT_USER:-${DOCKER_REGISTRY_USER:-$CI_REGISTRY_USER}}

    export TRIVY_PASSWORD=${DOCKER_REGISTRY_SNAPSHOT_PASSWORD:-${DOCKER_REGISTRY_PASSWORD:-$CI_REGISTRY_PASSWORD}}

    basename=$(echo "${DOCKER_SNAPSHOT_IMAGE}" | sed 's|[/:]|_|g')

    basename=$(echo "${DOCKER_SNAPSHOT_IMAGE}" | tr '[:punct:]' '_')

    mkdir -p ./reports

    if [[ -z "$TRIVY_SERVER" ]]; then

      log_warn "\\e[93mYou are using Trivy in standalone mode. To get faster scans, consider setting the TRIVY_SERVER variable to the address of a Trivy server. More info here: https://trivy.dev/docs/latest/references/modes/client-server/\\e[0m"
@@ -1351,7 +1368,7 @@ docker-sbom: 
      log_info "Syft version:"

      /syft version

    - mkdir -p -m 777 reports

    - basename=$(echo "${DOCKER_SNAPSHOT_IMAGE}" | sed 's|[/:]|_|g')

    - basename=$(echo "${DOCKER_SNAPSHOT_IMAGE}" | tr '[:punct:]' '_')

    - /syft scan ${TRACE+-vv} $DOCKER_SNAPSHOT_IMAGE $DOCKER_SBOM_OPTS -o cyclonedx-json=reports/docker-sbom-${basename}.cyclonedx.json -o json=reports/docker-sbom-${basename}.native.json

    - chmod a+r reports/docker-sbom-${basename}.cyclonedx.json reports/docker-sbom-${basename}.native.json

    - |
@@ -1461,7 +1478,7 @@ docker-publish: 
  artifacts:

    reports:

      dotenv:

        - docker.env

        - docker.env*

  rules:

    # on tag: if semrel info not enabled or semrel integration disabled

    - if: '$CI_COMMIT_TAG && ($SEMREL_INFO_ON == null || $SEMREL_INFO_ON == "" || $DOCKER_SEMREL_RELEASE_DISABLED == "true")'