Merge branch 'feat/archi-principles' into 'master'

}

.md-typeset .grid.cards>ol>li>:first-child, .md-typeset .grid.cards>ul>li>:first-child, .md-typeset .grid>.card>:first-child {

    margin-top: 0;

    font-size: .8rem;

}

.md-typeset ol li blockquote, .md-typeset ol li p, .md-typeset ul li blockquote, .md-typeset ul li p {

    margin: 0.5em 0;

  margin: 0;

}

/* .admonition > .highlight {

  margin: 0 -0.6rem !important;

  border: 0;

} */

/* ================== */

/* top navigation bar */

/* ================== */

img.tbc-tmpl-icon {

  max-width: 2rem;

  vertical-align: middle;

  margin-right: .25rem;

  margin-right: .2rem;

}

/* ============= */

---

author: Pierre Smeyers

---

# Architectural Principles

This page presents fundamental _to be continuous_ architectural principles and their justification.

## Common features

Here is the list of common _to be continuous_ features that must be implemented by every template:

* [Proxy Configuration support](../usage.md#proxy-configuration)

* [Custom Certificate Authority support](../usage.md#certificate-authority-configuration)

* [Scoped Variables support](../usage.md#scoped-variables)

* [`$TRACE` variable enables debug logging](../usage.md#debugging-to-be-continuous-jobs)

* [configurable Git references](../usage.md#configurable-git-references)

* [Merge Request Workflow](../usage.md#merge-request-workflow) by default, globally overridable

* [Adaptive Pipeline strategy](../usage.md#test-analysis-jobs-rules) by default, globally overridable

<!-- * release management & package publishing, -->

## Standard stages

Every template shall reuse defined [generic pipeline stages](../understand.md#generic-pipeline-stages).

If an additional stage seems to be required, that must be discussed with the core team first.

## Sensible defaults

Examples:

- When introducing a new toggleable feature (e.g. a new SAST job), decide wisely whether it should be enabled by default (opt-out) or disabled by default (opt-in).

- When defining default tool CLI options, select the most appropriate options. If unsure: prefer the option that raises the bar (of code quality/best practices) rather than the conservative one

## GitLab by default

When implementing a feature requiring an external service (ex: a packages registry)

and if GitLab provides such a service, then:

- Everything must be done to configure the feature to use GitLab's service by default (zero config).

- If not possible, at least all the required documentation shall be added to the `README.md` to explain what has to be done to configure the GitLab service properly.

Examples:

- The Docker template uses [GitLab's container registry](https://docs.gitlab.com/ee/user/packages/container_registry/) by default, although any other container registry can be used by configuration.

- The Python template uses [GitLab's PyPI registry](https://docs.gitlab.com/ee/user/packages/pypi_repository/) by default, although any other PyPI registry can be used by configuration.

- The Maven template can't use [GitLab's Maven registry](https://docs.gitlab.com/ee/user/packages/maven_repository/) by default, but documents what has to be done in its `README.md`.

## The right template perimeter

_to be continuous_ provides templates organized around languages and technologies.

But sometimes it's not very easy to determine what a template should do and where a new template 

should be created for additional features.

_to be continuous_ templates are organized around the input file(s) and/or a CLI tool that the job apply to.

Examples:

- The Maven template relies on the `mvn` tool and `pom.xml` in the repo, similarly the Gradle template relies on the `gradle`tool and the `build.gradle` file.

- The Python template relies on the presence of Python source code in the repository, but still supports various dependency management and build tools.

- Kubernetes relies on `kubectl`, but also embeds additional tools related to Kubernetes manifests.

- The Docker template provides a coherent set of tools to build & test container images from a `Dockerfile`. 

For instance it might (and it will soon) support various tools to bake the container image from the `Dockerfile` (e.g. Docker-in-Docker, `kaniko`, `podman` or `buildah`).

## No nested includes

[Nested includes](https://docs.gitlab.com/ee/ci/yaml/includes.html#use-nested-includes) shall not be used in _to be continuous_ templates for the following reasons:

- it would prevent _to be continuous_ templates from being [remote included](https://docs.gitlab.com/ee/ci/yaml/#includeremote) (b.t.w. this is the technique used by [R2Devops](https://r2devops.io/), also featuring _to be continuous_ templates)

- it would prevent _to be continuous_ templates from being included from another root path than `/to-be-continuous` (it may happen in some organizations when the groups hierarchy is locked by governance rules)

- should the nested include be versionned or not? both options have serious drawback (dependency hell vs. less control on impact management)

## Standard environments

Every infrastructure(-as-code) and deployment template shall support 4 kinds of environments (each being optional):

| Environment Type | Description                                       | Associated branch(es)                  |

| ---------------- | ------------------------------------------------- | -------------------------------------- |

| **Review**       | Those are dynamic and ephemeral environments to deploy your ongoing developments.<br/> It is a strict equivalent of GitLab's [Review Apps](https://docs.gitlab.com/ee/ci/review_apps/) feature. | All **development branches** (non-integration, non-production) |

| **Integration**  | A single environment to continuously deploy your integration branch. | The **integration branch** (`develop` by default) |

| **Staging**      | A single environment to continuously deploy your production branch.<br/> It is an iso-prod environment, meant for running the automated acceptance tests prior to deploying to the production env. | The **production branch** (`main` or `master`  by default) |

| **Production**   | _Well.. the prod!_ | The **production branch** (`main` or `master`  by default) |

## Container images

### Explicit official image registry

Not specifying explicitly the official image registry in the image name enables a supply 

chain attack by pushing a malicious image with the same name in alternate registries

and expect a mirror will pick one of the alternate registries before the official one.

In order to protect against this kind of attacks, _to be continuous_ always use fully qualified image names (i.e. including the registry).

### Latest version by default

As stated in [our documentation](../usage.md#docker-images-versions), _to be continuous_ templates use required tools as container images.

And when available, the _latest_ version is used.

Why latest?

- there is no _good default for everone_: every project should select a specific version that fits their needs

- the latest version has more chances to be up-to-date with security patches (at least for an official and maintained project)

## Test alongside build

For built languages, we've decided that build and test should stand in the same GitLab job (at least by default).

This choice is for performance reasons, as testing often requires to build first, and doing the 2 steps in separate jobs might involve redoing part of the build 2 times, even when cache is properly implemented.

## Tools reports

_to be continuous_ templates features many tools that produce reports (testing tools, SAST, DAST, linters, vulnerability scanners...).

### Report formats

We decided that those tools:

- must **always** produce the textual (human readable) report in the console,

- when available, they must produce the [GitLab-supported format report](https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html)

- when SonarQube template is detected (by the presence of `$SONAR_HOST_URL` variable) and when available, they shall produce the SonarQube-supported format report ([test execution report](https://docs.sonarqube.org/latest/analyzing-source-code/test-coverage/overview/#test-execution-reports), [coverage report](https://docs.sonarqube.org/latest/analyzing-source-code/test-coverage/overview/#coverage-support) and [third-party issues](https://docs.sonarqube.org/latest/analyzing-source-code/importing-external-issues/importing-third-party-issues/))

- when DefectDojo template is detected (by the presence of the corresponding `$DEFECTDOJO_XXX_REPORTS` variable) and when available, they shall produce the [DefectDojo-supported format report](https://defectdojo.github.io/django-DefectDojo/integrations/parsers/file/)

### Report files naming convention

The output report file path must comply to the following **convention**:

```

<project_root_dir>/reports/<job_name>.<format_name>.<extension>`

```

Examples:

- `$NG_WORKSPACE_DIR/reports/ng-e2e.xunit.xml` is the end-to-end test execution report produced by the Angular template (`ng`), using the xUnit/JUnit format (XML)

- `$CYPRESS_PROJECT_DIR/reports/cypress-*.xunit.xml` is the test execution report produced by the Cypress template, using the xUnit/JUnit format (XML)

- `$PYTHON_PROJECT_DIR/reports/py-coverage.cobertura.xml` is the coverage report produced by the Python template (`py`), using the Cobertura format (XML)

- `reports/docker-sbom-*.cyclonedx.json` is the SBOM report produced by the Docker template, using the CycloneDX format (JSON)

- `reports/docker-hadolint-*.codeclimate.json` is the Hadoling report produced by the Docker template, using the CodeClimate format (JSON)

## Dotenv artifacts

_to be continuous_ templates are composable. In other words they cooperate gracefully with others to minimize the amount of integration work.

It is common that a job _produces_ output that could be used by downstream jobs, possibly from other templates.

Those output can be artifacts (ex: result of a build), reports (ex: SAST tools reports consumed by the SonarQube template or DefectDojo), and sometimes

it could be a dynamic variable.

For this last need, _to be continuous_ uses the [dotenv report format](https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportsdotenv) to propagate output environment variables downstream.

Examples:

- every deployment template propagates the deployed environment as a `$environment_url`, that can be used by

any acceptance testing template, whichever the deployment technology or the testing tools

- the Docker and Cloud Native Buildpack templates both propagates several variables of the built container image (container registry, full url with tag, full url with digest, tag only...) that can be used in downstream templates to use pull this image

## Non-backward compatible GitLab features

Some GitLab features are not backward compatible. 

For instance features that introduce a new keyword that would break the `.gitlab-ci.yml` validation on 

previous versions of GitLab.

In that case, _to be continuous_ templates shall observe a _reasonable_ rollout time before implementing

them, not to break projects using them on a self-managed server.

For security patches, GitLab [maintains the two previous `major.minor` versions](https://docs.gitlab.com/ee/policy/maintenance.html).

We decided to observe the same delay before implementing a non-backward compatible feature.

I.e. wait that the GitLab version that introduced the feature is the oldest version maintained by GitLab.

In practice, this is about 3 months.

This page references GitLab CI development guidelines for people that wish to contribute with templates of their own.

## Guiding principle and rules

# _to be continuous_ architectural principles

As a template developer, you should make everything possible to honor [the guiding principles](../understand.md#the-guiding-principles).

!!! Success "The guiding principles"

    * continuous integration (CI) has to be **fast** (and to some extend _energy efficient_)

    * continuous deployment/delivery (CD) has to **secure** the deployment/delivery to production

!!! example "Corollary rules"

    1. the least stages, the best

    1. parallelize all jobs that can be

    1. long jobs (5 min or more) should not be auto-run by default, but rather [manually](https://docs.gitlab.com/ee/ci/yaml/#whenmanual)

       or [scheduled](https://docs.gitlab.com/ee/ci/pipelines/schedules.html). <br/>

       Projects willing to auto-run those jobs should be able to override the default.

## Stages

Every template shall reuse defined [generic pipeline stages](../understand.md#generic-pipeline-stages).

If an additional stage seems to be required, that must be discussed with the core team.

As a template developer, you should be aware of _to be continuous_ [architectural principles](./architecture.md)

as they generally apply to all templates.

## Naming conventions

_to be continuous_ templates are built to be:

* **Modular**: each template complies to the [Single-responsibility principle](https://en.wikipedia.org/wiki/Single-responsibility_principle) while observing common architectural principles (standard behaviour per template type, generic pipeline stages, common Git workflow principles, ...)

* **Modular**: each template complies to the [Single-responsibility principle](https://en.wikipedia.org/wiki/Single-responsibility_principle) while observing [common architectural principles](./dev/architecture.md) (standard behaviour per template type, generic pipeline stages, common Git workflow principles, ...)

* **Composable**: each template cooperates gracefully with others to minimize the amount of integration work. 

Let's illustrate this with an example.