doc(archi): new page with architecture principles

---

author: Pierre Smeyers

---

# Architectural Principles

This page presents fundamental _to be continuous_ architectural principles and their justification.

## Common features

Here is the list of common _to be continuous_ features that must be implemented by every template:

* [Proxy Configuration support](../usage.md#proxy-configuration)

* [Custom Certificate Authority support](../usage.md#certificate-authority-configuration)

* [Scoped Variables support](../usage.md#scoped-variables)

* [`$TRACE` variable enables debug logging](../usage.md#debugging-to-be-continuous-jobs)

* [Merge Request Workflow by default, globally overridable](../usage.md#merge-request-workflow)

* [Adaptive Pipeline strategy by default, globally overridable](../usage.md#test--analysis-jobs-rules)

<!-- * release management & package publishing, -->

## Sensible defaults

Examples:

- When introducing a new toggleable feature (e.g. a new SAST job), decide wisely whether it should be enabled by default (opt-out) or disabled by default (opt-in).

- When defining default tool CLI options, select the most appropriate options. If unsure: prefer the option that raises the bar (of code quality/best practices) rather than the conservative one

## The right template perimeter

_to be continuous_ provides templates organized around languages and technologies.

But sometimes it's not very easy to determine what a template should do and where a new template 

should be created for additional features.

_to be continuous_ templates are organized around the input file(s) and/or a CLI tool that the job apply to.

Examples:

- The Maven template relies on the `mvn` tool and `pom.xml` in the repo, similarly the Gradle template relies on the `gradle`tool and the `build.gradle` file.

- The Python template relies on the presence of Python source code in the repository, but still supports various dependency management and build tools.

- Kubernetes relies on `kubectl`, but also embeds additional tools related to Kubernetes manifests.

- The Docker template provides a coherent set of tools to build & test container images from a `Dockerfile`. 

For instance it might (and it will soon) support various tools to bake the container image from the `Dockerfile` (e.g. Docker-in-Docker, `kaniko`, `podman` or `buildah`).

## No nested includes

[Nested includes](https://docs.gitlab.com/ee/ci/yaml/includes.html#use-nested-includes) shall not be used in _to be continuous_ templates for the following reasons:

- it would prevent _to be continuous_ templates from being [remote included](https://docs.gitlab.com/ee/ci/yaml/#includeremote) (b.t.w. this is the technique used by [R2Devops](https://r2devops.io/), also featuring _to be continuous_ templates)

- it would prevent _to be continuous_ templates from being included from another root path than `/to-be-continuous` (it may happen in some organizations when the groups hierarchy is locked by governance rules)

- should the nested include be versionned or not? both options have serious drawback (dependency hell vs. less control on impact management)

## Standard stages

Every template shall reuse defined [generic pipeline stages](../understand.md#generic-pipeline-stages).

If an additional stage seems to be required, that must be discussed with the core team first.

## Standard environments

Every infrastructure(-as-code) and deployment template shall support 4 kinds of environments (each being optional):

| Environment Type | Description                                       | Associated branch(es)                  |

| ---------------- | ------------------------------------------------- | -------------------------------------- |

| **Review**       | Those are dynamic and ephemeral environments to deploy your ongoing developments.<br/> It is a strict equivalent of GitLab's [Review Apps](https://docs.gitlab.com/ee/ci/review_apps/) feature. | All **development branches** (non-integration, non-production) |

| **Integration**  | A single environment to continuously deploy your integration branch. | The **integration branch** (`develop` by default) |

| **Staging**      | A single environment to continuously deploy your production branch.<br/> It is an iso-prod environment, meant for running the automated acceptance tests prior to deploying to the production env. | The **production branch** (`main` or `master`  by default) |

| **Production**   | _Well.. the prod!_ | The **production branch** (`main` or `master`  by default) |

## Latest container images version by default

As stated in [our documentation](../usage.md#docker-images-versions), _to be continuous_ templates use required tools as container images.

And when available, the _latest_ version is used.

Why latest?

- there is no _good default for everone_: every project should select a specific version that fits their needs

- the latest version has more chances to be up-to-date with security patches (at least for an official and maintained project)

## Test alongside build

For built languages, we've decided that build and test should stand in the same GitLab job (at least by default).

This choice is for performance reasons, as testing often requires to build first, and doing the 2 steps in separate jobs might involve redoing part of the build 2 times, even when cache is properly implemented.

## Tools reports

_to be continuous_ templates features many tools that produce reports (testing tools, SAST, DAST, linters, vulnerability scanners...).

We decided that those tools:

- must **always** produce the textual (human readable) report in the console,

- when available, they must produce the [GitLab-supported format report](https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html)

- when SonarQube template is detected (by the presence of `$SONAR_HOST_URL` variable) and when available, they shall produce the SonarQube-supported format report ([test execution report](https://docs.sonarqube.org/latest/analyzing-source-code/test-coverage/overview/#test-execution-reports), [coverage report](https://docs.sonarqube.org/latest/analyzing-source-code/test-coverage/overview/#coverage-support) and [third-party issues](https://docs.sonarqube.org/latest/analyzing-source-code/importing-external-issues/importing-third-party-issues/))

- when DefectDojo template is detected (by the presence of the corresponding `$DEFECTDOJO_XXX_REPORTS` variable) and when available, they shall produce the [DefectDojo-supported format report](https://defectdojo.github.io/django-DefectDojo/integrations/parsers/file/)

The output report file path must also comply to the following **convention**:

```

<project_root_dir>/reports/<job_name>.<format_name>.<extension>`

```

Examples:

- `$NG_WORKSPACE_DIR/reports/ng-e2e.xunit.xml` is the end-to-end test execution report produced by the Angular template (`ng`), using the xUnit/JUnit format (XML)

- `$CYPRESS_PROJECT_DIR/reports/cypress-*.xunit.xml` is the test execution report produced by the Cypress template, using the xUnit/JUnit format (XML)

- `$PYTHON_PROJECT_DIR/reports/py-coverage.cobertura.xml` is the coverage report produced by the Python template (`py`), using the Cobertura format (XML)

- `reports/docker-sbom-*.cyclonedx.json` is the SBOM report produced by the Docker template, using the CycloneDX format (JSON)

- `reports/docker-hadolint-*.codeclimate.json` is the Hadoling report produced by the Docker template, using the CodeClimate format (JSON)

## Dotenv artifacts

Propagate dynamic output variables to downstream jobs...

## Guiding principle and rules

As a template developer, you should make everything possible to honor [the guiding principles](../understand.md#the-guiding-principles).

As a template developer, you should make everything possible to honor the _guiding principles_:

!!! Success "The guiding principles"

    * continuous integration (CI) has to be **fast** (and to some extend _energy efficient_)

## Docker Images Versions

_to be continuous_ templates use - whenever possible - required tools as Docker images.

_to be continuous_ templates use - whenever possible - required tools as container images.

And when available, the _latest_ image version is used.

In some cases, using the latest version is a good thing, and in some other cases, the latest version is bad.

* _latest_ is **good** for:

    * DevSecOps tools (Code Quality, Security Analysis, Dependency Check, Linters ...) as using the latest version of the tool is the best way to ensure

      you're likely to detect vulnerabilities as soon as possible (well, as soon as new vulnerabilities are known and covered by DevSecOps tools).

    * Public cloud CLI clients as there is only one version of the public cloud, and the official Docker image is likely to evolve at the same time as the APIs.

    * Public cloud CLI clients as there is only one version of the public cloud, and the official container image is likely to evolve at the same time as the APIs.

* _latest_ is **not good** for:

    * Build tools as your project is developped using one specific version of the language / the build tool, and you would like to control when you change version.

    * Infrastructure-as-Code tools for the same reason as above.

!!! Success "To summarize"

    1. Make sure you explicitely override the Docker image versions of your build, Infrastructure-as-Code, private cloud CLI clients and acceptance tests tools matching your project requirements.

    1. Make sure you explicitely override the container image versions of your build, Infrastructure-as-Code, private cloud CLI clients and acceptance tests tools matching your project requirements.

    2. Be aware that sometimes your pipeline may fail (without any change from you) due to a new version of DevSecOps tool that either highlights a new vulnerability (:tada:), or due to a bug or breaking change in the tool (:poop: happens).

## Secrets managements

    :red_circle: They **don't support _scoped variables_**:

    * variables used to parameterize the jobs Docker image(s) (ex: `MAVEN_IMAGE` or `K8S_KUBECTL_IMAGE`),

    * variables used to parameterize the jobs container image(s) (ex: `MAVEN_IMAGE` or `K8S_KUBECTL_IMAGE`),

    * variables that enable/disable some jobs behavior (ex: `MAVEN_DEPLOY_ENABLED`, `NODE_AUDIT_DISABLED` or `AUTODEPLOY_TO_PROD`),

    * variables used in [artifacts](https://docs.gitlab.com/ee/ci/yaml/#artifacts), [cache](https://docs.gitlab.com/ee/ci/yaml/#cache) or [rules](https://docs.gitlab.com/ee/ci/yaml/#rules) sections (ex: `PYTHON_PROJECT_DIR`, `NG_WORKSPACE_DIR` or `TF_PROJECT_DIR`).

Thus, if you wish to override something **for all the jobs from a specific template**, this is the right place to do

the magic.

### Example 1: add Docker services

### Example 1: add service containers

In this example, let's consider my Java project needs a MySQL database to run its unit tests.

Those changes will gracefully be merged with the `mvn-build` job, the rest of it (defined by the Maven template) will

remain unchanged.

!!! TIP 

    [More info about Service Containers](https://docs.gitlab.com/ee/ci/services/)

### Example 2: run on private runners with proxy

In this example, let's consider my project needs to deploy on a Kubernetes cluster that is only

  - Contribute:

    - Workflow: dev/workflow.md

    - Guidelines: dev/guidelines.md

    - Architecture: dev/architecture.md

  - Security: 

    - Overview: secu/index.md

    # INSERT_TRIVY_HERE