Merge branch 'feat/adaptive-pipeline' into 'master'

## Release process

The release process is based on [semantic-release](https://semantic-release.gitbook.io/).

As a result, releases are performed automatically when commits on the `master` branch indicate a new release is required (see below).

As a result, releases are performed automatically when commits on the `main` (or `master`) branch indicate a new release is required (see below).

## Git Commit Guidelines

      HTTPS_PROXY: "${https_proxy}"

      NO_PROXY: "${no_proxy}"

  ```

  _(developing this requires [advanced to be continuous knowledge](../../usage/#advanced-usage-override-yaml))_

  _(developing this requires [advanced to be continuous knowledge](../usage.md#advanced-usage-override-yaml))_

2. declare it in a `kicker-extras.json` file:

  ```json

  {

In that case, you just have to:

1. Create a GitLab project (if possible with `public` or at least `internal` visibility),

2. Develop your template following the [guidelines](../../dev/guidelines),

3. Declare the template with a [Kicker descriptor](../../dev/guidelines#kicker-descriptor),

2. Develop your template following the [guidelines](../dev/guidelines.md),

3. Declare the template with a [Kicker descriptor](../dev/guidelines.md#kicker-descriptor),

4. Make a first release by creating a Git tag.

Your template can then be used like any other _to be continuous_ one.

      ]

    }

    ```

4. Manually start a pipeline on the `master` branch: this will (re)generate a new Docker image with your

4. Manually start a pipeline on the `main` (or `master`) branch: this will (re)generate a new Docker image with your

   configuration that will now be used by every template job.

    If you need to modify template code, you'll have 2 options depending on your case:

    * if it's a general enhancement/fix: make a contribution to the Open Source [to be continuous project](https://gitlab.com/to-be-continuous/) and get the change through the synchronization task,

    * if it's a change specific to your company: see [advanced usage](../advanced/)

    * if it's a change specific to your company: see [advanced usage](../advanced.md)

By the way, you can manually trigger a pipeline in your [tools/gitlab-sync](https://gitlab.com/to-be-continuous/tools/gitlab-sync) project

anytime to synchronize your _to be continuous_ copy.

If your company is using non-Default Trusted Certificate Authorities to generate server certificates, it is highly probable that some _to be continuous_ template jobs will fail because such or such tool failed validating SSL certificates.

As explained in the [usage guide](../../usage/#certificate-authority-configuration), each project might

As explained in the [usage guide](../usage.md#certificate-authority-configuration), each project might

define a `CUSTOM_CA_CERTS` variable either as a group/project variable, or in its `.gitlab-ci.yml` file to declare the custom company Certificate Authorities.

But there is also a global way to fix this. Ask your GitLab administrator to declare `DEFAULT_CA_CERTS`

| ---------------- | ------------------------------------------------- | -------------------------------------- |

| **Review**       | Those are dynamic and ephemeral environments to deploy your ongoing developments.<br/> It is a strict equivalent of GitLab's [Review Apps](https://docs.gitlab.com/ee/ci/review_apps/) feature. | All **development branches** (non-integration, non-production) |

| **Integration**  | A single environment to continuously deploy your integration branch. | The **integration branch** (`develop` by default) |

| **Staging**      | A single environment to continuously deploy your production branch.<br/> It is an iso-prod environment, meant for running the automated acceptance tests prior to deploying to the production env. | The **production branch** (`master` or `main`  by default) |

| **Staging**      | A single environment to continuously deploy your production branch.<br/> It is an iso-prod environment, meant for running the automated acceptance tests prior to deploying to the production env. | The **production branch** (`main` or `master`  by default) |

| **Production**   | _Well.. the prod!_ | The **production branch** (`main` or `master`  by default) |

A few remarks:

    least one active job is mapped to it. Therefore - for e.g. - if you're not using any packaging template, the 

    `package-xxx` stages will never show up in your pipelines.

## Git workflows

## Development workflow

So far, we've presented a quite _static_ vision of what a CI/CD pipeline should be, but the reality is somewhat different

depending whether it's triggered on a **development branch** (that's CI) or on the **production branch** (that's CD).

So far, we've presented a quite _static_ vision of what a CI/CD pipeline should be, but there will be differences

depending on where you are in the **development workflow**:

### The guiding principles

| Stage               | Description                            |

| ------------------- | -------------------------------------- |

| dev/early           | Working in a **feature branch**, associated to **no Merge Request** |

| dev/work-in-progress| Working in a **feature branch**, associated to a **Draft Merge Request** |

| dev/review          | Working in a **feature branch**, associated to a **Ready Merge Request** |

| integration         | A change is being pushed to the **integration branch** (if any) |

| deployment          | A change is being pushed pushed to the **production branch** |

!!! Success "The guiding principles"

### Adaptive pipeline

    * continuous integration (CI) has to be **fast** (and to some extend _energy efficient_)

    * continuous deployment/delivery (CD) has to **secure** the deployment/delivery to production

Ask a developer and a security expert what is the top requirement when building a CI/CD pipeline?

**Continuous integration** is what a developer keeps doing 90% of the day: change a bit of code, commit, push and wait

for the code to be up and running somewhere.

Such a cycle may occur up to 30 times per day, sometimes more.

That's why it has to be **as fast as possible**, as every saved minute is a huge gain for the developer's productivity at

the end of the day.

That's also why it may not be wise to start a complete non-regression automated test campaign every time the developer 

changes a comma in his code.

The developer would probably answer: 

On the contrary, **continuous deployment/delivery** occurs much less often but denotes the will of rolling out a new version

of your code.

That's why it has to make everything possible to **maximize the confidence** in what you are doing.

> **Speed**<br/>Divide the pipeline execution time by 2, you’ll double my productivity.

### How is it mapped on Git workflows?

While the security expert would argue: 

> **Tests & Security checks**<br/>Software must be thoroughly tested and analyzed at every stage to avoid bugs, poor quality or even worse: security vulnerabilities

Those 2 goals look antinomic as adding more tests and analysis tools in the pipeline will obviously increase the overall execution time.

Still, you can conciliate both developer's productivity and software quality and security if you implement an adaptive pipeline depending on the stage in the development workflow: prioritize speed in the early development stages, and gradually introduce DevSecOps checks as you get closer to production.

_to be continuous_ implements the above idea through the **adaptive pipeline** strategy:

![Adaptive Pipeline](img/Adaptive-Pipeline.drawio.svg)

### Supported Git workflows

Using Git, there are [many possible workflows](https://www.atlassian.com/git/tutorials/comparing-workflows).

You are free to use whichever you want, but our templates make strong hypothesis you should be aware of:

* the `master` (or `main`) branch is **production** (triggers the CD pipeline),

* the `main` (or `master`) branch is **production** (triggers the CD pipeline),

* the `develop` branch is **integration** (triggers an hybrid pipeline),<br/>

  :warning: _the use of an integration branch is **optional**, and even 

  [discouraged in the general case](#when-to-use-an-integration-branch)_

    > :bulb: If you're using a complete [Gitflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow) workflow,

    > _release_ branches are managed the same way as regular _development_ branches, possibly deploying to dedicated review environments.

Now, here is a table that summarizes the generic templates behavior for each branch:

<table border="1">

<thead>

<tr>

<th>Stage/<br>Branch</th>

<th style="vertical-align: middle">build</th>

<th style="vertical-align: middle">test</th>

<th style="vertical-align: middle">infra/deploy</th>

<th style="vertical-align: middle">acceptance</th>

<th style="vertical-align: middle">infra-prod/production</th>

</tr>

</thead>

<tbody>

<tr style="background-color: #0074D940;">

<td><strong>development</strong> (any branch other than <code>develop</code>, <code>main</code> or <code>master</code>)</td>

<td rowspan=3 style="vertical-align: middle; background-color: #ddd;">compile &amp; unit test</td>

<td><strong>quick analysis</strong> <i>(when possible)</i>, or <strong>manual</strong> trigger</td>

<td>create/deploy a <code>review</code> environment</td>

<td><strong>manual</strong> trigger</td>

<td>N/A</td>

</tr>

<tr style="background-color: #FF851B40;">

<td><strong>integration</strong> (<code>develop</code>&nbsp;branch)</td>

<td rowspan=2 style="vertical-align: middle; background-color: #eee;"><strong>deep analysis</strong> / <strong>automatic</strong> trigger</td>

<td>create/deploy the <code>integration</code> environment</td>

<td rowspan=2 style="vertical-align: middle; background-color: #eee;"><strong>automatic</strong> trigger</td>

<td>N/A</td>

</tr>

<tr style="background-color: #FF413640;">

<td><strong>production</strong> (<code>main</code>&nbsp;or&nbsp;<code>master</code>&nbsp;branch)</td>

<td>create/deploy the <code>staging</code> environment</td>

<td>create/deploy the <code>production</code> environment</td>

</tr>

</tbody>

</table>

### When to use an integration branch?

#### When to use an integration branch?

Let's state it clearly: **the most efficient Git workflow is the simplest one** that fits your needs.

Consequently:

Using an integration branch is discouraged in the general case, but there are some acceptable reasons to adopt one. 

The following chapters present the 3 main ones.

#### Can't afford Review environments

##### Can't afford Review environments

Instantiating a dedicated hosting environment for each development branch in progress might be indeed a cost difficult to afford.

In that case your developers will need a single, shared environment to integrate all their work.

That's exactly the purpose of an integration branch.

#### Not mature enough for continuous deployment

##### Not mature enough for continuous deployment

Lack of automated testing, need of manual acceptance tests campaign on a dedicated environment, poor software quality… 

There are many reasons why you may not feel ready for continuous deployment.

In that case, an integration branch with its associated integration environment might be the adapted solution.

#### Release-oriented deployment

##### Release-oriented deployment

In release-oriented projects, several features get bundled into a release and then deployed all at once.

With the release often comes a versioning strategy, release notes, roadmap, something very common in software edition business.