Commit 681d465e authored by Pierre Smeyers's avatar Pierre Smeyers
Browse files

Merge branch 'doc-focus-on-variables' into 'master' 

reword to focus on variables used to configure the sync job

See merge request to-be-continuous/tools/gitlab-sync!38
parents 9eb26ef8 386714bd

.gitlab-ci.yml

+0 −96
Original line number Diff line number Diff line 
include:

  # $TBC_NAMESPACE is a group variable; can be globally overridden

  - project: '$TBC_NAMESPACE/bash'

    ref: '3.3.0'

    file: '/templates/gitlab-ci-bash.yml'



variables:

  BASH_SHELLCHECK_FILES: "*.sh"

  GITLAB_SYNC_IMAGE: "registry.hub.docker.com/library/alpine:latest"

  # variabilized gitlab-cp image

  GITLABCP_IMAGE: registry.gitlab.com/to-be-continuous/tools/gitlab-cp:latest

  # variabilized sync tool to use (either 'legacy' or 'gitlab-cp')

  SYNC_TOOL: gitlab-cp



stages:

  - build

  - test



sync-tbc-legacy:

  image:

    name: "${GITLAB_SYNC_IMAGE}"

    entrypoint: [""]

  stage: build

  variables:

    # exclude samples group from sync

    EXCLUDE: samples,custom

  before_script:

    - |

      function install_ca_certs() {

        certs=$1

        if [[ -z "$certs" ]]

        then

          return

        fi

        if echo "$certs" >> /etc/ssl/certs/ca-certificates.crt

        then

          echo -e "CA certificates imported in \\e[33;1m/etc/ssl/certs/ca-certificates.crt\\e[0m"

        fi

        if echo "$certs" >> /etc/ssl/cert.pem

        then

          echo -e "CA certificates imported in \\e[33;1m/etc/ssl/cert.pem\\e[0m"

        fi

      }

    - install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}"

    - |

      function maybe_install_pkg() {

        pkg_name="$1"

        cmd_name="${2:-$pkg_name}"

        if ! command -v "$cmd_name" > /dev/null

        then

          echo "[INFO] $pkg_name not found: install..."

          if command -v apt-get > /dev/null

          then

            apt-get install -yq "$pkg_name"

          elif command -v apk > /dev/null

          then

            apk add --no-cache "$pkg_name"

          elif command -v yum > /dev/null

          then

            yum install -yq "$pkg_name"

          elif command -v dnf > /dev/null

          then

            dnf install -yq "$pkg_name"

          elif command -v zypper > /dev/null

          then

            zypper -nq install "$pkg_name"

          else

            echo "[ERROR] ... didn't find any known package manager to install $pkg_name!"

            exit 1

          fi

        fi

      }

    - maybe_install_pkg git

    - maybe_install_pkg curl

    - maybe_install_pkg jq

    - maybe_install_pkg bash

    # done twice because /etc/ssl/certs/ca-certificates.crt is altered during apk add

    # we can't use update-ca-certificates --fresh because $CUSTOM_CA_CERTS can contains several certificates

    - install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}"

  script:

    - bash ${CI_DEBUG_TRACE:+-x} ./gitlab-sync.sh

  rules:

    # disable on gitlab.com

    - if: '$CI_SERVER_HOST == "gitlab.com"'

      when: never

    # disable if $SYNC_TOOL != 'legacy'

    - if: '$SYNC_TOOL != "legacy"'

      when: never

    # allow manual source

    - if: '$CI_PIPELINE_SOURCE == "web"'

    # allow scheduled source

    - if: '$CI_PIPELINE_SOURCE == "schedule"'



sync-tbc-legacy-dryrun:

  extends: sync-tbc-legacy

  script:

    # unset dest GitLab (dryrun)

    - bash ${CI_DEBUG_TRACE:+-x} ./gitlab-sync.sh --dest-api ""

  rules:

    # run only on gitlab.com (to test at least the script correctly crawls tbc projects)

    - if: '$CI_SERVER_HOST == "gitlab.com"'



sync-tbc:

  image:

    name: "$GITLABCP_IMAGE"
@@ -147,9 +54,6 @@ sync-tbc: 
    # disable on gitlab.com

    - if: '$CI_SERVER_HOST == "gitlab.com"'

      when: never

    # disable if $SYNC_TOOL != 'gitlab-cp'

    - if: '$SYNC_TOOL != "gitlab-cp"'

      when: never

    # allow manual source

    - if: '$CI_PIPELINE_SOURCE == "web"'

    # allow scheduled source

README.md

+41 −24
Original line number Diff line number Diff line
@@ -19,23 +19,47 @@ Once copied _to be continuous_ to your GitLab server, you shall then schedule a 
The job only requires a GitLab token, that shall be configured declaring a `$GITLAB_TOKEN` CI/CD project variable.

All the other parameters/variables are automatically determined by the job but might be overwritten with the appropriate project variables if need be:



| CLI option                 | Env. Variable                    | Description                                                                                            | Default Value                           |

| -------------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------- |

| `--src-api`                | `$SRC_GITLAB_API`                | GitLab source API url                                                                                  | `https://gitlab.com/api/v4`             |

| `--src-sync-path`          | `$SRC_SYNC_PATH`                 | GitLab source root group path to synchronize                                                           | `to-be-continuous`                      |

| `--dest-api`               | `$DEST_GITLAB_API`               | GitLab destination API url                                                                             | `$CI_API_V4_URL`                        |

| `--dest-token`             | `$DEST_TOKEN` or `$GITLAB_TOKEN` | GitLab destination token with at least scopes `api,read_repository,write_repository` and `Owner` role  | _none_ (**mandatory**)                  |

| `--dest-sync-path`         | `$DEST_SYNC_PATH`                | GitLab destination root group path to synchronize (defaults to `--src-sync-path`)                      | determined from `$CI_PROJECT_NAMESPACE` |

| `--max-visibility`         | `$MAX_VISIBILITY`                | maximum visibility of projects in destination group (defaults to `public`)                             | `$CI_PROJECT_VISIBILITY`                |

| `--exclude`                | `$EXCLUDE`                       | project/group path(s) to exclude (multiple CLI option; env. variable is a coma separated list)         | `samples,custom`                        |

| `--insecure`               | `$INSECURE`                      | skip SSL verification                                                                                  | `false`                                 |

| `--update-release`         | `$UPDATE_RELEASE`                | set to force the update of the latest release (in order to trigger GitLab CI/CD catalog publication)   | `false`                                 |

| `--update-avatar`          | `$UPDATE_AVATAR`                 | force update the avatar images even when they exist and look the same                                  | `false`                                 |

| `--no-group-description`   | `$GROUP_DESCRIPTION_DISABLED`    | don't synchronize group description                                                                    | `false`                                 |

| `--no-project-description` | `$PROJECT_DESCRIPTION_DISABLED`  | don't synchronize project description                                                                  | `false`                                 |

| `--dry-run`                | _none_                           | dry run (don't execute any write action)                                                               | `false`                                 |

| `--halt-on-error`          | _none_                           | halt synchronizing when an error occurs                                                                | `false`                                 |

| `--cache-dir`              | `$CACHE_DIR`                     | cache directory (used to download resources such as images and Git repositories) (defaults to `.work`) | `.work`                                 |

| Env. Variable                    | Description                                                                                                      | Default Value                           |

| -------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------------------------- |

| `$SRC_GITLAB_API`                | GitLab source API url                                                                                            | `https://gitlab.com/api/v4`             |

| `$SRC_SYNC_PATH`                 | GitLab source root group path to synchronize                                                                     | `to-be-continuous`                      |

| `$DEST_GITLAB_API`               | GitLab destination API url                                                                                       | `$CI_API_V4_URL`                        |

| `$DEST_TOKEN` or `$GITLAB_TOKEN` | GitLab destination token with at least scopes `api,read_repository,write_repository` and `Owner` role            | _none_ (**mandatory**)                  |

| `$DEST_SYNC_PATH`                | GitLab destination root group path to synchronize (defaults to `--src-sync-path`)                                | determined from `$CI_PROJECT_NAMESPACE` |

| `$MAX_VISIBILITY`                | maximum visibility of projects in destination group (defaults to `public`)                                       | `$CI_PROJECT_VISIBILITY`                |

| `$SKIP_VISIBILITY`               | skip updating the destination group or project visibility (when it exists already)                               | `false`                                 |

| `$EXCLUDE`                       | project/group path(s) to exclude (multiple CLI option; env. variable is a coma separated list)                   | `samples,custom`                        |

| `$INCLUDE`                       | project/group path(s) to include (multiple CLI option; env. variable is a coma separated list)                   | `tools,docker,ansible`                  |

| `$INSECURE`                      | skip SSL verification                                                                                            | `false`                                 |

| `$UPDATE_RELEASE`                | set to force the update of the latest release (in order to trigger GitLab CI/CD catalog publication)             | `false`                                 |

| `$UPDATE_AVATAR`                 | force update the avatar images even when they exist and look the same                                            | `false`                                 |

| `$GROUP_DESCRIPTION_DISABLED`    | don't synchronize group description                                                                              | `false`                                 |

| `$PROJECT_DESCRIPTION_DISABLED`  | don't synchronize project description                                                                            | `false`                                 |

| `$NEW_GROUP_OPTIONS`             | a JSON string with [extra options for groups creation](https://docs.gitlab.com/api/groups/#create-a-group)       | _none_                                  |

| `$NEW_PROJECT_OPTIONS`           | a JSON string with [extra options for projects creation](https://docs.gitlab.com/api/projects/#create-a-project) | _none_                                  |

| `$CACHE_DIR`                     | cache directory (used to download resources such as images and Git repositories) (defaults to `.work`)           | `.work`                                 |



### Customizing copied project options



Projects are copied with the following default options:



```json

{

  "issues_access_level": "disabled",

  "merge_requests_access_level": "disabled"

}

```



You can customize those default options with your own ones with the `$NEW_PROJECT_OPTIONS` variable.

For instance if you want to globally disable CI/CD pipelines in copied projects:



```json

{

  "builds_access_level": "disabled"

}

```



Similarly, you can customize default group options with the `$NEW_GROUP_OPTIONS` variable.



## CI/CD Catalog
@@ -43,10 +67,3 @@ To use _to be continuous_ templates as CI/CD Catalog resources in your GitLab se 
Then you'll have to re-publish a release to make the template appear in the CI/CD Catalog.



:information_source: You may re-run the GitLab Copy CLI tool with the `--update-release` (or `$UPDATE_RELEASE` variable) set to force re-publishig the latest release of each template.
 


## Legacy `gitlab-sync.sh` script



This project was previously using the `gitlab-sync.sh` script to perform the recursive copy/sync.

It has now been replaced by the [GitLab Copy CLI](https://gitlab.com/to-be-continuous/tools/gitlab-cp) tool that has numerous advantages over it.



Anyhow, the legacy script can still be reactivated by setting the `SYNC_TOOL` CI/CD project variable to `legacy`.

gitlab-sync.sh

deleted100755 → 0
+0 −535

File deleted.

Preview size limit exceeded, changes collapsed.