This project provides a script able to recursively copy/synchronize a GitLab group from one GitLab server to another.
This project allows to recursively copy/synchronize the [to be continuous](https://gitlab.com/to-be-continuous) project from gitlab.com to your self-managed GitLab instance.
It can be run manually (command line) and also as scheduled CI/CD job to regularly synchronize a GitLab group mirror.
It can be run manually (manual pipeline) and also as scheduled CI/CD pipeline to regularly retrieve upstream changes.
## Pre-requisites
Under the hood, it uses the [GitLab Copy CLI](https://gitlab.com/to-be-continuous/tools/gitlab-cp) tool as a Docker image, [directly pulled from gitlab.com](https://gitlab.com/to-be-continuous/tools/gitlab-cp/container_registry/6261230).
The GitLab Synchronization Script has the following requirements:
## Usage: CLI (first-time copy)
* Bash interpreter <br/>_Trivial on Linux or MacOS, tested with [Git Bash](https://www.atlassian.com/git/tutorials/git-bash) on Windows (available in [Git for Windows](https://gitforwindows.org/))_
*[curl tool](https://curl.se/) installed and accessible as `curl` command from the Bash interpreter
*[jq tool](https://stedolan.github.io/jq/download/) installed and accessible as `jq` command from the Bash interpreter
The GitLab Copy CLI tool can be used directly from your computer (this is the recommended way to clone _to be continuous_ for the first time).
## Usage: script
More info is available in [_to be continuous_ documentation](https://to-be-continuous.gitlab.io/doc/self-managed/basic/#copy-tbc-to-your-gitlab).
```bash
gitlab-sync.sh \
[--src-api{GitLab source API url}]\
[--src-token{GitLab source token}]\
[--src-sync-path{GitLab source root group path to synchronize}]\
--dest-api{GitLab destination API url}\
--dest-token{GitLab destination token}\
[--dest-sync-path{GitLab destination root group path to synchronize}]\
[--max-visibility{max visibility}]\
[--exclude{coma separated list of project/group path(s) to exclude}]\
[--insecure]\
[--update-release]\
[--no-group-description{do not synchronise group description}]\
[--no-project-description{do not synchronise project description}]
```
## Usage: CI/CD
Once copied _to be continuous_ to your GitLab server, you shall then schedule a pipeline in this project (`to-be-continuous/tools/gitlab-sync`) - for instance every night - to keep synchronized with the upstream project.
The job only requires a GitLab token, that shall be configured declaring a `$GITLAB_TOKEN` CI/CD project variable.
All the other parameters/variables are automatically determined by the job but might be overwritten with the appropriate project variables if need be:
| `--src-token` | `$SRC_TOKEN` | GitLab source token (_optional_ if source GitLab group and sub projects have `public` visibility) | _none_ |
| `--src-sync-path` | `$SRC_SYNC_PATH` | GitLab source root group path to synchronize | `to-be-continuous` |
| `--dest-api` | `$DEST_GITLAB_API` | GitLab destination API url (**mandatory**) | `$CI_API_V4_URL` (defined when running in GitLab CI) |
| `--dest-token` | `$DEST_TOKEN` or `$GITLAB_TOKEN` | GitLab destination token with at least scopes `api,read_repository,write_repository` and `Owner` role (**mandatory**) | _none_ |
| `--dest-sync-path` | `$DEST_SYNC_PATH` | GitLab destination root group path to synchronize | guessed from GitLab CI env or `to-be-continuous` |
| `--max-visibility` | `$MAX_VISIBILITY` | maximum visibility of projects in destination group | `$CI_PROJECT_VISIBILITY` (when running in GitLab CI) or `public` |
| `--exclude` | `$EXCLUDE` | coma separated list of project/group path(s) to exclude | _none_ |
| `--insecure` | `$INSECURE` | set to skip TLS check on curl requests | `false` |
| `--update-release` | `$UPDATE_RELEASE` | set to update the releases even if they exists | `false` |
| `--no-group-description` | `$GROUP_DESCRIPTION_DISABLED` | do not synchronise group description | _none_ |
| `--no-project-description` | `$PROJECT_DESCRIPTION_DISABLED` | do not synchronise project description | _none_ |
You shall use this script to copy the _to be continuous_ project to your own GitLab server for the first time with the following command:
```bash
curl -s https://gitlab.com/to-be-continuous/tools/gitlab-sync/-/raw/master/gitlab-sync.sh | bash /dev/stdin --dest-api{your GitLab server API url}--dest-token{your GitLab token}--exclude samples,custom
| `--dest-token` | `$DEST_TOKEN` or `$GITLAB_TOKEN` | GitLab destination token with at least scopes `api,read_repository,write_repository` and `Owner` role | _none_ (**mandatory**) |
| `--dest-sync-path` | `$DEST_SYNC_PATH` | GitLab destination root group path to synchronize (defaults to `--src-sync-path`) | determined from `$CI_PROJECT_NAMESPACE` |
| `--max-visibility` | `$MAX_VISIBILITY` | maximum visibility of projects in destination group (defaults to `public`) | `$CI_PROJECT_VISIBILITY` |
| `--exclude` | `$EXCLUDE` | project/group path(s) to exclude (multiple CLI option; env. variable is a coma separated list) | `samples,custom` |
| `--update-release` | `$UPDATE_RELEASE` | set to force the update of the latest release (in order to trigger GitLab CI/CD catalog publication) | `false` |
| `--update-avatar` | `$UPDATE_AVATAR` | force update the avatar images even when they exist and look the same | `false` |
| `--dry-run` | _none_ | dry run (don't execute any write action) | `false` |
| `--halt-on-error` | _none_ | halt synchronizing when an error occurs | `false` |
| `--cache-dir` | `$CACHE_DIR` | cache directory (used to download resources such as images and Git repositories) (defaults to `.work`) | `.work` |
:warning: Each CLI option may alternately be specified with an environment variable (see in the table above). This might be useful to configure the CI/CD job.
## Usage: CI/CD
## CI/CD Catalog
Once copied _to be continuous_ to your GitLab server, you shall then schedule a pipeline in this project (`to-be-continuous/tools/gitlab-sync`) - for instance every night - to keep synchronized with source project.
To use _to be continuous_ templates as CI/CD Catalog resources in your GitLab server, you'll have to manually activate the _CI/CD Catalog resource_ option (`Settings > General > Visibility > Project Features > Permissions`) in each template project.
Then you'll have to re-publish a release to make the template appear in the CI/CD Catalog.
The script will only require a GitLab token, that shall be configured declaring a `$GITLAB_TOKEN` CI/CD project variable. (`--dest-api` will be implicitly retrieved using predefined `$CI_API_V4_URL`).
:information_source: You may re-run the GitLab Copy CLI tool with the `--update-release` (or `$UPDATE_RELEASE` variable) set to force re-publishig the latest release of each template.
You can define a custom alpine image to use by setting `$GITLAB_SYNC_IMAGE` (default to `registry.hub.docker.com/alpine:latest`)
## Legacy `gitlab-sync.sh` script
## CI/CD Catalog
This project was previously using the `gitlab-sync.sh` script to perform the recursive copy/sync.
It has now been replaced by the [GitLab Copy CLI](https://gitlab.com/to-be-continuous/tools/gitlab-cp) tool that has numerous advantages over it.
To use _to be continuous_ templates as CI/CD Catalog resources on a your GitLab Server, you have to manually activate the _CI/CD Catalog resource_ option in `Settings`/`General`/`Visibility, project features, permissions`.
No newline at end of file
Anyhow, the legacy script can still be reactivated by setting the `SYNC_TOOL` CI/CD project variable to `legacy`.
