Merge branch 'feat/poetry-improvments' into 'master' (dda82d21) · Commits · to be continuous... / Python

README.md

+89 −64

Original line number	Diff line number	Diff line
		@@ -2,7 +2,7 @@

		This project implements a generic GitLab CI template for [Python](https://www.python.org/).

		It provides several features, usable in different modes (by configuration) following those [recommendations](to-be-continuous.gitlab.io/doc/usage/)
		It provides several features, usable in different modes (by configuration).

		## Usage

		@@ -22,33 +22,39 @@ The Python template uses some global configuration used throughout all jobs.
		\| Name \| description \| default value \|
		\| -------------------- \| ------------------------------------------------------------------------------------- \| ------------------ \|
		\| `PYTHON_IMAGE` \| The Docker image used to run Python <br/>:warning: set the version required by your project \| `python:3` \|
		\| `PIP_INDEX_URL` \| Python repository url \| _none_ \|
		\| `PYTHON_PROJECT_DIR` \| Python project root directory \| `.` \|
		\| `REQUIREMENTS_FILE` \| Path to requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ \| `requirements.txt` \|
		\| `PIP_OPTS` \| pip extra [options](https://pip.pypa.io/en/stable/reference/pip/#general-options) \| _none_ \|
		\| `PYTHON_BUILD_SYSTEM`\| Python build-system to use to install dependencies, build and package the project (see below) \| _none_ (auto-detect) \|
		\| `PIP_INDEX_URL` \| Python repository url \| _none_ \|
		\| `PIP_OPTS` \| pip [extra options](https://pip.pypa.io/en/stable/reference/pip/#general-options) \| _none_ \|
		\| `PYTHON_EXTRA_DEPS` \| Python extra sets of dependencies to install<br/>For [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies) or [Poetry](https://python-poetry.org/docs/pyproject/#extras) only \| _none_ \|
		\| `PYTHON_REQS_FILE` \| Main requirements file _(relative to `$PYTHON_PROJECT_DIR`)_<br/>For [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only \| `requirements.txt` \|
		\| `PYTHON_EXTRA_REQS_FILES` \| Extra dev requirements file(s) to install _(relative to `$PYTHON_PROJECT_DIR`)_ \| `requirements-dev.txt` \|

		The cache policy also declares the `.cache/pip` directory as cached (not to download Python dependencies over and over again).
		The cache policy also makes the necessary to manage pip cache (not to download Python dependencies over and over again).

		Default configuration follows [this Python project structure](https://docs.python-guide.org/writing/structure/)
		## Multi build-system support

		### Poetry support
		The Python template supports the most popular dependency management & build systems.

		The Python template supports [Poetry](https://python-poetry.org/) as packaging and dependency management tool.
		By default it tries to auto-detect the build system used by the project (based on the presence of `pyproject.toml`
		and/or `setup.py` and/or `requirements.txt`), but the build system might also be set explicitly using the
		`$PYTHON_BUILD_SYSTEM` variable:

		If a `pyproject.toml` file is detected at the root of your Python project, requirements will automatically be generated from Poetry.
		Poetry support can be explicitly disabled by setting `PYTHON_POETRY_DISABLED` to `true`.
		\| Value \| Build System (scope) \|
		\| ---------------- \| ---------------------------------------------------------- \|
		\| _none_ (default) or `auto` \| The template tries to auto-detect the actual build system \|
		\| `setuptools` \| [Setuptools](https://setuptools.pypa.io/) (dependencies, build & packaging) \|
		\| `poetry` \| [Poetry](https://python-poetry.org/) (dependencies, build, test & packaging) \|
		\| `pipenv` \| [Pipenv](https://pipenv.pypa.io/) (dependencies only) \|
		\| `reqfile` \| [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) (dependencies only) \|

		:warning: If no `poetry.lock` file is found, the template will emit a (non-blocking) warning message, to enforce [Poetry recommendation](https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control):
		## Jobs

		> You should commit the `poetry.lock` file to your project repo so that all people working on the project are locked to the same versions of dependencies.
		### `py-package` job

		Poetry support uses the following variables:
		This job allows building your Python project [distribution packages](https://packaging.python.org/en/latest/glossary/#term-Distribution-Package).

		\| Name \| description \| default value \|
		\| ------------------------ \| ---------------------------------------------------------- \| ----------------- \|
		\| `PYTHON_POETRY_EXTRAS` \| Poetry [extra sets of dependencies](https://python-poetry.org/docs/pyproject/#extras) to include, space separated \| _none_ \|

		## Jobs
		It is bound to the `build` stage, it is disabled by default and can be enabled by setting `$PYTHON_PACKAGE_ENABLED` to `true`.

		### Lint jobs

		@@ -88,7 +94,6 @@ It is bound to the `build` stage, and uses the following variables:

		\| Name \| description \| default value \|
		\| ------------------------ \| -------------------------------------------------------------------- \| ----------------------- \|
		\| `TEST_REQUIREMENTS_FILE` \| Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ \| `test-requirements.txt` \|
		\| `UNITTEST_ARGS` \| Additional xmlrunner/unittest CLI options \| _none_ \|

		This job produces the following artifacts, kept for one day:
		@@ -119,7 +124,6 @@ It is bound to the `build` stage, and uses the following variables:

		\| Name \| description \| default value \|
		\| ------------------------ \| --------------------------------------------------------------------------------------------------------------------------------------------- \| ----------------------- \|
		\| `TEST_REQUIREMENTS_FILE` \| Path to test requirements file _(relative `$PYTHON_PROJECT_DIR`)_ \| `test-requirements.txt` \|
		\| `PYTEST_ARGS` \| Additional [pytest](https://docs.pytest.org/en/stable/usage.html) or [pytest-cov](https://github.com/pytest-dev/pytest-cov#usage) CLI options \| _none_ \|

		This job produces the following artifacts, kept for one day:
		@@ -150,7 +154,6 @@ It is bound to the `build` stage, and uses the following variables:

		\| Name \| description \| default value \|
		\| ------------------------ \| --------------------------------------------------------------------------------------- \| ----------------------- \|
		\| `TEST_REQUIREMENTS_FILE` \| Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ \| `test-requirements.txt` \|
		\| `NOSETESTS_ARGS` \| Additional [nose CLI options](https://nose.readthedocs.io/en/latest/usage.html#options) \| _none_ \|

		By default coverage will be run on all the directory. You can restrict it to your packages by setting NOSE_COVER_PACKAGE variable.
		@@ -243,64 +246,86 @@ It is bound to the `test` stage, and uses the following variables:
		This job outputs a textual report in the console, and in case of failure also exports a JSON report in the `reports/`
		directory _(relative to project root dir)_.

		### Package jobs
		### `py-release` job

		#### `py-package` job
		This job is disabled by default and allows to perform a complete release of your Python code:

		This job is disabled by default and performs a packaging of your Python code.
		1. increase the Python project version,
		2. Git commit changes and create a Git tag with the new version number,
		3. build the [Python packages](https://packaging.python.org/),
		4. publish the built packages to a PyPI compatible repository ([GitLab packages](https://docs.gitlab.com/ee/user/packages/pypi_repository/) by default).

		It is bound to the `package-build` stage, applies only on git tags and uses the following variables:
		The Python template supports two packaging systems:

		\| Name \| description \| default value \|
		\| --------------- \| ---------------------------------------------------- \| ------------- \|
		\| `PYTHON_FORCE_PACKAGE` \| Set to `true` to force the packaging even if not on tag related event \| _none_ (disabled) \|
		* [Poetry](https://python-poetry.org/): uses Poetry-specific [version](https://python-poetry.org/docs/cli/#version), [build](https://python-poetry.org/docs/cli/#build) and [publish](https://python-poetry.org/docs/cli/#publish) commands.
		* [Setuptools](https://setuptools.pypa.io/): uses [Bumpversion](https://github.com/peritus/bumpversion) as version management, [build](https://pypa-build.readthedocs.io/) as package builder and [Twine](https://twine.readthedocs.io/) to publish.

		### Publish jobs
		The release job is bound to the `publish` stage, appears only on production and integration branches and uses the following variables:

		#### `py-release` job
		\| Name \| description \| default value \|
		\| ----------------------- \| ----------------------------------------------------------------------- \| ----------------- \|
		\| `PYTHON_RELEASE_ENABLED`\| Set to `true` to enable the release job \| _none_ (disabled) \|
		\| `PYTHON_RELEASE_NEXT` \| The part of the version to increase (one of: `major`, `minor`, `patch`) \| `minor` \|
		\| `PYTHON_SEMREL_RELEASE_DISABLED`\| Set to `true` to disable [semantic-release integration](#semantic-release-integration) \| _none_ (disabled) \|
		\| `GIT_USERNAME` \| Git username for Git push operations (see below) \| _none_ \|
		\| :lock: `GIT_PASSWORD` \| Git password for Git push operations (see below) \| _none_ \|
		\| :lock: `GIT_PRIVATE_KEY`\| SSH key for Git push operations (see below) \| _none_ \|
		\| `PYTHON_REPOSITORY_URL`\| Target PyPI repository to publish packages \| _[GitLab project's PyPI packages repository](https://docs.gitlab.com/ee/user/packages/pypi_repository/)_ \|
		\| `PYTHON_REPOSITORY_USERNAME`\| Target PyPI repository username credential \| `gitlab-ci-token` \|
		\| :lock: `PYTHON_REPOSITORY_PASSWORD`\| Target PyPI repository password credential \| `$CI_JOB_TOKEN` \|

		#### Setuptools tip

		If you're using a `setup.cfg` declarative file for your project Setuptools configuration, then you will have to write a
		`.bumpversion.cfg` file to workaround a bug that prevents Bumpversion from updating the project version in your `setup.cfg` file.

		Example of `.bumpversion.cfg` file:

		```ini
		[bumpversion]
		# same version as in your setup.cfg
		current_version = 0.5.0

		[bumpversion:file:setup.cfg]
		# any additional config here
		# see: https://github.com/peritus/bumpversion#file-specific-configuration
		```

		This job is disabled by default and performs an automatic tagging of your Python code.
		#### `semantic-release` integration

		* [Bumpversion](https://github.com/peritus/bumpversion) Python library is used for version management.
		* Looks for an existing `.bumpversion.cfg` at the project root. If found, it will be the configuration used by bumpversion. If not, the `$RELEASE_VERSION_PART` variable and `setup.py` will be used instead.
		* Creating a Git tag involves an authenticated and authorized Git user.
		If you activate the [`semantic-release-info` job from the `semantic-release` template](https://gitlab.com/to-be-continuous/semantic-release/#semantic-release-info-job), the `py-release` job will rely on the generated next version info.
		Thus, a release will be performed only if a next semantic release is present.

		**Don't use your personal password !!!
		Use an [access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) with write_repository rights.
		If you have a generic account, add it to the project and generate access token from this account.**
		You should disable the `semantic-release` job (as it's the `py-release` job that will perform the release and so we only need the `semantic-release-info` job) by setting `SEMREL_RELEASE_DISABLED` to `true`.

		It is bound to the `publish` stage, applies only on master branch and uses the following variables:
		Finally, the semantic-release integration can be disabled with the `PYTHON_SEMREL_RELEASE_DISABLED` variable.

		\| Name \| description \| default value \|
		\| ---------------------- \| ----------------------------------------------------------------------- \| ----------------- \|
		\| `RELEASE_VERSION_PART` \| The part of the version to increase (one of: `major`, `minor`, `patch`) \| `minor` \|
		\| `RELEASE_USERNAME` \| Username credential for git push \| _none_ (disabled) \|
		\| `RELEASE_ACCESS_TOKEN` \| Password credential for git push \| _none_ \|
		####

		#### `py-publish` job
		#### Git authentication

		This job is disabled by default and performs a publication of your Python code.
		A Python release involves some Git push operations.

		It is bound to the `publish` stage, applies only on git tags and uses the following variables:
		You can either use a SSH key or user/password credentials.

		\| Name \| description \| default value \|
		\| ---------------------- \| -------------------------------------------------------- \| ----------------- \|
		\| `PYTHON_PUBLISH_ENABLED`\| Set to `true` to enable the publish job \| _none_ (disabled) \|
		\| `TWINE_REPOSITORY_URL` \| Where to publish your Python project \| GitLab Project's Pypi Packages registry \|
		\| `TWINE_USERNAME` \| Username credential to publish to \$TWINE_REPOSITORY_URL \| `gitlab-ci-token` \|
		\| `TWINE_PASSWORD` \| Password credential to publish to \$TWINE_REPOSITORY_URL \| `$CI_JOB_TOKEN` \|
		##### Using a SSH key

		More info:
		We recommend you to use a [project deploy key](https://docs.gitlab.com/ee/user/project/deploy_keys/#project-deploy-keys) with write access to your project.

		The key should not have a passphrase (see [how to generate a new SSH key pair](https://docs.gitlab.com/ce/ssh/README.html#generating-a-new-ssh-key-pair)).

		* [Python Packaging User Guide](https://packaging.python.org/)
		* [PyPI packages in the Package Registry](https://docs.gitlab.com/ee/user/packages/pypi_repository/)
		Specify :lock: `$GIT_PRIVATE_KEY` as secret project variable with the private part of the deploy key.

		If you want to automatically create tag and publish your Python package, please have a look [here](#release-python)
		```PEM
		-----BEGIN OPENSSH PRIVATE KEY-----
		blablabla
		-----END OPENSSH PRIVATE KEY-----
		```

		#### `py-docs` job
		The template handles both classic variable and file variable.

		This job is no longer supported in this version of the template. It might come back later on with a more generic & configurable implementation.
		##### Using user/password credentials

		## GitLab compatibility
		Simply specify :lock: `$GIT_USERNAME` and :lock: `$GIT_PASSWORD` as secret project variables.

		:information_source: This template is actually tested and validated on GitLab Community Edition instance version 13.12.11
		Note that the password should be an access token (preferably a [project](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) or [group](https://docs.gitlab.com/ee/user/group/settings/group_access_tokens.html) access token) with `read_repository` and `write_repository` scopes.

kicker.json

+53 −74

Original line number	Diff line number	Diff line
		@@ -15,11 +15,25 @@
		"default": "."
		},
		{
		"name": "REQUIREMENTS_FILE",
		"description": "Full path to `requirements.txt` file _(relative to `$PYTHON_PROJECT_DIR`)_",
		"name": "PYTHON_BUILD_SYSTEM",
		"description": "Python build-system to use to install dependencies, build and package the project",
		"type": "enum",
		"values": ["auto", "setuptools", "poetry", "pipenv", "reqfile"],
		"default": "auto",
		"advanced": true
		},
		{
		"name": "PYTHON_REQS_FILE",
		"description": "Main requirements file _(relative to `$PYTHON_PROJECT_DIR`)_\n\nFor [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only",
		"default": "requirements.txt",
		"advanced": true
		},
		{
		"name": "PYTHON_EXTRA_REQS_FILES",
		"description": "Extra dev requirements file(s) to install _(relative to `$PYTHON_PROJECT_DIR`)_\n\nFor [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only",
		"default": "requirements-dev.txt",
		"advanced": true
		},
		{
		"name": "PYTHON_COMPILE_ARGS",
		"description": "[`compileall` CLI options](https://docs.python.org/3/library/compileall.html)",
		@@ -32,15 +46,8 @@
		"advanced": true
		},
		{
		"name": "PYTHON_POETRY_DISABLED",
		"description": "Disable poetry support",
		"type": "boolean",
		"advanced": true
		},
		{
		"name": "PYTHON_POETRY_EXTRAS",
		"description": "Poetry [extra sets of dependencies](https://python-poetry.org/docs/pyproject/#extras) to include, space separated",
		"advanced": true
		"name": "PYTHON_EXTRA_DEPS",
		"description": "Extra sets of dependencies to install\n\nFor [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies) or [Poetry](https://python-poetry.org/docs/pyproject/#extras) only"
		}
		],
		"features": [
		@@ -68,12 +75,6 @@
		"description": "Unit tests based on [unittest](https://docs.python.org/3/library/unittest.html) framework",
		"enable_with": "UNITTEST_ENABLED",
		"variables": [
		{
		"name": "TEST_REQUIREMENTS_FILE",
		"description": "Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_",
		"default": "test-requirements.txt",
		"advanced": true
		},
		{
		"name": "UNITTEST_ARGS",
		"description": "Additional xmlrunner/unittest CLI options",
		@@ -87,12 +88,6 @@
		"description": "Unit tests based on [pytest](https://docs.pytest.org/) framework",
		"enable_with": "PYTEST_ENABLED",
		"variables": [
		{
		"name": "TEST_REQUIREMENTS_FILE",
		"description": "Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_",
		"default": "test-requirements.txt",
		"advanced": true
		},
		{
		"name": "PYTEST_ARGS",
		"description": "Additional [pytest](https://docs.pytest.org/en/stable/usage.html) or [pytest-cov](https://github.com/pytest-dev/pytest-cov#usage) CLI options",
		@@ -106,12 +101,6 @@
		"description": "Unit tests based on [nose](https://nose.readthedocs.io/) framework",
		"enable_with": "NOSETESTS_ENABLED",
		"variables": [
		{
		"name": "TEST_REQUIREMENTS_FILE",
		"description": "Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_",
		"default": "test-requirements.txt",
		"advanced": true
		},
		{
		"name": "NOSETESTS_ARGS",
		"description": "Additional [nose CLI options](https://nose.readthedocs.io/en/latest/usage.html#options)",
		@@ -161,51 +150,14 @@
		}
		]
		},
		{
		"id": "package",
		"name": "package",
		"description": "Packaging of your Python code",
		"variables": [
		{
		"name": "PYTHON_FORCE_PACKAGE",
		"description": "Force the packaging even if not on tag related event",
		"type": "boolean"
		}
		]
		},
		{
		"id": "publish",
		"name": "Publish",
		"description": "Publish your code to a [Twine](https://pypi.org/project/twine/) repository",
		"enable_with": "PYTHON_PUBLISH_ENABLED",
		"variables": [
		{
		"name": "TWINE_REPOSITORY_URL",
		"type": "url",
		"description": "Twine repository url to publish you python project",
		"default": "${CI_SERVER_URL}/api/v4/projects/${CI_PROJECT_ID}/packages/pypi"
		},
		{
		"name": "TWINE_USERNAME",
		"description": "Twine repository username credential",
		"secret": true,
		"default": "gitlab-ci-token"
		},
		{
		"name": "TWINE_PASSWORD",
		"description": "Twine repository password credential",
		"secret": true,
		"default": "$CI_JOB_TOKEN"
		}
		]
		},
		{
		"id": "release",
		"name": "Release",
		"description": "Manually trigger a release of your code (uses [bumpversion](https://pypi.org/project/bumpversion/))",
		"enable_with": "PYTHON_RELEASE_ENABLED",
		"variables": [
		{
		"name": "RELEASE_VERSION_PART",
		"name": "PYTHON_RELEASE_NEXT",
		"type": "enum",
		"values": [
		"",
		@@ -218,16 +170,43 @@
		"advanced": true
		},
		{
		"name": "RELEASE_USERNAME",
		"description": "Username credential for Git push",
		"name": "PYTHON_SEMREL_RELEASE_DISABLED",
		"description": "Disable semantic-release integration",
		"type": "boolean",
		"advanced": true
		},
		{
		"name": "GIT_USERNAME",
		"description": "Git username for Git push operations",
		"secret": true
		},
		{
		"name": "GIT_PASSWORD",
		"description": "Git password for Git push operations",
		"secret": true
		},
		{
		"name": "GIT_PRIVATE_KEY",
		"description": "SSH key for Git push operations",
		"secret": true
		},
		{
		"name": "PYTHON_REPOSITORY_URL",
		"type": "url",
		"description": "Target PyPI repository to publish packages.\n\n_defaults to [GitLab project's packages repository](https://docs.gitlab.com/ee/user/packages/pypi_repository/)_",
		"default": "${CI_SERVER_URL}/api/v4/projects/${CI_PROJECT_ID}/packages/pypi"
		},
		{
		"name": "PYTHON_REPOSITORY_USERNAME",
		"description": "Target PyPI repository username credential",
		"secret": true,
		"mandatory": true
		"default": "gitlab-ci-token"
		},
		{
		"name": "RELEASE_ACCESS_TOKEN",
		"description": "Password credential for Git push",
		"name": "PYTHON_REPOSITORY_PASSWORD",
		"description": "Target PyPI repository password credential",
		"secret": true,
		"mandatory": true
		"default": "$CI_JOB_TOKEN"
		}
		]
		}

templates/gitlab-ci-python.yml

+320 −138

File changed.

Preview size limit exceeded, changes collapsed.