feat: add multi build-system support (Poetry, Setuptools or requirements file) (130e2102) · Commits · to be continuous... / Python

README.md

+22 −25

Original line number	Diff line number	Diff line
		@@ -22,31 +22,35 @@ The Python template uses some global configuration used throughout all jobs.
		\| Name \| description \| default value \|
		\| -------------------- \| ------------------------------------------------------------------------------------- \| ------------------ \|
		\| `PYTHON_IMAGE` \| The Docker image used to run Python <br/>:warning: set the version required by your project \| `python:3` \|
		\| `PIP_INDEX_URL` \| Python repository url \| _none_ \|
		\| `PYTHON_PROJECT_DIR` \| Python project root directory \| `.` \|
		\| `REQUIREMENTS_FILE` \| Path to requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ \| `requirements.txt` \|
		\| `PIP_OPTS` \| pip extra [options](https://pip.pypa.io/en/stable/reference/pip/#general-options) \| _none_ \|

		The cache policy also declares the `.cache/pip` directory as cached (not to download Python dependencies over and over again).
		\| `PYTHON_BUILD_SYSTEM`\| Python build-system to use to install dependencies, build and package the project (see below) \| _none_ (auto-detect) \|
		\| `PIP_INDEX_URL` \| Python repository url \| _none_ \|
		\| `PIP_OPTS` \| pip [extra options](https://pip.pypa.io/en/stable/reference/pip/#general-options) \| _none_ \|
		\| `PYTHON_EXTRA_DEPS` \| Python extra sets of dependencies to install<br/>For [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies) or [Poetry](https://python-poetry.org/docs/pyproject/#extras) only \| _none_ \|
		\| `REQUIREMENTS_FILE` \| Name of requirements file _(relative to `$PYTHON_PROJECT_DIR`)_<br/>For [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only \| `requirements.txt` \|
		\| `TEST_REQUIREMENTS_FILE` \| Name of dev/test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ \| `test-requirements.txt` \|

		Default configuration follows [this Python project structure](https://docs.python-guide.org/writing/structure/)
		The cache policy also makes the necessary to manage pip cache (not to download Python dependencies over and over again).

		### Poetry support
		## Multi build-system support

		The Python template supports [Poetry](https://python-poetry.org/) as packaging and dependency management tool.
		The Python template supports 3 popular dependency management & build systems:

		If a `pyproject.toml` file is detected at the root of your Python project, requirements will automatically be generated from Poetry.
		Poetry support can be explicitly disabled by setting `PYTHON_POETRY_DISABLED` to `true`.
		* [Setuptools](https://setuptools.pypa.io/),
		* [Poetry](https://python-poetry.org/),
		* [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) (dependency management only).

		:warning: If no `poetry.lock` file is found, the template will emit a (non-blocking) warning message, to enforce [Poetry recommendation](https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control):
		By default the template tries to auto-detect the build system used by the project (based on presence of `pyproject.toml`
		and/or `setup.py` and/or `requirements.txt`), but the build system might also be explicitly set using the `$PYTHON_BUILD_SYSTEM` variable.

		> You should commit the `poetry.lock` file to your project repo so that all people working on the project are locked to the same versions of dependencies.
		Supported values of `$PYTHON_BUILD_SYSTEM`:

		Poetry support uses the following variables:

		\| Name \| description \| default value \|
		\| ------------------------ \| ---------------------------------------------------------- \| ----------------- \|
		\| `PYTHON_POETRY_EXTRAS` \| Poetry [extra sets of dependencies](https://python-poetry.org/docs/pyproject/#extras) to include, space separated \| _none_ \|
		\| Value \| Description \|
		\| ---------------- \| ---------------------------------------------------------- \|
		\| _none_ (default) \| The template tries to auto-detect the actual build system, based of the presence of some key files \|
		\| `setuptools` \| [Setuptools](https://setuptools.pypa.io/) will be used to install dependencies, build and package the project \|
		\| `poetry` \| [Poetry](https://python-poetry.org/) will be used to install dependencies, build, test and package the project \|
		\| `reqfile` \| [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) will be used to install dependencies \|

		## Jobs

		@@ -88,7 +92,6 @@ It is bound to the `build` stage, and uses the following variables:

		\| Name \| description \| default value \|
		\| ------------------------ \| -------------------------------------------------------------------- \| ----------------------- \|
		\| `TEST_REQUIREMENTS_FILE` \| Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ \| `test-requirements.txt` \|
		\| `UNITTEST_ARGS` \| Additional xmlrunner/unittest CLI options \| _none_ \|

		This job produces the following artifacts, kept for one day:
		@@ -119,7 +122,6 @@ It is bound to the `build` stage, and uses the following variables:

		\| Name \| description \| default value \|
		\| ------------------------ \| --------------------------------------------------------------------------------------------------------------------------------------------- \| ----------------------- \|
		\| `TEST_REQUIREMENTS_FILE` \| Path to test requirements file _(relative `$PYTHON_PROJECT_DIR`)_ \| `test-requirements.txt` \|
		\| `PYTEST_ARGS` \| Additional [pytest](https://docs.pytest.org/en/stable/usage.html) or [pytest-cov](https://github.com/pytest-dev/pytest-cov#usage) CLI options \| _none_ \|

		This job produces the following artifacts, kept for one day:
		@@ -150,7 +152,6 @@ It is bound to the `build` stage, and uses the following variables:

		\| Name \| description \| default value \|
		\| ------------------------ \| --------------------------------------------------------------------------------------- \| ----------------------- \|
		\| `TEST_REQUIREMENTS_FILE` \| Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ \| `test-requirements.txt` \|
		\| `NOSETESTS_ARGS` \| Additional [nose CLI options](https://nose.readthedocs.io/en/latest/usage.html#options) \| _none_ \|

		By default coverage will be run on all the directory. You can restrict it to your packages by setting NOSE_COVER_PACKAGE variable.
		@@ -297,10 +298,6 @@ More info:

		If you want to automatically create tag and publish your Python package, please have a look [here](#release-python)

		#### `py-docs` job

		This job is no longer supported in this version of the template. It might come back later on with a more generic & configurable implementation.

		## GitLab compatibility

		:information_source: This template is actually tested and validated on GitLab Community Edition instance version 13.12.11

kicker.json

+16 −28

Original line number	Diff line number	Diff line
		@@ -14,12 +14,25 @@
		"description": "Python project root directory",
		"default": "."
		},
		{
		"name": "PYTHON_BUILD_SYSTEM",
		"description": "Python build-system to use to install dependencies, build and package the project",
		"type": "enum",
		"values": ["", "setuptools", "poetry", "reqfile"],
		"advanced": true
		},
		{
		"name": "REQUIREMENTS_FILE",
		"description": "Full path to `requirements.txt` file _(relative to `$PYTHON_PROJECT_DIR`)_",
		"description": "Name of requirements file _(relative to `$PYTHON_PROJECT_DIR`)_\n\nFor [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only",
		"default": "requirements.txt",
		"advanced": true
		},
		{
		"name": "TEST_REQUIREMENTS_FILE",
		"description": "Name of dev/test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_\n\nFor [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only",
		"default": "test-requirements.txt",
		"advanced": true
		},
		{
		"name": "PYTHON_COMPILE_ARGS",
		"description": "[`compileall` CLI options](https://docs.python.org/3/library/compileall.html)",
		@@ -32,15 +45,8 @@
		"advanced": true
		},
		{
		"name": "PYTHON_POETRY_DISABLED",
		"description": "Disable poetry support",
		"type": "boolean",
		"advanced": true
		},
		{
		"name": "PYTHON_POETRY_EXTRAS",
		"description": "Poetry [extra sets of dependencies](https://python-poetry.org/docs/pyproject/#extras) to include, space separated",
		"advanced": true
		"name": "PYTHON_EXTRA_DEPS",
		"description": "Extra sets of dependencies to install\n\nFor [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies) or [Poetry](https://python-poetry.org/docs/pyproject/#extras) only"
		}
		],
		"features": [
		@@ -68,12 +74,6 @@
		"description": "Unit tests based on [unittest](https://docs.python.org/3/library/unittest.html) framework",
		"enable_with": "UNITTEST_ENABLED",
		"variables": [
		{
		"name": "TEST_REQUIREMENTS_FILE",
		"description": "Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_",
		"default": "test-requirements.txt",
		"advanced": true
		},
		{
		"name": "UNITTEST_ARGS",
		"description": "Additional xmlrunner/unittest CLI options",
		@@ -87,12 +87,6 @@
		"description": "Unit tests based on [pytest](https://docs.pytest.org/) framework",
		"enable_with": "PYTEST_ENABLED",
		"variables": [
		{
		"name": "TEST_REQUIREMENTS_FILE",
		"description": "Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_",
		"default": "test-requirements.txt",
		"advanced": true
		},
		{
		"name": "PYTEST_ARGS",
		"description": "Additional [pytest](https://docs.pytest.org/en/stable/usage.html) or [pytest-cov](https://github.com/pytest-dev/pytest-cov#usage) CLI options",
		@@ -106,12 +100,6 @@
		"description": "Unit tests based on [nose](https://nose.readthedocs.io/) framework",
		"enable_with": "NOSETESTS_ENABLED",
		"variables": [
		{
		"name": "TEST_REQUIREMENTS_FILE",
		"description": "Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_",
		"default": "test-requirements.txt",
		"advanced": true
		},
		{
		"name": "NOSETESTS_ARGS",
		"description": "Additional [nose CLI options](https://nose.readthedocs.io/en/latest/usage.html#options)",

templates/gitlab-ci-python.yml

+153 −80

Original line number	Diff line number	Diff line
		@@ -44,13 +44,6 @@ variables:
		PYTHON_TRIVY_IMAGE: aquasec/trivy:latest
		PYTHON_TRIVY_ARGS: "--vuln-type library"


		# Docs
		DOCS_REQUIREMENTS_FILE: docs-requirements.txt
		DOCS_DIRECTORY: docs
		DOCS_BUILD_DIR: public
		DOCS_MAKE_ARGS: html BUILDDIR=${DOCS_BUILD_DIR}

		RELEASE_VERSION_PART: "minor"

		# By default, publish on the Packages registry of the project
		@@ -219,46 +212,107 @@ variables:
		log_info "... done"
		}

		function guess_build_system() {
		if [[ "$PYTHON_BUILD_SYSTEM" ]]
		then
		case "$PYTHON_BUILD_SYSTEM" in
		poetry)
		log_info "--- Build system explictly declared: Poetry"
		return
		;;
		setuptools)
		log_info "--- Build system explictly declared: Setuptools"
		return
		;;
		reqfile)
		log_info "--- Build system explictly declared: requirements file"
		return
		;;
		*)
		log_warn "--- Unknown declared build system: \\e[33;1m${PYTHON_BUILD_SYSTEM}\\e[0m: please read template doc"
		;;
		esac
		fi

		if [[ -f "pyproject.toml" ]]
		then
		# that might be PEP 517 if a build-backend is specified
		# otherwise it might be only used as configuration file for development tools...
		build_backend=$(sed -rn 's/^build-backend = "([^"])"./\1/p' pyproject.toml)
		if [[ "$build_backend" ]]
		then
		case "$build_backend" in
		poetry.core.masonry.api)
		log_info "--- Build system auto-detected: PEP 517 with Poetry backend"
		export PYTHON_BUILD_SYSTEM="poetry"
		return
		;;
		setuptools.build_meta)
		log_info "--- Build system auto-detected: PEP 517 with Setuptools backend"
		export PYTHON_BUILD_SYSTEM="setuptools"
		return
		;;
		*)
		log_error "--- Build system auto-detected: PEP 517 with unsupported backend \\e[33;1m${build_backend}\\e[0m: please read template doc"
		exit 1
		;;
		esac
		fi
		fi

		if [[ -f "setup.py" ]]
		then
		log_info "--- Build system auto-detected: Setuptools (legacy)"
		export PYTHON_BUILD_SYSTEM="setuptools"
		elif [[ -f "${REQUIREMENTS_FILE}" ]]
		then
		log_info "--- Build system auto-detected: requirements file"
		export PYTHON_BUILD_SYSTEM="reqfile"
		else
		log_error "--- Build system auto-detect failed: please read template doc"
		exit 1
		fi
		}

		# install requirements
		# arg1: 'build' (build only) or 'test' (build + test)
		function install_requirements() {
		target=$1
		if [[ -f "pyproject.toml" ]] && [[ "${PYTHON_POETRY_DISABLED}" != "true" ]]; then
		case "$PYTHON_BUILD_SYSTEM" in
		poetry)
		if [[ ! -f "poetry.lock" ]]; then
		log_warn "Poetry detected but \\e[33;1mpoetry.lock\\e[0m file not found: you shall commit it with your project files"
		fi
		pip install poetry
		if [[ "$target" == "build" ]]; then
		log_info "--- Poetry detected: install build only requirements"
		poetry install --no-dev ${PYTHON_POETRY_EXTRAS:+--extras "$PYTHON_POETRY_EXTRAS"}
		else
		log_info "--- Poetry detected: install build and dev requirements"
		poetry install ${PYTHON_POETRY_EXTRAS:+--extras "$PYTHON_POETRY_EXTRAS"}
		log_warn "Using Poetry but \\e[33;1mpoetry.lock\\e[0m file not found: you shall commit it with your project files"
		fi
		elif [[ -f "${REQUIREMENTS_FILE}" ]]; then
		# shellcheck disable=SC2086
		pip install ${PIP_OPTS} poetry
		poetry install ${PYTHON_EXTRA_DEPS:+--extras "$PYTHON_EXTRA_DEPS"}
		;;
		setuptools)
		# shellcheck disable=SC2086
		pip install ${PIP_OPTS} setuptools
		# shellcheck disable=SC2086
		pip install ${PIP_OPTS} ".${PYTHON_EXTRA_DEPS:+[$PYTHON_EXTRA_DEPS]}"
		;;
		reqfile)
		if [[ -f "${REQUIREMENTS_FILE}" ]]; then
		log_info "--- installing build requirements from \\e[33;1m${REQUIREMENTS_FILE}\\e[0m"
		# shellcheck disable=SC2086
		pip install ${PIP_OPTS} -r "${REQUIREMENTS_FILE}"
		if [[ "$target" == "test" ]] && [[ -f "${TEST_REQUIREMENTS_FILE}" ]]; then
		if [[ -f "${TEST_REQUIREMENTS_FILE}" ]]; then
		log_info "--- installing test requirements from \\e[33;1m${TEST_REQUIREMENTS_FILE}\\e[0m"
		# shellcheck disable=SC2086
		pip install ${PIP_OPTS} -r "${TEST_REQUIREMENTS_FILE}"
		fi
		elif [[ -f "${SETUP_PY_DIR}/setup.py" ]]; then
		log_info "--- installing requirements from \\e[33;1m${SETUP_PY_DIR}/setup.py\\e[0m"
		# shellcheck disable=SC2086
		pip install ${PIP_OPTS} "${SETUP_PY_DIR}/"
		else
		log_info "--- no dependency management tool, nor requirements file nor setup.py file found: skip install dependencies"
		log_warn "--- requirements build system defined, but no ${REQUIREMENTS_FILE} file found"
		fi
		;;
		esac
		}

		function _run() {
		if [[ -f "pyproject.toml" ]] && [[ "${PYTHON_POETRY_DISABLED}" != "true" ]]; then
		if ! command -v poetry > /dev/null
		if [[ "${PYTHON_BUILD_SYSTEM}" == "poetry" ]]
		then
		pip install poetry
		fi
		# shellcheck disable=SC2086
		if ! command -v poetry > /dev/null; then pip install ${PIP_OPTS} poetry; fi
		poetry run "$@"
		else
		"$@"
		@@ -270,43 +324,60 @@ variables:
		}

		function _pip() {
		_run pip "$@"
		# shellcheck disable=SC2086
		_run pip ${PIP_OPTS} "$@"
		}

		function _package() {
		if [[ -f "pyproject.toml" ]] && [[ "${PYTHON_POETRY_DISABLED}" != "true" ]]; then
		pip install poetry
		case "$PYTHON_BUILD_SYSTEM" in
		poetry)
		# shellcheck disable=SC2086
		if ! command -v poetry > /dev/null; then pip install ${PIP_OPTS} poetry; fi
		poetry build
		else
		pip install setuptools
		python setup.py sdist bdist_wheel
		fi
		;;
		setuptools)
		# shellcheck disable=SC2086
		pip install ${PIP_OPTS} setuptools build
		python -m build
		;;
		reqfile)
		log_error "--- packaging is unsupported with requirements build system: read template doc"
		exit 1
		;;
		esac
		}

		function _publish() {
		if [[ -f "pyproject.toml" ]] && [[ "${PYTHON_POETRY_DISABLED}" != "true" ]]; then
		pip install poetry
		case "$PYTHON_BUILD_SYSTEM" in
		poetry)
		# shellcheck disable=SC2086
		if ! command -v poetry > /dev/null; then pip install ${PIP_OPTS} poetry; fi
		poetry config repositories.user_defined "$TWINE_REPOSITORY_URL"
		poetry publish --username "$TWINE_USERNAME" --password "$TWINE_PASSWORD" --repository user_defined
		else
		pip install twine
		pip list

		;;
		setuptools)
		# shellcheck disable=SC2086
		pip install ${PIP_OPTS} twine
		twine upload --verbose dist/*.tar.gz
		twine upload --verbose dist/*.whl
		fi
		;;
		reqfile)
		log_error "--- publish is unsupported with requirements build system: read template doc"
		exit 1
		;;
		esac
		}

		function _release() {
		if [[ -f "pyproject.toml" ]] && [[ "${PYTHON_POETRY_DISABLED}" != "true" ]]; then
		pip install poetry
		if [[ "${PYTHON_BUILD_SYSTEM}" == "poetry" ]]
		then
		# shellcheck disable=SC2086
		if ! command -v poetry > /dev/null; then pip install ${PIP_OPTS} poetry; fi
		poetry version "${RELEASE_VERSION_PART}"
		else
		pip install bumpversion
		release_args
		bumpversion "${bumpversion_args}"
		fi
		}
		function release_args() {
		# shellcheck disable=SC2086
		pip install ${PIP_OPTS} bumpversion

		if [[ -f ".bumpversion.cfg" ]]; then
		log_info "--- .bumpversion.cfg file found "
		export bumpversion_args="${RELEASE_VERSION_PART} --verbose"
		@@ -321,10 +392,10 @@ variables:
		fi
		fi
		log_info "--- Release args: ${bumpversion_args}"
		}



		bumpversion "${bumpversion_args}"
		fi
		}

		function get_latest_template_version() {
		tag_json=$(wget -T 5 -q -O - "$CI_API_V4_URL/projects/to-be-continuous%2F$1/repository/tags?per_page=1" \|\| echo "")
		@@ -368,6 +439,7 @@ variables:
		- *python-scripts
		- install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}"
		- cd ${PYTHON_PROJECT_DIR}
		- guess_build_system

		###############################################################################################
		# stages definition #
		@@ -387,7 +459,7 @@ py-lint:
		script:
		- mkdir -p reports
		- chmod o+rwx reports
		- install_requirements build
		- install_requirements
		- _pip install pylint_gitlab
		- \|
		if ! _run pylint --ignore=.cache --output-format=text ${PYLINT_ARGS} ${PYLINT_FILES:-$(find -type f -name "*.py")}
		@@ -423,7 +495,7 @@ py-compile:
		extends: .python-base
		stage: build
		script:
		- install_requirements build
		- install_requirements
		- _python -m compileall $PYTHON_COMPILE_ARGS
		rules:
		# exclude merge requests
		@@ -441,7 +513,7 @@ py-unittest:
		script:
		- mkdir -p reports
		- chmod o+rwx reports
		- install_requirements test
		- install_requirements
		# code coverage
		- _pip install coverage
		# JUnit XML report
		@@ -473,7 +545,7 @@ py-pytest:
		script:
		- mkdir -p reports
		- chmod o+rwx reports
		- install_requirements test
		- install_requirements
		- _pip install pytest pytest-cov coverage
		- _python -m pytest --junit-xml=reports/TEST-pytests.xml --cov --cov-report term --cov-report xml:reports/coverage.xml ${PYTEST_ARGS}
		coverage: /^TOTAL.+?(\d+\%)$/
		@@ -500,7 +572,7 @@ py-nosetests:
		script:
		- mkdir -p reports
		- chmod o+rwx reports
		- install_requirements test
		- install_requirements
		- _run nosetests --with-xunit --xunit-file=reports/TEST-nosetests.xml --with-coverage --cover-erase --cover-xml --cover-xml-file=reports/coverage.xml --cover-html --cover-html-dir=reports/coverage ${NOSETESTS_ARGS}
		coverage: /^TOTAL.+?(\d+\%)$/
		artifacts:
		@@ -529,6 +601,7 @@ py-bandit:
		script:
		- mkdir -p reports
		- chmod o+rwx reports
		- install_requirements
		- _pip install bandit
		- \|
		if ! _run bandit ${TRACE+--verbose} ${BANDIT_ARGS}
		@@ -565,8 +638,8 @@ py-safety:
		script:
		- mkdir -p reports
		- chmod o+rwx reports
		- install_requirements
		- _pip install safety
		- install_requirements build
		- \|
		if ! _pip freeze \| _run safety check --stdin ${SAFETY_ARGS}
		then