Commit c3cf8d04 authored by Gaëtan Montury's avatar Gaëtan Montury
Browse files

Merge branch 'feat/generic-pep517' into 'master' 

feat: allow in best-effort generic pip install following pep 517

Closes #118

See merge request to-be-continuous/python!175
parents 5ff8387a de447e99

README.md

+12 −3
Original line number Diff line number Diff line
@@ -50,7 +50,8 @@ The Python template uses some global configuration used throughout all jobs. 
| `PIP_INDEX_URL`      | Python repository url                                                                 | _none_             |

| `PIP_EXTRA_INDEX_URL` | Extra Python repository url                                                           | _none_             |

| `pip-opts` / `PIP_OPTS` | pip [extra options](https://pip.pypa.io/en/stable/cli/pip/#general-options)     | _none_             |

| `extra-deps` / `PYTHON_EXTRA_DEPS` | Python extra sets of dependencies to install<br/>For [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies) or [Poetry](https://python-poetry.org/docs/pyproject/#extras) or [uv](https://docs.astral.sh/uv/) only | _none_ |

| `pip-install-deps-opts` / `PIP_INSTALL_DEPS_OPTS` | pip install dependencies extra [options](https://pip.pypa.io/en/stable/cli/pip/#general-options) used with `pip` build-system only | _none_             |

| `extra-deps` / `PYTHON_EXTRA_DEPS` | Python extra sets of dependencies to install for:<br/>- [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies)<br/>- [Poetry](https://python-poetry.org/docs/pyproject/#extras)<br/>- [uv](https://docs.astral.sh/uv/concepts/projects/dependencies/#dependency-groups)<br/>- pip with pyproject.toml ([PEP517](https://peps.python.org/pep-0517/) [PEP735](https://peps.python.org/pep-0735/)) require pip >= 25.1 | _none_ |

| `reqs-file` / `PYTHON_REQS_FILE` | Main requirements file _(relative to `$PYTHON_PROJECT_DIR`)_<br/>For [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only | `requirements.txt` |

| `extra-reqs-files` / `PYTHON_EXTRA_REQS_FILES` | Extra dev requirements file(s) to install _(relative to `$PYTHON_PROJECT_DIR`)_ | `requirements-dev.txt` |
@@ -73,9 +74,17 @@ and/or `setup.py` and/or `requirements.txt`), but the build system might also be 
| `hatch`          | [![Hatch project](https://img.shields.io/badge/%F0%9F%A5%9A-Hatch-4051b5.svg)](https://hatch.pypa.io/latest/) ![Hatch](https://img.shields.io/badge/dependencies_%7C_build_%7C_test_%7C_packaging-blue) |

| `pdm`            | [![pdm](https://img.shields.io/endpoint?url=https%3A%2F%2Fcdn.jsdelivr.net%2Fgh%2Fpdm-project%2F.github%2Fbadge.json)](https://pdm-project.org) ![PDM](https://img.shields.io/badge/dependencies_%7C_build_%7C_test_%7C_packaging-blue)

| `pipenv`         | [![Pipenv](https://img.shields.io/badge/Pipenv-grey)](https://pipenv.pypa.io/) ![Pipenv](https://img.shields.io/badge/dependencies-blue)                                                  |

| `reqfile`        | [![Requirements Files](https://img.shields.io/badge/Requirements_Files-grey)](https://pip.pypa.io/en/stable/user_guide/#requirements-files) ![Pipenv](https://img.shields.io/badge/dependencies-blue) |

| `reqfile`        | [![Requirements Files](https://img.shields.io/badge/Requirements_Files-grey)](https://pip.pypa.io/en/stable/user_guide/#requirements-files) ![Pip](https://img.shields.io/badge/dependencies-blue) |

| `pip`            | [![pip with pyproject.toml File](https://img.shields.io/badge/pyproject.toml_File_with_pip-grey)](https://peps.python.org/pep-0517/) ![Pip](https://img.shields.io/badge/dependencies_%7C_test-blue) look note below |



:warning: You can explicitly set the build tool version by setting `$PYTHON_BUILD_SYSTEM` variable including a [version identification](https://peps.python.org/pep-0440/). For example `PYTHON_BUILD_SYSTEM="poetry==1.1.15"`

:warning: You can explicitly set the build tool version by setting `$PYTHON_BUILD_SYSTEM` variable including a [version identification](https://peps.python.org/pep-0440/). For example `PYTHON_BUILD_SYSTEM="poetry==1.1.15"`. Same logic can be used with `pip` build-system like `PYTHON_BUILD_SYSTEM="pip==25.1"` to force upgrade `pip` used to version 25.1.  



### pip with pyproject.toml File



This template can handle a generic backend (following PEP 517, PEP 518, PEP 735, ...) aka pyproject.toml with [build-system] ...  by using `pip`.  

The installation takes place inside a virtual environment managed by `python -m venv`.  



:exclamation: But the support is on a best-effort basis. In particular, release and packaging may not work as expected, or may come with significant restrictions.



### Hatch

kicker.json

+7 −2
Original line number Diff line number Diff line
@@ -21,7 +21,7 @@ 
      "name": "PYTHON_BUILD_SYSTEM",

      "description": "Python build-system to use to install dependencies, build and package the project",

      "type": "enum",

      "values": ["auto", "setuptools", "poetry", "pipenv", "reqfile", "uv", "hatch", "pdm"],

      "values": ["auto", "setuptools", "poetry", "pipenv", "reqfile", "uv", "hatch", "pdm", "pip"],

      "default": "auto",

      "advanced": true

    },
@@ -48,9 +48,14 @@ 
      "description": "pip extra [options](https://pip.pypa.io/en/stable/cli/pip/#general-options)",

      "advanced": true

    },

    {

      "name": "PIP_INSTALL_DEPS_OPTS",

      "description": "pip install dependencies extra [options](https://pip.pypa.io/en/stable/cli/pip/#general-options) used with `pip` build-system only",

      "advanced": true

    },

    {

      "name": "PYTHON_EXTRA_DEPS",

      "description": "Extra sets of dependencies to install\n\nFor [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies) or [Poetry](https://python-poetry.org/docs/pyproject/#extras) only"

      "description": "Python extra sets of dependencies to install for:\n- [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies)\n- [Poetry](https://python-poetry.org/docs/pyproject/#extras)\n- [uv](https://docs.astral.sh/uv/concepts/projects/dependencies/#dependency-groups)\n- pip with pyproject.toml ([PEP517](https://peps.python.org/pep-0517/) [PEP735](https://peps.python.org/pep-0735/)) require pip >= 25.1"

    }

  ],

  "features": [

templates/gitlab-ci-python.yml

+58 −15
Original line number Diff line number Diff line
@@ -32,6 +32,7 @@ spec: 
      - uv

      - hatch

      - pdm

      - pip

      default: auto

    reqs-file:

      description: |-
@@ -51,11 +52,16 @@ spec: 
    pip-opts:

      description: pip extra [options](https://pip.pypa.io/en/stable/cli/pip/#general-options)

      default: ''

    pip-install-deps-opts:

      description:  pip install dependencies extra [options](https://pip.pypa.io/en/stable/cli/pip/#general-options) used with `pip` build-system only

      default: ''

    extra-deps:

      description: |-

        Extra sets of dependencies to install



        For [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies) or [Poetry](https://python-poetry.org/docs/pyproject/#extras) only

        Python extra sets of dependencies to install for:

        - [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies)

        - [Poetry](https://python-poetry.org/docs/pyproject/#extras)

        - [uv](https://docs.astral.sh/uv/concepts/projects/dependencies/#dependency-groups)

        - pip with pyproject.toml ([PEP517](https://peps.python.org/pep-0517/) [PEP735](https://peps.python.org/pep-0735/)) require pip >= 25.1

      default: ''

    package-enabled:

      description: Enable package
@@ -331,6 +337,7 @@ variables: 


  PYTHON_BUILD_SYSTEM: $[[ inputs.build-system ]]

  PIP_OPTS: $[[ inputs.pip-opts ]]

  PIP_INSTALL_DEPS_OPTS: $[[ inputs.pip-install-deps-opts ]]

  PYTHON_EXTRA_DEPS: $[[ inputs.extra-deps ]]

  PYTHON_PACKAGE_ENABLED: $[[ inputs.package-enabled ]]

  PYLINT_ENABLED: $[[ inputs.pylint-enabled ]]
@@ -665,10 +672,11 @@ variables: 
      _p3dir=$(dirname "$_p3")

      ln -s "$_p3" "$_p3dir/python"

      if [ -n "$TRACE" ]; then

        log_info "python3 symlinked to $_p3dir/python"

        log_info "--- python3 symlinked to $_p3dir/python"

      fi

    fi

  }



  function guess_build_system() {

    _start_time=$(get_current_ts_ms)
@@ -680,6 +688,11 @@ variables: 
      log_info "--- Build system explicitly declared: ${PYTHON_BUILD_SYSTEM} cmd=\\e[33m$PYTHON_BUILD_SYSTEM_CMD\\e[0m"

      return

      ;;

    pip*)

      export PYTHON_BUILD_SYSTEM_CMD="pip"

      log_info "--- Build system explicitly declared: ${PYTHON_BUILD_SYSTEM} cmd=\\e[33m$PYTHON_BUILD_SYSTEM_CMD\\e[0m"

      return

      ;;

    reqfile)

      log_info "--- Build system explicitly declared: requirements file"

      return
@@ -741,8 +754,10 @@ variables: 
        return

        ;;

      *)

        log_error "--- Build system auto-detected: PEP 517 with unsupported backend \\e[33;1m${build_backend}\\e[0m: please read template doc"

        exit 1

        log_warn "--- Build system auto-detected: falling back to generic PEP support (backend \\e[33;1m${build_backend}\\e[0m). Not all jobs are guaranteed to work (notably for release and packaging)."

        export PYTHON_BUILD_SYSTEM="pip"

        export PYTHON_BUILD_SYSTEM_CMD="pip"

        return

        ;;

      esac

    fi
@@ -764,6 +779,19 @@ variables: 
    }



  function maybe_install_build_system() {

    case "${PYTHON_BUILD_SYSTEM}" in

    pip*)

      # extract version from string like "pip==25.1"

      pip_version=${PYTHON_BUILD_SYSTEM#pip}

      if [[ -n "${pip_version}" ]]; then

        log_info "--- Upgrade pip to \\e[33;1m${pip_version}\\e[0m"

        # shellcheck disable=SC2086

        pip install ${PIP_OPTS} --upgrade "pip${pip_version}"

      fi

      return

    ;;

    esac



    if ! command -v "$PYTHON_BUILD_SYSTEM_CMD" > /dev/null

    then

      # shellcheck disable=SC2086
@@ -783,7 +811,7 @@ variables: 
      maybe_install_build_system

      poetry install ${PYTHON_EXTRA_DEPS:+--extras "$PYTHON_EXTRA_DEPS"}

      ;;

    setuptools*)

    setuptools)

      maybe_install_build_system

      # shellcheck disable=SC2086

      pip install ${PIP_OPTS} ".${PYTHON_EXTRA_DEPS:+[$PYTHON_EXTRA_DEPS]}"
@@ -797,6 +825,21 @@ variables: 
        pipenv sync --dev --system

      fi

      ;;

    pip*)

      log_info "--- Creating virtual environment..."

      python -m venv .venv

      echo "*" >> .venv/.gitignore

      # shellcheck disable=SC1091

      . .venv/bin/activate



      maybe_install_build_system

      extra_groups=""

      for group in ${PYTHON_EXTRA_DEPS}; do

        extra_groups="${extra_groups} --group $group"

      done

      # shellcheck disable=SC2086

      pip install . ${PIP_OPTS} ${extra_groups} ${PIP_INSTALL_DEPS_OPTS}

      ;;

    reqfile)

      if [[ -f "${PYTHON_REQS_FILE}" ]]; then

        log_info "--- installing main requirements from \\e[33;1m${PYTHON_REQS_FILE}\\e[0m"
@@ -840,6 +883,11 @@ variables: 
        maybe_install_build_system

        $PYTHON_BUILD_SYSTEM_CMD run "$@"

        ;;

      pip)

        # shellcheck disable=SC1091

        . .venv/bin/activate

        "$@"

        ;;

      *)

        "$@"

        ;;
@@ -872,7 +920,7 @@ variables: 
    rm -fr "$PYTHON_PROJECT_DIR/reports"



    case "$PYTHON_BUILD_SYSTEM" in

      poetry*|uv*)

      poetry*|uv*|pdm*)

        maybe_install_build_system

        log_info "--- build packages ($PYTHON_BUILD_SYSTEM_CMD)..."

        $PYTHON_BUILD_SYSTEM_CMD build ${TRACE+--verbose}
@@ -882,11 +930,6 @@ variables: 
        log_info "--- build packages (hatch)..."

        $PYTHON_BUILD_SYSTEM_CMD build

        ;;

      pdm*)

        maybe_install_build_system

        log_info "--- build packages (pdm)..."

        pdm build ${TRACE+--verbose}

        ;;

      *)

        log_info "--- build packages ..."

        # shellcheck disable=SC2086
@@ -1046,7 +1089,7 @@ variables: 


      py_git_tag_commit_release_change "$py_cur_version" "$py_next_version"

    else

      # Setuptools / bump-my-version

      # Setuptools / bump-my-version / pip with pyproject

      # shellcheck disable=SC2086

      pip install ${PIP_OPTS} bump-my-version

      if [[ "$py_next_version" ]]
@@ -1181,7 +1224,7 @@ stages: 
    HATCH_CACHE_DIR: "$CI_PROJECT_DIR/.cache/hatch/.cache/"

    HATCH_DATA_DIR: "$CI_PROJECT_DIR/.cache/hatch/.local"

    PDM_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pdm"

    POETRY_VIRTUALENVS_IN_PROJECT: "false"

    POETRY_VIRTUALENVS_IN_PROJECT: "true"

  cache:

    key: "$CI_COMMIT_REF_SLUG-python"

    when: always