Commit 6c8b2d65 authored by Cédric OLIVIER's avatar Cédric OLIVIER
Browse files

Merge branch 'kustomize-with-var' into 'master' 

feat: enable kustomize support

Closes #1

See merge request to-be-continuous/kubernetes!38
parents ae55e2dd 3ad8b84a

README.md

+39 −2
Original line number Diff line number Diff line
@@ -75,6 +75,7 @@ The Kubernetes template uses some global configuration used throughout all jobs. 
| `K8S_CA_CERT`          | the default Kubernetes server certificate authority | **optional if using exploded kubeconfig parameters** |

| :lock: `K8S_TOKEN`     | default service account token         | **required if using exploded kubeconfig parameters** |

| `K8S_SCRIPTS_DIR`      | directory where k8s scripts (hook scripts) are located | `.` _(root project dir)_ |

| `K8S_KUSTOMIZE_ENABLED` | Set to `true` to force using [Kustomize](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/) | _none_ (disabled) |

| `DOCKER_CONTAINER_STABLE_IMAGE` | Docker image name to use for staging/prod | **has to be defined when not chaining execution from Docker template** |

| `DOCKER_CONTAINER_UNSTABLE_IMAGE` | Docker image name to use for review | **has to be defined when not chaining execution from Docker template** |
@@ -189,6 +190,8 @@ The GitLab CI template for Kubernetes supports two policies for deploying your c 


1. script-based deployment

2. template-based deployment

    * using raw Kubernetes manifests (with variables substitution),

    * using [Kustomization files](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/).



#### 1: script-based deployment
@@ -221,6 +224,24 @@ The template processes the following steps: 


All scripts and Kubernetes deployment files may use [dynamic variables](#dynamic-variables).



#### 3: Kustomize-based deployment



In this mode, you have to provide a [Kustomization file](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/)

in your project structure, and let the template [`kubectl apply`](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#apply) it.



The template processes the following steps:



1. _optionally_ executes the `k8s-pre-apply.sh` script in your project to perform specific environment pre-initialization (for e.g. create required services),

2. looks for your Kustomization file and [`kubectl apply`](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#apply) it,

    1. looks for an environment-specific [overlay](https://kubectl.docs.kubernetes.io/references/kustomize/glossary/#overlay) file `./$env/kustomization.yaml` (e.g. `./staging/kustomization.yaml ` for staging environment),

    2. fallbacks to default `kustomization.yaml`.

3. _optionally_ executes the `k8s-post-apply.sh` script in your project to perform specific environment post-initialization stuff,



:warning: `k8s-pre-apply.sh` or `k8s-post-apply.sh` needs to be executable, you can add flag execution with:  `git update-index --chmod=+x k8s-pre-apply.sh`



All scripts and Kustomization files may use [dynamic variables](#dynamic-variables). 

Variables substitution will be performed by the deprecated feature from Kustomize based on `configMapGenerator`, using a non-valuated variable from a config map.



#### readiness



After deployment (either script-based or template-based), the GitLab CI template _optionally_ executes the `k8s-readiness-check.sh` hook script to wait & check for the application to be ready (if not found, the template assumes the application was successfully started).
@@ -235,6 +256,8 @@ The GitLab CI template for Kubernetes supports two policies for destroying an en 


1. script-based cleanup

2. template-based cleanup

    * using raw Kubernetes manifests (with variables substitution),

    * using [Kustomization files](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/).



#### 1: script-based cleanup
@@ -270,6 +293,20 @@ The template processes the following steps: 


All scripts and Kubernetes deployment files may use [dynamic variables](#dynamic-variables).



#### 3: Kustomize-based cleanup



In this mode, you mainly let Kubernetes delete all objects from your [Kustomization file(s)](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/).



The template processes the following steps:



1. _optionally_ executes the `k8s-pre-cleanup.sh` script in your project to perform specific environment pre-cleanup stuff,

2. looks for your Kustomization file and [`kubectl delete`](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#apply) it,

    1. looks for an environment-specific [overlay](https://kubectl.docs.kubernetes.io/references/kustomize/glossary/#overlay) file `./$env/kustomization.yaml` (e.g. `./staging/kustomization.yaml ` for staging environment),

    2. fallbacks to default `kustomization.yaml`.

3. _optionally_ executes the `k8s-post-cleanup.sh` script in your project to perform specific environment post-cleanup (for e.g. delete bound services).



:warning: `k8s-pre-cleanup.sh` or `k8s-post-cleanup.sh` needs to be executable, you can add flag execution with:  `git update-index --chmod=+x k8s-pre-cleanup.sh`



#### Cleanup job limitations



When using this template, you have to be aware of one limitation (bug) with the cleanup job.
@@ -349,7 +386,7 @@ Here are its parameters: 


| Name                   | description                                                          | default value     |

| ---------------------- | -------------------------------------------------------------------- | ----------------- |

| `K8S_KUBE_SCORE_IMAGE` | Docker image to run [kube-score](https://github.com/zegl/kube-score) | `zegl/kube-score:latest-helm` **it is recommended to set a tool version compatible with your Kubernetes cluster** |

| `K8S_KUBE_SCORE_IMAGE` | Docker image to run [kube-score](https://github.com/zegl/kube-score) | `zegl/kube-score:latest-kustomize` **it is recommended to set a tool version compatible with your Kubernetes cluster** |

| `K8S_SCORE_DISABLED`   | Set to `true` to disable the `kube-score` analysis                             | _none_ (enabled) |

| `K8S_SCORE_EXTRA_OPTS` | [Additional options](https://github.com/zegl/kube-score#configuration) to `kube-score` command line | _none_ |

kicker.json

+7 −1
Original line number Diff line number Diff line
@@ -40,6 +40,12 @@ 
      "description": "directory where Kubernetes scripts (templates, hook scripts) are located",

      "default": ".",

      "advanced": true

    },

    {

      "name": "K8S_KUSTOMIZE_ENABLED",

      "description": "Set to `true` to enable [Kustomize](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/)",

      "type": "boolean",

      "advanced": true

    }

  ],

  "features": [
@@ -52,7 +58,7 @@ 
        {

          "name": "  K8S_KUBE_SCORE_IMAGE",

          "description": "Docker image to run [kube-score](https://github.com/zegl/kube-score)",

          "default": "zegl/kube-score:latest-helm"

          "default": "zegl/kube-score:latest-kustomize"

        },

        {

          "name": "K8S_SCORE_EXTRA_OPTS",

templates/gitlab-ci-k8s.yml

+172 −79
Original line number Diff line number Diff line
@@ -46,10 +46,11 @@ variables: 


  # Docker Image with Kubernetes CLI tool (can be overridden)

  K8S_KUBECTL_IMAGE: "bitnami/kubectl:latest"

  K8S_KUBE_SCORE_IMAGE: "zegl/kube-score:latest-helm"

  K8S_KUBE_SCORE_IMAGE: "zegl/kube-score:latest-kustomize"

  K8S_BASE_APP_NAME: "$CI_PROJECT_NAME"

  K8S_SCRIPTS_DIR: "."

  K8S_REVIEW_ENVIRONMENT_SCHEME: "https"



  # default production ref name (pattern)

  PROD_REF: '/^(master|main)$/'

  # default integration ref name (pattern)
@@ -325,6 +326,41 @@ stages: 
    kubectl ${TRACE+-v=5} version

  }



  function do_kubectl() {

    action=$1



    if [[ "${K8S_KUSTOMIZE_ENABLED}" == "true" ]]

    then

      kustofile=$(ls -1 "$K8S_SCRIPTS_DIR/${env}/kustomization.yaml" 2>/dev/null || ls -1 "$K8S_SCRIPTS_DIR/kustomization.yaml" 2>/dev/null || ls -1 "$K8S_SCRIPTS_DIR/${env}/kustomization.yml" 2>/dev/null || ls -1 "$K8S_SCRIPTS_DIR/kustomization.yml" 2>/dev/null || echo "")

      if [[ -z "$kustofile" ]]

      then

        log_error "--- kustomize enabled but nor ${env}/kustomization.yaml neither kustomization.yaml was found"

        exit 1

      fi

      deploymentdir=$(dirname "$kustofile")



      # apply/delete deployment descriptor

      log_info "--- \\e[32mkubectl apply\\e[0m"

      kubectl ${TRACE+-v=5} "$action" -k "$deploymentdir"

    else



      # find deployment file

      deploymentfile=$(ls -1 "$K8S_SCRIPTS_DIR/deployment-${env}.yml" 2>/dev/null || ls -1 "$K8S_SCRIPTS_DIR/deployment.yml" 2>/dev/null || echo "")

      if [[ -z "$deploymentfile" ]]

      then

        log_error "--- deployment file not found"

        exit 1

      fi



      # replace variables (alternative for envsubst which is not present in image)

      awkenvsubst < "$deploymentfile" > generated-deployment.yml



      log_info "--- \\e[32mkubectl apply\\e[0m"

      kubectl ${TRACE+-v=5} "$action" -f ./generated-deployment.yml



    fi

  }



  function deploy() {

    export env=$1

    # for backward compatibility
@@ -366,20 +402,7 @@ stages: 
        log_info "--- \\e[32mpre-apply hook\\e[0m (\\e[33;1m${prescript}\\e[0m) not found: skip"

      fi



      # find deployment file

      deploymentfile=$(ls -1 "$K8S_SCRIPTS_DIR/deployment-${env}.yml" 2>/dev/null || ls -1 "$K8S_SCRIPTS_DIR/deployment.yml" 2>/dev/null || echo "")

      if [[ -z "$deploymentfile" ]]

      then

        log_error "--- deployment file not found"

        exit 1

      fi



      # replace variables (alternative for envsubst which is not present in image)

      awkenvsubst < "$deploymentfile" > generated-deployment.yml



      # apply/rollout deployment descriptor

      log_info "--- \\e[32mkubectl apply\\e[0m"

      kubectl ${TRACE+-v=5} apply -f ./generated-deployment.yml

      do_kubectl apply



      # maybe execute post apply script

      postscript="$K8S_SCRIPTS_DIR/k8s-post-apply.sh"
@@ -488,23 +511,10 @@ stages: 
        log_info "--- \\e[32mpre-cleanup hook\\e[0m (\\e[33;1m${prescript}\\e[0m) not found: skip"

      fi



      # find deployment file

      deploymentfile=$(ls -1 "$K8S_SCRIPTS_DIR/deployment-${env}.yml" 2>/dev/null || ls -1 "$K8S_SCRIPTS_DIR/deployment.yml" 2>/dev/null || echo "")

      if [[ -z "$deploymentfile" ]]

      then

        log_error "--- deployment file not found"

        exit 1

      fi



      # has to be valuated for envsubst

      export hostname=hostname



      # replace variables (alternative for envsubst which is not present in image)

      awkenvsubst < "$deploymentfile" > generated-deployment.yml



      # delete all objects from deployment file

      log_info "--- \\e[32mkubectl delete\\e[0m"

      kubectl ${TRACE+-v=5} delete -f ./generated-deployment.yml

      do_kubectl delete



      # maybe execute post cleanup script

      postscript="$K8S_SCRIPTS_DIR/k8s-post-cleanup.sh"
@@ -518,30 +528,31 @@ stages: 
  }



  function score() {

    deploymentfiles=""

    if [[ -f "$K8S_SCRIPTS_DIR/deployment-review.yml" ]]; then

      awkenvsubst < "$K8S_SCRIPTS_DIR/deployment-review.yml" > "$K8S_SCRIPTS_DIR/generated-deployment-review.yml"

      deploymentfiles="$K8S_SCRIPTS_DIR/generated-deployment-review.yml"

    fi

    if [[ -f "$K8S_SCRIPTS_DIR/deployment-integration.yml" ]]; then

      awkenvsubst < "$K8S_SCRIPTS_DIR/deployment-integration.yml" > "$K8S_SCRIPTS_DIR/generated-deployment-integration.yml"

      deploymentfiles="$deploymentfiles $K8S_SCRIPTS_DIR/generated-deployment-integration.yml"

    fi

    if [[ -f "$K8S_SCRIPTS_DIR/deployment-staging.yml" ]]; then

      awkenvsubst < "$K8S_SCRIPTS_DIR/deployment-staging.yml" > "$K8S_SCRIPTS_DIR/generated-deployment-staging.yml"

      deploymentfiles="$deploymentfiles $K8S_SCRIPTS_DIR/generated-deployment-staging.yml"

    fi

    if [[ -f "$K8S_SCRIPTS_DIR/deployment-production.yml" ]]; then

      awkenvsubst < "$K8S_SCRIPTS_DIR/deployment-production.yml" > "$K8S_SCRIPTS_DIR/generated-deployment-production.yml"

      deploymentfiles="$deploymentfiles $K8S_SCRIPTS_DIR/generated-deployment-production.yml"

    export env=$1

    if [[ "$K8S_KUSTOMIZE_ENABLED" == "true" ]]

    then

      kustofile=$(ls -1 "$K8S_SCRIPTS_DIR/${env}/kustomization.yaml" 2>/dev/null || ls -1 "$K8S_SCRIPTS_DIR/kustomization.yaml" 2>/dev/null || ls -1 "$K8S_SCRIPTS_DIR/${env}/kustomization.yml" 2>/dev/null || ls -1 "$K8S_SCRIPTS_DIR/kustomization.yml" 2>/dev/null || echo "")

      if [[ -z "$kustofile" ]]

      then

        log_error "--- kustomize enabled but nor ${env}/kustomization.yaml neither kustomization.yaml was found"

        exit 1

      fi

    if [[ -f "$K8S_SCRIPTS_DIR/deployment.yml" ]]; then

      awkenvsubst < "$K8S_SCRIPTS_DIR/deployment.yml" > "$K8S_SCRIPTS_DIR/generated-deployment.yml"

      deploymentfiles="$deploymentfiles $K8S_SCRIPTS_DIR/generated-deployment.yml"

      deploymentdir=$(dirname "$kustofile")



      # shellcheck disable=SC2086

      kustomize build "${deploymentdir}" | /usr/bin/kube-score score $K8S_SCORE_EXTRA_OPTS -

    else

      # find deployment file

      deploymentfile=$(ls -1 "$K8S_SCRIPTS_DIR/deployment-${env}.yml" 2>/dev/null || ls -1 "$K8S_SCRIPTS_DIR/deployment.yml" 2>/dev/null || echo "")

      if [[ -z "$deploymentfile" ]]

      then

        log_error "--- deployment file not found"

        exit 1

      fi



      # shellcheck disable=SC2086

    /usr/bin/kube-score score $K8S_SCORE_EXTRA_OPTS $deploymentfiles

      /usr/bin/kube-score score $K8S_SCORE_EXTRA_OPTS $deploymentfile

    fi

  }



  # export tool functions (might be used in after_script)
@@ -563,22 +574,15 @@ stages: 
    - *k8s-scripts

    - install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}"



k8s-score:

.k8s-score:

  extends: .k8s-base

  stage: test

  image:

    name: $K8S_KUBE_SCORE_IMAGE

    entrypoint: [""]

  before_script:

    - *k8s-scripts

    - install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}"

  script:

    - score

  rules:

    # exclude when $K8S_SCORE_DISABLED is set

    - if: '$K8S_SCORE_DISABLED == "true"'

      when: never

    - !reference [.test-policy, rules]

    - score "$ENV_TYPE"





# Deploy job prototype

# Can be extended to define a concrete environment
@@ -640,6 +644,28 @@ k8s-score: 
  environment:

    action: stop



k8s-score-review:

  extends: .k8s-score

  variables:

    ENV_TYPE: review

  environment:

    name: review/$CI_COMMIT_REF_NAME

    url: "${K8S_REVIEW_ENVIRONMENT_SCHEME}://${CI_PROJECT_NAME}-${CI_ENVIRONMENT_SLUG}.${K8S_REVIEW_ENVIRONMENT_DOMAIN}"

  rules:

    # exclude tags

    - if: $CI_COMMIT_TAG

      when: never

    # exclude when $K8S_SCORE_DISABLED is set

    - if: '$K8S_SCORE_DISABLED == "true"'

      when: never

    # exclude if $K8S_REVIEW_SPACE unset

    - if: '$K8S_REVIEW_SPACE == null || $K8S_REVIEW_SPACE == ""'

      when: never

    # only on non-production, non-integration branches

    - if: '$CI_COMMIT_REF_NAME =~ $PROD_REF || $CI_COMMIT_REF_NAME =~ $INTEG_REF'

      when: never

    - !reference [.test-policy, rules]



k8s-review:

  extends: .k8s-deploy

  variables:
@@ -686,6 +712,28 @@ k8s-cleanup-review: 
      when: manual

      allow_failure: true



k8s-score-integration:

  extends: .k8s-score

  variables:

    ENV_TYPE: integration

  environment:

    name: integration

    url: "${K8S_INTEG_ENVIRONMENT_URL}"

  rules:

    # exclude tags

    - if: $CI_COMMIT_TAG

      when: never

    # exclude when $K8S_SCORE_DISABLED is set

    - if: '$K8S_SCORE_DISABLED == "true"'

      when: never

    # exclude if $K8S_INTEG_SPACE unset

    - if: '$K8S_INTEG_SPACE == null || $K8S_INTEG_SPACE == ""'

      when: never

    # only on integration branch

    - if: '$CI_COMMIT_REF_NAME !~ $INTEG_REF'

      when: never

    - !reference [.test-policy, rules]



k8s-integration:

  extends: .k8s-deploy

  variables:
@@ -710,6 +758,29 @@ k8s-integration: 
# If you prefer to automatically deploy to staging and

# only manually promote to production, enable this job by setting

# $K8S_STAGING_SPACE.



k8s-score-staging:

  extends: .k8s-score

  variables:

    ENV_TYPE: staging

  environment:

    name: staging

    url: "${K8S_STAGING_ENVIRONMENT_URL}"

  rules:

    # exclude tags

    - if: $CI_COMMIT_TAG

      when: never

    # exclude when $K8S_SCORE_DISABLED is set

    - if: '$K8S_SCORE_DISABLED == "true"'

      when: never

    # exclude if $K8S_STAGING_SPACE unset

    - if: '$K8S_STAGING_SPACE == null || $K8S_STAGING_SPACE == ""'

      when: never

    # only on prod branch

    - if: '$CI_COMMIT_REF_NAME !~ $PROD_REF'

      when: never

    - !reference [.test-policy, rules]



k8s-staging:

  extends: .k8s-deploy

  variables:
@@ -728,6 +799,28 @@ k8s-staging: 
    # only on production branch(es), with $K8S_STAGING_SPACE set

    - if: '$K8S_STAGING_SPACE && $CI_COMMIT_REF_NAME =~ $PROD_REF'



k8s-score-production:

  extends: .k8s-score

  variables:

    ENV_TYPE: production

  environment:

    name: production

    url: "${K8S_PROD_ENVIRONMENT_URL}"

  rules:

    # exclude tags

    - if: $CI_COMMIT_TAG

      when: never

    # exclude when $K8S_SCORE_DISABLED is set

    - if: '$K8S_SCORE_DISABLED == "true"'

      when: never

    # exclude if $K8S_PROD_SPACE unset

    - if: '$K8S_PROD_SPACE == null || $K8S_PROD_SPACE == ""'

      when: never

    # only on prod branch

    - if: '$CI_COMMIT_REF_NAME !~ $PROD_REF'

      when: never

    - !reference [.test-policy, rules]



# deploy to production if on branch master and variable K8S_PROD_SPACE defined and AUTODEPLOY_TO_PROD is set

k8s-production:

  extends: .k8s-deploy