Commit 9503d862 authored by Pierre Smeyers's avatar Pierre Smeyers
Browse files

Merge branch 'feature/implicit-backend-and-vars' into 'master' 

Implicit backend and vars

Closes #34 and #33

See merge request to-be-continuous/terraform!39
parents 72a28d1a 00ebb978

README.md

+24 −5
Original line number Diff line number Diff line
@@ -97,9 +97,15 @@ This template enables [Terraform integration in Merge Requests](https://docs.git 


As a result if you enabled your `production` environment, every merge request will compute and display infrastructure changes compared to `master` branch.



### GitLab managed Terraform State

### Terraform Backend management



By default, this template enables [GitLab managed Terraform State](https://docs.gitlab.com/ee/user/infrastructure/terraform_state.html) (set `$TF_GITLAB_BACKEND_DISABLED` to disable).

By default, this template enables [GitLab managed Terraform State](https://docs.gitlab.com/ee/user/infrastructure/terraform_state.html). 

As mentionned in GitLab's documentation, that requires that your Terraform scripts declare the 

Terraform [HTTP backend](https://www.terraform.io/docs/language/settings/backends/http.html), the templates

does the rest to configure it automatically.



This default behavior can be disabled by setting `$TF_GITLAB_BACKEND_DISABLED` to `false`.

In that case, you'll have to declare and configure your backend and tfstate by yourself (see [Implicit Backend configuration support](#implicit-backend-configuration-support) below).



#### _Error acquiring the state lock_ workaround
@@ -151,6 +157,16 @@ terraform init \ 
    -backend-config=retry_wait_min="${TF_HTTP_RETRY_WAIT_MIN}"

```



#### Implicit Backend configuration support



If you disabled the GitLab-managed Terraform state (by setting `$TF_GITLAB_BACKEND_DISABLED` to `false`),

the template supports an implicit [backend configuration](https://www.terraform.io/language/settings/backends/configuration#file) mechanism:



1. Looks for a `$env.tfbackend` file (ex: `staging.tfbackend` for staging environment),

2. Fallbacks to `default.tfbackend` file.



If one of those files are found, it is automatically used by the template in the `terraform init` command (using the `-backend-config` CLI option).



### Environments configuration



As seen above, the Terraform template may support up to 4 environments (`review`, `integration`, `staging` and `production`).
@@ -277,10 +293,13 @@ You have to be aware that your Terraform code has to be able to cope with variou 


In order to be able to implement some **genericity** in your code, you should use [Terraform variables](https://www.terraform.io/docs/language/values/variables.html) (in your Terraform files), and environment variables (in your hook scripts):



1. any [predefined GitLab CI variable](https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables) may be freedly used in your hook scripts or extra options variables (ex: `TF_EXTRA_OPTS: "-var project_name=$CI_PROJECT_NAME"`)

2. you may also use [custom GitLab variables](https://docs.gitlab.com/ee/ci/variables/#custom-cicd-variables) to pass values to your hook script or even directly as Terraform variables [using the right syntax](https://www.terraform.io/docs/cli/config/environment-variables.html#tf_var_name)

1. Use [`tfvars` files](https://www.terraform.io/language/values/variables#variable-definitions-tfvars-files) for non-secret configuration:

    * default `terraform.tfvars[.json]` and `*.auto.tfvars[.json]` files are obviously supported by Terraform,

    * the template also auto-detects any file named `$env.env.tfvars[.json]` (ex: `staging.env.tfvars` for staging environment) and uses it with all related `terraform` commands.

2. any [predefined GitLab CI variable](https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables) may be freedly used in your hook scripts or extra options variables (ex: `TF_EXTRA_OPTS: "-var project_name=$CI_PROJECT_NAME"`)

3. you may also use [custom GitLab variables](https://docs.gitlab.com/ee/ci/variables/#custom-cicd-variables) to pass values to your hook script or even directly as Terraform variables [using the right syntax](https://www.terraform.io/docs/cli/config/environment-variables.html#tf_var_name)

    (ex: env variable `$TF_VAR_ssh_private_key_file` will be visible as `ssh_private_key_file` Terraform variable in your code)

3. **dynamic variables** provided by the template:

4. **dynamic variables** provided by the template:

    * `environment_type`: the environment type (`review`, `integration`, `staging` or `production`)

    * `environment_name` (set as `$CI_ENVIRONMENT_NAME`): the full environment name (ex: `review/fix-prometheus-configuration`, `integration`, `staging` or `production`)

    * `environment_slug` (set as `$CI_ENVIRONMENT_SLUG`): the _slugified_ environment name (ex: `review-fix-promet-r13zmu`, `integration`, `staging` or `production`)

templates/gitlab-ci-terraform.yml

+68 −30
Original line number Diff line number Diff line
@@ -285,6 +285,7 @@ stages: 
    extra_opts=$2



    log_info "--- \\e[32minit\\e[0m"

    log_info "--- Positioned environment (can also be used as TF vars):"

    # shellcheck disable=SC2154

    log_info "--- environment_type: \\e[33;1m${environment_type}\\e[0m"

    # shellcheck disable=SC2154
@@ -312,6 +313,16 @@ stages: 
    # dump terraform version

    terraform --version



    # maybe execute pre init script

    prescript="$TF_SCRIPTS_DIR/tf-pre-init.sh"

    if [[ -f "$prescript" ]]; then

      log_info "--- \\e[32mpre-init\\e[0m hook (\\e[33;1m${prescript}\\e[0m) found: execute"

      chmod +x "$prescript"

      "$prescript"

    else

      log_info "--- \\e[32mpre-init\\e[0m hook (\\e[33;1m${prescript}\\e[0m) not found: skip"

    fi



    if [[ "$TF_GITLAB_BACKEND_DISABLED" != "true" ]]

    then

      # impl inspired by GitLab Terraform image script
@@ -350,16 +361,15 @@ stages: 
      tf_backend_opts="$tf_backend_opts -backend-config=lock_method=${TF_HTTP_LOCK_METHOD}"

      tf_backend_opts="$tf_backend_opts -backend-config=unlock_method=${TF_HTTP_UNLOCK_METHOD}"

      tf_backend_opts="$tf_backend_opts -backend-config=retry_wait_min=${TF_HTTP_RETRY_WAIT_MIN}"

    fi



    # maybe execute pre init script

    prescript="$TF_SCRIPTS_DIR/tf-pre-init.sh"

    if [[ -f "$prescript" ]]; then

      log_info "--- \\e[32mpre-init\\e[0m hook (\\e[33;1m${prescript}\\e[0m) found: execute"

      chmod +x "$prescript"

      "$prescript"

    else

      log_info "--- \\e[32mpre-init\\e[0m hook (\\e[33;1m${prescript}\\e[0m) not found: skip"

      backend_cfg=$(ls -1 "${environment_type}.tfbackend" 2>/dev/null || ls -1 "default.tfbackend" 2>/dev/null || echo "")

      if [[ -f "$backend_cfg" ]]

      then

        log_info "--- backend config file (\\e[33;1m${backend_cfg}\\e[0m) found: use"

        tf_backend_opts="-backend-config=${backend_cfg}"

      else

        log_info "--- no backend config file found: ignore"

      fi

    fi



    # shellcheck disable=SC2154,SC2086,SC2046
@@ -374,8 +384,18 @@ stages: 


    # shellcheck disable=SC2154

    log_info "--- \\e[32mplan\\e[0m"



    # implicit tfvars

    env_vars=$(ls -1 "${environment_type}.env.tfvars" 2>/dev/null || ls -1 "${environment_type}.env.tfvars.json" 2>/dev/null || echo "")

    if [[ -f "$env_vars" ]]

    then

      log_info "--- environment-specific tfvars file (\\e[33;1m${env_vars}\\e[0m) found: use"

    else

      log_info "--- no environment-specific tfvars file found: ignore"

    fi



    # shellcheck disable=SC2154,SC2086,SC2046

    terraform plan -out "$tf_plan" $(echo "$extra_opts" | envsubst_cli) $(echo "$opts" | envsubst_cli)

    terraform plan ${env_vars:+-var-file=${env_vars}} -out "$tf_plan" $(echo "$extra_opts" | envsubst_cli) $(echo "$opts" | envsubst_cli)



    # then generate GitLab TF report

    if ! command -v jq > /dev/null
@@ -389,17 +409,6 @@ stages: 
    terraform show --json "$tf_plan" | jq -r '([.resource_changes[]?.change.actions?]|flatten)|{"create":(map(select(.=="create"))|length),"update":(map(select(.=="update"))|length),"delete":(map(select(.=="delete"))|length)}' > "$gitlab_report"

  }



  tf_infracost() {

    # shellcheck disable=SC2154

    log_info "--- \\e[32minfracost\\e[0m"



    if [[ "${TF_INFACOST_USAGE_FILE}" ]];then

      tf_usagefile="--sync-usage-file --usage-file ${TF_INFACOST_USAGE_FILE}"

    fi

    # shellcheck disable=SC2086

    infracost ${TF_INFRACOST_ARGS} ${tf_usagefile} --path .

  }



  function tf_apply() {

    opts=$1

    extra_opts=$2
@@ -419,13 +428,22 @@ stages: 
      log_info "--- \\e[32mpre-apply\\e[0m hook (\\e[33;1m${prescript}\\e[0m) not found: skip"

    fi



    # implicit tfvars

    env_vars=$(ls -1 "${environment_type}.env.tfvars" 2>/dev/null || ls -1 "${environment_type}.env.tfvars.json" 2>/dev/null || echo "")

    if [[ -f "$env_vars" ]]

    then

      log_info "--- environment-specific tfvars file (\\e[33;1m${env_vars}\\e[0m) found: use"

    else

      log_info "--- no environment-specific tfvars file found: ignore"

    fi



    if [[ "$tf_plan" ]]; then

      log_info "--- applying upstream plan: \\e[33;1m${tf_plan}\\e[0m"

      terraform apply -auto-approve "$tf_plan"

      terraform apply -auto-approve ${env_vars:+-var-file=${env_vars}} "$tf_plan"

    else

      log_info "--- auto plan + apply"

      # shellcheck disable=SC2154,SC2086,SC2046

      terraform apply -auto-approve $(echo "$extra_opts" | envsubst_cli) $(echo "$opts" | envsubst_cli)

      terraform apply -auto-approve ${env_vars:+-var-file=${env_vars}} $(echo "$extra_opts" | envsubst_cli) $(echo "$opts" | envsubst_cli)

    fi



    # maybe execute post apply script
@@ -459,8 +477,17 @@ stages: 
      log_info "--- \\e[32mpre-destroy\\e[0m hook (\\e[33;1m${prescript}\\e[0m) not found: skip"

    fi



    # implicit tfvars

    env_vars=$(ls -1 "${environment_type}.env.tfvars" 2>/dev/null || ls -1 "${environment_type}.env.tfvars.json" 2>/dev/null || echo "")

    if [[ -f "$env_vars" ]]

    then

      log_info "--- environment-specific tfvars file (\\e[33;1m${env_vars}\\e[0m) found: use"

    else

      log_info "--- no environment-specific tfvars file found: ignore"

    fi



    # shellcheck disable=SC2154,SC2086,SC2046

    terraform destroy -auto-approve $(echo "$extra_opts" | envsubst_cli) $(echo "$opts" | envsubst_cli)

    terraform destroy -auto-approve ${env_vars:+-var-file=${env_vars}} $(echo "$extra_opts" | envsubst_cli) $(echo "$opts" | envsubst_cli)



    # remove gitlab-managed tf state

    if [[ "$TF_GITLAB_BACKEND_DISABLED" != "true" ]]
@@ -484,6 +511,17 @@ stages: 
    fi

  }



  tf_infracost() {

    # shellcheck disable=SC2154

    log_info "--- \\e[32minfracost\\e[0m"



    if [[ "${TF_INFACOST_USAGE_FILE}" ]];then

      tf_usagefile="--sync-usage-file --usage-file ${TF_INFACOST_USAGE_FILE}"

    fi

    # shellcheck disable=SC2086

    infracost ${TF_INFRACOST_ARGS} ${tf_usagefile} --path .

  }



  unscope_variables

  eval_secrets