Commit a01b22f7 authored by Pierre Smeyers's avatar Pierre Smeyers
Browse files

Merge branch 'feat/sonar-improvements' into 'master' 

Multiple Sonar improvements

See merge request to-be-continuous/maven!35
parents fb061b69 d47e40df

README.md

+10 −29
Original line number Diff line number Diff line
@@ -81,41 +81,22 @@ It is bound to the `test` stage, and uses the following variables: 


| Name                     | description                            | default value     |

| ------------------------ | -------------------------------------- | ----------------- |

| `SONAR_URL`              | SonarQube server url            | _none_ (disabled) |

| :lock: `SONAR_AUTH_TOKEN`| SonarQube authentication [token](https://docs.sonarqube.org/latest/user-guide/user-token/) (depends on your authentication method) | _none_ |

| `SONAR_HOST_URL`         | SonarQube server url                   | _none_ (disabled) |

| :lock: `SONAR_TOKEN`     | SonarQube authentication [token](https://docs.sonarqube.org/latest/user-guide/user-token/) (depends on your authentication method) | _none_ |

| :lock: `SONAR_LOGIN`     | SonarQube login (depends on your authentication method)                | _none_ |

| :lock: `SONAR_PASSWORD`  | SonarQube password (depends on your authentication method)             | _none_ |

| `SONAR_BASE_ARGS`        | SonarQube [analysis arguments](https://docs.sonarqube.org/latest/analysis/analysis-parameters/) | `sonar:sonar -Dsonar.host.url=${SONAR_URL} -Dsonar.links.homepage=${CI_PROJECT_URL} -Dsonar.links.ci=${CI_PROJECT_URL}/-/pipelines -Dsonar.links.issue=${CI_PROJECT_URL}/-/issues` |

| :lock: `SONAR_GITLAB_TOKEN` | GitLab [access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) with `api` scope. When set, activates the [Sonar GitLab plugin](https://github.com/gabrie-allaigre/sonar-gitlab-plugin/#plugins-properties) integration. | _none_ |

| `SONAR_BRANCH_ANALYSIS_DISABLED` | Set to `true` to disable automatic [Pull Request Analysis](https://docs.sonarqube.org/latest/analysis/pull-request/) and [Branch Analysis](https://docs.sonarqube.org/latest/branches/overview/)  | _none_ (enabled) |

| `SONAR_GITLAB_ARGS`      | Extra arguments to use with [Sonar GitLab plugin](https://github.com/gabrie-allaigre/sonar-gitlab-plugin/#plugins-properties) | `-Dsonar.gitlab.url=${CI_SERVER_URL} -Dsonar.gitlab.user_token=${SONAR_GITLAB_TOKEN} -Dsonar.gitlab.project_id=${CI_PROJECT_ID} -Dsonar.gitlab.commit_sha=${CI_COMMIT_SHA} -Dsonar.gitlab.ref_name=${CI_COMMIT_REF_NAME}` |

| `SONAR_QUALITY_GATE_ENABLED` | Set to `true` to enables check of SonarQube [Quality Gate](https://docs.sonarqube.org/latest/user-guide/quality-gates/) | _none_ (disabled) |

| `SONAR_BASE_ARGS`        | SonarQube [analysis arguments](https://docs.sonarqube.org/latest/analysis/analysis-parameters/) | `sonar:sonar -Dsonar.links.homepage=${CI_PROJECT_URL} -Dsonar.links.ci=${CI_PROJECT_URL}/-/pipelines -Dsonar.links.issue=${CI_PROJECT_URL}/-/issues` |

| `SONAR_QUALITY_GATE_ENABLED` | Set to `true` to enable SonarQube [Quality Gate](https://docs.sonarqube.org/latest/user-guide/quality-gates/) verification.<br/>_Uses `sonar.qualitygate.wait` parameter ([see doc](https://docs.sonarqube.org/latest/analysis/ci-integration-overview/#header-1))._ | _none_ (disabled) |



#### Automatic Branch Analysis & Pull Request Analysis

#### Automatic Branch Analysis & Merge Request Analysis



By default, this template tries to auto-detect and use [Pull Request Analysis](https://docs.sonarqube.org/latest/analysis/pull-request/) or [Branch Analysis](https://docs.sonarqube.org/latest/branches/overview/) (depending on the context).

This template relies on SonarScanner's [GitLab integration](https://docs.sonarqube.org/latest/analysis/gitlab-integration), that is able to auto-detect whether to launch Branch Analysis or Merge Request Analysis

from GitLab's environment variables.



Those is a great SonarQube features but it assumes one of the following conditions:

:warning: This feature also depends on your SonarQube server version and license.

If using Community Edition, you'll have to install the [sonarqube-community-branch-plugin](https://github.com/mc1arke/sonarqube-community-branch-plugin) to enable automatic Branch & Merge Request analysis (only works from SonarQube version 8).



* you are using a [Developer Edition](https://www.sonarqube.org/developer-edition/) version,

* or you are using Community Edition with an opensource plugin emulating those features, such as [sonarqube-community-branch-plugin](https://github.com/mc1arke/sonarqube-community-branch-plugin).



If you're not in one of those cases, then you shall disable this feature by setting `SONAR_BRANCH_ANALYSIS_DISABLED`.



If you leave the feature enabled, if `SONAR_AUTH_TOKEN` is provided, the template will try to autodetect (using GitLab APIs) an opened merge request matching the current branch:



* If one is found, a SonarQube [Pull Request Analysis](https://docs.sonarqube.org/latest/analysis/pull-request/) will be made.

* Otherwise, a simple [Branch Analysis](https://docs.sonarqube.org/latest/branches/overview/) is performed on the current branch.



#### About Sonar GitLab plugin



The [Sonar GitLab plugin](https://github.com/gabrie-allaigre/sonar-gitlab-plugin) uses the GitLab APIs to inline comments

into your commits directly in GitLab for each new anomaly.



As explained above, this template automatically enables the Sonar GitLab plugin if `SONAR_GITLAB_TOKEN` is set.

It will then simply append the `SONAR_GITLAB_ARGS` (overridable) to the SonarQube analysis arguments.



Comments added to GitLab will appear as owned by the user associated to the GitLab [access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html).

:warning: Merge Request Analysis only works if you're running [Merge Request pipeline](https://docs.gitlab.com/ee/ci/yaml/workflow.html#switch-between-branch-pipelines-and-merge-request-pipelines) strategy (default).



### `mvn-dependency-check` job

kicker.json

+4 −20
Original line number Diff line number Diff line
@@ -47,13 +47,13 @@ 
      "description": "Code quality and security analysis with [SONARQube](https://www.sonarqube.org/)",

      "variables": [

        {

          "name": "SONAR_URL",

          "name": "SONAR_HOST_URL",

          "type": "url",

          "description": "SonarQube server url",

          "mandatory": true

        },

        {

          "name": "SONAR_AUTH_TOKEN",

          "name": "SONAR_TOKEN",

          "description": "SonarQube authentication [token](https://docs.sonarqube.org/latest/user-guide/user-token/) (depends on your authentication method)",

          "secret": true

        },
@@ -70,28 +70,12 @@ 
        {

          "name": "SONAR_BASE_ARGS",

          "description": "SonarQube [analysis arguments](https://docs.sonarqube.org/latest/analysis/analysis-parameters/)",

          "default": "sonar:sonar -Dsonar.host.url=${SONAR_URL} -Dsonar.links.homepage=${CI_PROJECT_URL} -Dsonar.links.ci=${CI_PROJECT_URL}/-/pipelines -Dsonar.links.issue=${CI_PROJECT_URL}/-/issues",

          "advanced": true

        },

		{

          "name": "SONAR_GITLAB_TOKEN",

          "description": "GitLab API access token. When set, activates the [Sonar GitLab plugin](https://github.com/gabrie-allaigre/sonar-gitlab-plugin/#plugins-properties) integration, and enables SonarQube [Pull Request Analysis](https://docs.sonarqube.org/latest/analysis/pull-request/)",

          "secret": true

        },

        {

          "name": "SONAR_BRANCH_ANALYSIS_DISABLED",

          "description": "Set to disable automatic [Pull Request Analysis](https://docs.sonarqube.org/latest/analysis/pull-request/) and [Branch Analysis](https://docs.sonarqube.org/latest/branches/overview/)",

          "type": "boolean"

        },

        {

          "name": "SONAR_GITLAB_ARGS",

          "description": "Extra arguments to use with [Sonar GitLab plugin](https://github.com/gabrie-allaigre/sonar-gitlab-plugin/#plugins-properties)",

          "default": "-Dsonar.gitlab.url=${CI_SERVER_URL} -Dsonar.gitlab.user_token=${SONAR_GITLAB_TOKEN} -Dsonar.gitlab.project_id=${CI_PROJECT_ID} -Dsonar.gitlab.commit_sha=${CI_COMMIT_SHA} -Dsonar.gitlab.ref_name=${CI_COMMIT_REF_NAME}",

          "default": "sonar:sonar -Dsonar.links.homepage=${CI_PROJECT_URL} -Dsonar.links.ci=${CI_PROJECT_URL}/-/pipelines -Dsonar.links.issue=${CI_PROJECT_URL}/-/issues",

          "advanced": true

        },

        {

          "name": "SONAR_QUALITY_GATE_ENABLED",

          "description": "Enable blocking check of SonarQube [Quality Gate](https://docs.sonarqube.org/latest/user-guide/quality-gates/) (for `master` branch)",

          "description": "Enables SonarQube [Quality Gate](https://docs.sonarqube.org/latest/user-guide/quality-gates/) verification.\n\n_Uses `sonar.qualitygate.wait` parameter ([see doc](https://docs.sonarqube.org/latest/analysis/ci-integration-overview/#header-1))._",

          "type": "boolean"

        }

      ]

templates/gitlab-ci-maven.yml

+29 −93
Original line number Diff line number Diff line
@@ -86,21 +86,10 @@ variables: 
  # default uses branch analysis: https://docs.sonarqube.org/latest/branches/overview/

  SONAR_BASE_ARGS: >-

    sonar:sonar

    -Dsonar.host.url=${SONAR_URL}

    -Dsonar.links.homepage=${CI_PROJECT_URL}

    -Dsonar.links.ci=${CI_PROJECT_URL}/-/pipelines

    -Dsonar.links.issue=${CI_PROJECT_URL}/-/issues



  # Sonar GitLab plugin args

  # see: https://github.com/gabrie-allaigre/sonar-gitlab-plugin/#plugins-properties

  # -> enabled if $SONAR_GITLAB_TOKEN is set

  SONAR_GITLAB_ARGS: >-

    -Dsonar.gitlab.url=${CI_SERVER_URL}

    -Dsonar.gitlab.user_token=${SONAR_GITLAB_TOKEN}

    -Dsonar.gitlab.project_id=${CI_PROJECT_ID}

    -Dsonar.gitlab.commit_sha=${CI_COMMIT_SHA}

    -Dsonar.gitlab.ref_name=${CI_COMMIT_REF_NAME}



  # Dependency Check arguments

  MAVEN_DEPENDENCY_CHECK_ARGS: >-

    org.owasp:dependency-check-maven:aggregate
@@ -149,78 +138,6 @@ stages: 
    fi

  }



  function sonar_http_auth_args() {

    if [[ -n "$SONAR_AUTH_TOKEN" ]]

    then

      echo "$SONAR_AUTH_TOKEN:"

    elif [[ -n "$SONAR_LOGIN" ]] && [[ -n "$SONAR_PASSWORD" ]]

    then

      echo "$SONAR_LOGIN:$SONAR_PASSWORD"

    else

     echo ""

    fi

  }

  

  function sonar_autodetect_mr() {

    if [[ "$SONAR_BRANCH_ANALYSIS_DISABLED" == "true" ]]

    then

      log_info "Branch Analysis and Merge Request Analysis are disabled"

      return

    fi

    if [[ "$CI_MERGE_REQUEST_ID" ]]

    then

      # we are in an MR pipeline: no need to pass arguments as the SonarScanner for Maven will

      log_info "Merge Request pipeline detected: let Maven plugin handle..."

      return

    fi

    if [[ -n "$SONAR_GITLAB_TOKEN" ]]

    then 

      curl -sS --header "PRIVATE-TOKEN: $SONAR_GITLAB_TOKEN" "$CI_API_V4_URL/projects/${CI_PROJECT_ID}/merge_requests?state=opened&source_branch=${CI_COMMIT_REF_NAME}" -o mr.json

      if [ "$(cat mr.json)" != "[]" ]

      then

        mr_title=$(sed -E 's/\[\{[^{]*"title":"([^"]*)".*/\1/g' < mr.json)

        mr_target=$(sed -E 's/\[\{[^{]*"target_branch":"([^"]*)".*/\1/g' < mr.json)

        mr_id=$(sed -E 's/\[\{[^{]*"iid":([0-9]+).*/\1/g' < mr.json)

        log_info "Merge Request \\e[33;1m$mr_title\\e[0m detected associated to this branch: trigger MR analysis..."

        export sonar_mr_args="-Dsonar.pullrequest.key=$mr_id -Dsonar.pullrequest.branch=${CI_COMMIT_REF_NAME} -Dsonar.pullrequest.base=$mr_target"

      else 

        log_info "No Merge Request associated to this branch: trigger branch analysis..."

        export sonar_mr_args="-Dsonar.branch.name=${CI_COMMIT_REF_NAME}"

      fi

    else 

      log_info "\$SONAR_GITLAB_TOKEN unset: trigger branch analysis..."

      export sonar_mr_args="-Dsonar.branch.name=${CI_COMMIT_REF_NAME}"

    fi

  }



  function sonar_quality_gate_check() {

    log_info "--- Waiting for Sonar analysis end..."

    taskId=$(grep ceTaskId target/sonar/report-task.txt | cut -c10-)

    curl -sS -k --user "$(sonar_http_auth_args)" "$SONAR_URL/api/ce/task?id=$taskId" -o analysis.json

    analysisStatus=$(grep --only-matching --extended-regexp "\"status\":\"[A-Z_-]+\"" analysis.json | awk -F: '{print $2}' | tr -d '"')

    while [ "$analysisStatus" == "IN_PROGRESS" ] || [ "$analysisStatus" == "PENDING" ]; do

      sleep 5s

      curl -sS -k --user "$(sonar_http_auth_args)" "$SONAR_URL/api/ce/task?id=$taskId" -o analysis.json

      analysisStatus=$(grep --only-matching --extended-regexp "\"status\":\"[A-Z_-]+\"" analysis.json | awk -F: '{print $2}' | tr -d '"')

    done



    # Check quality gate

    if [ "$analysisStatus" == "SUCCESS" ]; then

      analysisId=$(grep --only-matching --extended-regexp "\"analysisId\":\"[a-zA-Z0-9_-]+\"" analysis.json | awk -F: '{print $2}' | tr -d '"')

      qualityGateStatus=$(curl -sS -k --user "$(sonar_http_auth_args)" "$SONAR_URL/api/qualitygates/project_status?analysisId=$analysisId" | grep --only-matching --extended-regexp "{\"projectStatus\":{\"status\":\"[A-Z_]+\"" | awk -F: '{print $3}' | tr -d '"')

      if [ "$qualityGateStatus" == "ERROR" ]; then

        log_error "... quality gate FAILED"

        exit 1

      else

        log_info "... quality gate SUCCEEDED"

        exit 0

      fi

    else

        log_error "... Sonar analysis FAILED"

      exit 1

    fi

  }



  function maybe_set_version_from_git() {

    if [[ -n "$MAVEN_RELEASE_VERSION_FROM_GIT" ]]; then

      # compute version as timestamp of latest commit
@@ -470,7 +387,7 @@ stages: 
  cache:

    key: "$CI_COMMIT_REF_SLUG-maven"

    paths:

      - ${MAVEN_CFG_DIR}/repository

      - "${MAVEN_PROJECT_DIR}/${MAVEN_CFG_DIR}/repository"



mvn-build:

  extends: .mvn-base
@@ -496,17 +413,36 @@ mvn-sonar: 
  stage: test

  extends: .mvn-base

  variables:

    # Tells git to fetch all the branches of the project, required by the analysis task

    # see: https://docs.sonarqube.org/8.5/analysis/gitlab-cicd/#header-2

    GIT_DEPTH: 0

    # see: https://docs.sonarqube.org/latest/analysis/gitlab-integration/#header-4

    SONAR_USER_HOME: "${CI_PROJECT_DIR}/${MAVEN_PROJECT_DIR}/.sonar" # Defines the location of the analysis task cache

    GIT_DEPTH: 0 # Tells git to fetch all the branches of the project, required by the analysis task

  cache:

    key: "$CI_COMMIT_REF_SLUG-maven-sonar"

    paths:

      - "${MAVEN_PROJECT_DIR}/${MAVEN_CFG_DIR}/repository"

      - "${MAVEN_PROJECT_DIR}/.sonar/cache"

  script:

    - sonar_autodetect_mr

    - if [[ "$SONAR_GITLAB_TOKEN" ]]; then sonar_extra_args="$SONAR_GITLAB_ARGS"; fi

    - mvn ${TRACE+-Dsonar.verbose=true} $MAVEN_CLI_OPTS $mvn_settings_opt $java_proxy_args ${SONAR_AUTH_TOKEN+-Dsonar.login=$SONAR_AUTH_TOKEN} ${SONAR_LOGIN+-Dsonar.login=$SONAR_LOGIN} ${SONAR_PASSWORD+-Dsonar.password=$SONAR_PASSWORD} $SONAR_BASE_ARGS $sonar_extra_args $sonar_mr_args

    - if [[ "$SONAR_QUALITY_GATE_ENABLED" == "true" ]]; then sonar_quality_gate_check; fi

    - |

      if [[ "$SONAR_URL" ]] && [[ -z "$SONAR_HOST_URL" ]]

      then 

        log_warn '$SONAR_URL variable defined: use $SONAR_HOST_URL instead (see doc)'

        export SONAR_HOST_URL="$SONAR_URL"

      fi

    - |

      if [[ "$SONAR_AUTH_TOKEN" ]] && [[ -z "$SONAR_TOKEN" ]]

      then 

        log_warn '$SONAR_AUTH_TOKEN variable defined: use $SONAR_TOKEN instead (see doc)'

        export SONAR_TOKEN="$SONAR_AUTH_TOKEN"

      fi

    - >-

      mvn ${TRACE+-Dsonar.verbose=true} $MAVEN_CLI_OPTS $mvn_settings_opt $java_proxy_args

      ${SONAR_LOGIN+-Dsonar.login=$SONAR_LOGIN}

      ${SONAR_PASSWORD+-Dsonar.password=$SONAR_PASSWORD}

      ${SONAR_QUALITY_GATE_ENABLED+-Dsonar.qualitygate.wait=$SONAR_QUALITY_GATE_ENABLED}

      $SONAR_BASE_ARGS

  rules:

    # exclude if $SONAR_URL not set

    - if: '$SONAR_URL == null || $SONAR_URL == ""'

    # exclude if $SONAR_URL and $SONAR_HOST_URL not set

    - if: '($SONAR_HOST_URL == null || $SONAR_HOST_URL == "") && ($SONAR_URL == null || $SONAR_URL == "")'

      when: never

    - !reference [.test-policy, rules]