Merge branch '458-update-job-aws-s3-sync-should-use-aws-v2-client' into 'latest'

## Objective

This job uses the AWS CLI (v1 for now) in order to sync files between a directory and a target S3 bucket. It's compatible with all s3 object storage (not only AWS).

This job uses the latest AWS CLI version (v2 for now) in order to sync files between a directory and a target S3 bucket. It's compatible with all s3 object storage (not only AWS).

[AWS CLI](https://aws.amazon.com/cli/){:target="_blank"} is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts.

    ```yaml

      - remote: 'https://jobs.r2devops.io/latest/aws_s3_sync.yml'

    ```

1. Set your crendentials variables in the Gitlab CI/CD variables section of your project (see the [S3 documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html){:target="_blank"} about credentials) .

1. Set your credentials variables in the Gitlab CI/CD variables section of your project (see the [S3 documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html){:target="_blank"} about credentials) .

    ??? summary "Need a custom Endpoint ⚓ ?"

      Just type your custom url in the variable `AWS_ENDPOINT`

## Job details

* Job name: `aws_s3_sync`

* Docker image: [bitnami/aws-cli:2.2.41](https://hub.docker.com/r/bitnami/aws-cli){:target="blank"}

* Docker image: [alpine:3.13.6](https://hub.docker.com/_/alpinei){:target="blank"}

* Default stage: `deploy`

### Variables

| `AWS_ACCESS_KEY_ID` | Access key | ` ` |

| `AWS_SECRET_ACCESS_KEY` | Secret key | ` ` |

| `AWS_DEFAULT_REGION` | Region used | ` ` |

| `AWS_BUCKET_NAME`| The name of the bucket | ` ` |

| `AWS_ENDPOINT` | Custom endpoint if needed | ` ` |

| `AWS_ACL` | If you want to add an ACL (e.g. `public-read`) | ` ` |

| `SYNC_DIR` | Directory to sync | `build` |

| `BUCKET_NAME`| The name of the bucket | ` ` |

| `S3_SIGNATURE_VERSION` | S3 signature version | `s3v4` |

| `S3_MAX_CONCURRENT_REQUESTS` | The maximum concurrent requests' number | `100` |

| `S3_MAX_QUEUE_SIZE` | The maximum queue's size | `1000` |

| `S3_MULTIPART_THRESHOLD` | The multipart threshold (MB at end) | `50MB` |

| `S3_MULTIPART_CHUNCKSIZE` | The multipart chuncksize (MB at end) | `10MB` |

| `S3API_ENDPOINTURL` | The S3 endpoint URL | ` ` |

 No newline at end of file

| `DELETE_OLD_FILE` | Delete files that exist in the destination but not in the source  | `false` |

aws_s3_sync:

  stage: deploy

  image: 

    name: bitnami/aws-cli:2.2.41

    entrypoint: [""]

    name: alpine:3.13.6

  variables:

    AWS_ACCESS_KEY_ID: " "

    AWS_SECRET_ACCESS_KEY: " "

    AWS_DEFAULT_REGION: " "

    AWS_BUCKET_NAME: " "

    AWS_ENDPOINT: " "

    SYNC_DIR: "build"

    BUCKET_NAME: " "

    AWS_ACL: " "

    S3_SIGNATURE_VERSION: "s3v4"

    S3_MAX_CONCURRENT_REQUESTS: "100"

    S3_MAX_QUEUE_SIZE: "1000"

    S3_MULTIPART_THRESHOLD: "50MB"

    S3_MULTIPART_CHUNCKSIZE: "10MB"

    S3API_ENDPOINTURL: " "

    SYNC_DIR: "build"

    DELETE_OLD_FILE: "false"

  script:

    - pip3 install awscli-plugin-endpoint

    - touch new_config_aws

    - export AWS_CONFIG_FILE=new_config_aws

    - aws configure set plugins.endpoint awscli_plugin_endpoint

    - aws configure set region $AWS_DEFAULT_REGION

    # set custom endpoint if populated

    - if [ ! -z ${AWS_ENDPOINT} ]; then

    -   aws configure set default.s3.endpoint_url ${AWS_ENDPOINT}

    - fi

    - aws configure set default.s3.signature_version ${S3_SIGNATURE_VERSION}

    - aws configure set default.s3.max_concurrent_requests ${S3_MAX_CONCURRENT_REQUESTS}

    - aws configure set default.s3.max_queue_size ${S3_MAX_QUEUE_SIZE}

    - aws configure set default.s3.multipart_threshold ${S3_MULTIPART_THRESHOLD}

    - aws configure set default.s3.multipart_chunksize ${S3_MULTIPART_CHUNCKSIZE}

    - if [ ! -z ${S3API_ENDPOINTURL} ]; then

    -   aws configure set default.s3api.endpointurl ${S3API_ENDPOINTURL}

    - fi

    #update the ACL

    - if [ ! -z "$AWS_ACL" ]; then

    -   aws s3api put-bucket-acl $AWS_ACL

    - fi

    #sync to the specified bucket

    - aws s3 sync $SYNC_DIR s3://${BUCKET_NAME}

 No newline at end of file

    # Installing AWS CLI

    - |

      apk --no-cache add \

        binutils \

        curl \

        && GLIBC_VER=$(curl -s https://api.github.com/repos/sgerrand/alpine-pkg-glibc/releases/latest | grep tag_name | cut -d: -f 2,3 | tr -d \",' ') \

      && curl -sL https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub -o /etc/apk/keys/sgerrand.rsa.pub \

      && curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-${GLIBC_VER}.apk \

      && curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-bin-${GLIBC_VER}.apk \

      && apk add --no-cache \

        glibc-${GLIBC_VER}.apk \

        glibc-bin-${GLIBC_VER}.apk \

      && curl -sL https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip \

      && unzip -qq awscliv2.zip \

      && aws/install \

      && apk --no-cache del \

        binutils \

        curl

    # Generating the options

    - options=""

    - $([ ! -z ${AWS_ENDPOINT} ]) && options="${options} --endpoint-url ${AWS_ENDPOINT}"

    - $([ ! -z ${AWS_ACL} ]) && options="${options} --AWS_ACL ${AWS_ACL}"

    - $([ ${DELETE_OLD_FILE} == "true" ]) && options="${options} --delete"

    # Synchronise the directory

    - aws s3 sync ${SYNC_DIR} s3://${AWS_BUCKET_NAME}/ ${options}

 No newline at end of file

maintainer: GridexX

license: MIT

images: 

  "bitnami/aws-cli": '2.2.41'

  "alpine": '3.13.6'

tools:

  AWS CLI: "1.20.44"

  Python: "3.7.12"

  AWS CLI: "2.2.45"

labels:

 - GitLab

 - AWS

* Updating aws CLI to v2

* Adding delete option

* Remove API config variables

 No newline at end of file