Commit 5eaea82b authored by Thomas Boni's avatar Thomas Boni
Browse files

Merge branch...

Merge branch '599-update-job-aws_s3_sync-to-deploy-a-static-website-and-create-a-bucket' into 'latest'

Resolve "[Update job] - aws_s3_sync to deploy a static website and create a bucket"

Closes #599

See merge request r2devops/hub!369
parents ad88ee36 5cebe26b
Loading
Loading
Loading
Loading
+3 −2
Original line number Diff line number Diff line
@@ -44,3 +44,4 @@ aws_s3_sync:
    - $([ ${DELETE_OLD_FILE} == "true" ]) && options="${options} --delete"
    # Synchronise the directory
    - aws s3 sync ${SYNC_DIR} s3://${AWS_BUCKET_NAME}/ ${options}
+7 −0
Original line number Diff line number Diff line
# Changelog

All notable changes to this job will be documented in this file.

## [0.1.0] - 2022-07-05
* Initial version

jobs/s3_sync/README.md

0 → 100644
+50 −0
Original line number Diff line number Diff line
## Objective

Deploy your static website on following S3-compatible cloud-providers:
- AWS
- Scaleway

## How to use it

1. Include this job in your configuration. Copy/paste link in the right panel
   and see [use the
   hub](https://docs.r2devops.io/get-started-use-the-hub/#quick-setup)
1. Set credentials variables `S3_ACCESS_KEY_ID` and `S3_SECRET_ACCESS_KEY` in
   the Gitlab CI/CD variables section of your project. Follow these guides
   depending of your Cloud-provider:
    - [AWS](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-creds-create){:target="_blank"}
    - [Scaleway](https://www.scaleway.com/en/docs/console/my-project/how-to/generate-api-key/){:target="_blank"}
1. If you are not using AWS: specify the S3 endpoint of your provider in
   `S3_ENDPOINT` variable.
    - [Scaleway](https://www.scaleway.com/en/docs/storage/object/api-cli/object-storage-aws-cli/){:target="_blank"}
1. If you need to customize the job (stage, variables, ...) 👉 check the [jobs
   customization](/use-the-hub/#jobs-customization)
1. Well done, your job is ready to work ! 😀

### Variables

| Name | Description | Default |
| ---- | ----------- | ------- |
| `S3_PROVIDER` | Name of the Provider, could be `aws` or `scaleway`  | `aws` |
| `S3_ACCESS_KEY_ID` | Access key | ` ` |
| `S3_SECRET_ACCESS_KEY` | Secret key | ` ` |
| `S3_REGION` | Region used | `us-west-1` |
| `S3_SYNC_DIR` | Directory to sync | `website_build` |
| `S3_BUCKET_NAME`| The name of the bucket | `$CI_PROJECT_PATH_SLUG` |
| `S3_ACL` | Use custom ACL ([from this list](https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl)) | `public-read` |
| `S3_ENDPOINT` | Custom endpoint if needed | ` ` |
| `S3_DELETE_OLD_FILE` | Delete files that exist in the destination but not in the source  | `true` |
| `S3_BUCKET_POLICY_FILE` | The policy applied to the bucket. If not set will apply `${S3_SNIPPET_POLICY_LINK}/bucket_policy-${S3_PROVIDER}.json`. Otherwise, it should be declared in Gitlab CI/CD variables section as `file` | ` ` |
| `S3_DEPLOY_WEBSITE` | Should deploy a static website on the bucket | `true` |
| `S3_WEBSITE_HOMEPAGE` | The file for the homepage | `index.html` |
| `S3_WEBSITE_ERRORPAGE` | The file for the error page | `error.html` |
| `S3_CLI_VERSION` | The version of `AWS` cli | `2.7.7` |
| `S3_SNIPPET_POLICY_LINK` | The link where to fetch policy files | `https://gitlab.com/r2devops/hub/-/snippets/2351961` |
| `S3_OPTIONS` | Additional option(s) to use in AWS CLI | ` ` |
| `IMAGE_TAG` | The default tag for the image | `2.7.12` |

### Author

This resource is an **[official job](https://docs.r2devops.io/faq-labels/)**
added in [**R2Devops repository**](https://gitlab.com/r2devops/hub) by
[@GridexX](https://gitlab.com/GridexX)
+92 −0
Original line number Diff line number Diff line
# Job from R2Devops hub --> r2devops.io

stages:
  - deploy

s3_sync:
  stage: deploy
  image:
    name: amazon/aws-cli:${IMAGE_TAG}
    entrypoint: [""]
  variables:
    S3_PROVIDER: "aws"
    S3_ENDPOINT: ""
    S3_ACCESS_KEY_ID: ""
    S3_SECRET_ACCESS_KEY: ""
    S3_REGION: "us-west-1"
    S3_SYNC_DIR: "website_build"
    S3_BUCKET_NAME: "$CI_PROJECT_PATH_SLUG"
    S3_ACL: "public-read"
    S3_OPTIONS: ""
    S3_DELETE_OLD_FILE: "true"
    S3_BUCKET_POLICY_FILE: ""
    S3_DEPLOY_WEBSITE: "true"
    S3_WEBSITE_HOMEPAGE: "index.html"
    S3_WEBSITE_ERRORPAGE: "error.html"
    S3_SNIPPET_POLICY_LINK: "https://gitlab.com/r2devops/hub/-/snippets/2351961/raw/main"
    IMAGE_TAG: "2.7.12"
    AWS_ACCESS_KEY_ID: "$S3_ACCESS_KEY_ID"
    AWS_SECRET_ACCESS_KEY: "$S3_SECRET_ACCESS_KEY"
  script:

      # Check provider
    - |
      if [ -z ${S3_PROVIDER} ]; then
        echo "S3_PROVIDER not set, will use default AWS provider"
        S3_PROVIDER="aws"
      fi

      # Set some variables
    - |
      S3_OPTIONS="--region ${S3_REGION} ${S3_OPTIONS}"
      if [ -z ${CI_ENVIRONMENT_SLUG} ]; then
        S3_ENV="production"
      fi
      S3_BUCKET_NAME="${S3_BUCKET_NAME}-${S3_ENV}"

      # Set Endpoint URL
    - |
      if [ ! -z ${S3_ENDPOINT} ]; then
        S3_OPTIONS="--endpoint-url ${S3_ENDPOINT} ${S3_OPTIONS}"
      fi

      # Create the bucket if doesn't exists
    - |
      if aws s3 ls ${S3_OPTIONS} | awk '{print $3}' | egrep "^${S3_BUCKET_NAME}$"; then
        echo "Bucket ${S3_BUCKET_NAME} already exists"
      else
        echo "Creating bucket ${S3_BUCKET_NAME}"
        aws s3 mb s3://${S3_BUCKET_NAME} ${S3_OPTIONS} 2>&1 > ./bucket_creation.log
        echo "✅ Bucket ${S3_BUCKET_NAME} successfully created"
      fi

      # Enable website mode
    - |
      if [ ${S3_DEPLOY_WEBSITE} == "true" ]; then
        aws s3 website s3://${S3_BUCKET_NAME}/ --index-document ${S3_WEBSITE_HOMEPAGE} \
        --error-document ${S3_WEBSITE_ERRORPAGE} ${S3_OPTIONS}
      fi

      # Configuring policy and substitute variables
    - |
      if [ -z ${S3_BUCKET_POLICY_FILE} ]; then
        echo "No bucket policy file provided, use the default file for ${S3_PROVIDER}"
        S3_BUCKET_POLICY_FILE="${S3_PROVIDER}.json"
        curl -L -o $S3_BUCKET_POLICY_FILE ${S3_SNIPPET_POLICY_LINK}/${S3_BUCKET_POLICY_FILE} || (echo "S3 provider ${S3_PROVIDER} not yet supported" && exit 1)
      fi
      yum install -y gettext
      envsubst < $S3_BUCKET_POLICY_FILE > bucket-policy.json
      aws s3api put-bucket-policy --bucket ${S3_BUCKET_NAME} --policy file://bucket-policy.json ${S3_OPTIONS}
      echo "✅ Bucket policy file successfully created and applied : "

      # Syncing website
    - |
      $([ ! -z ${S3_ACL} ]) && S3_OPTIONS="${S3_OPTIONS} --acl ${S3_ACL}"
      $([ ${S3_DELETE_OLD_FILE} == "true" ]) && S3_OPTIONS="${S3_OPTIONS} --delete"
      aws s3 sync ${S3_SYNC_DIR} s3://${S3_BUCKET_NAME}/ ${S3_OPTIONS}
      if [ ${S3_PROVIDER} == "scaleway" ]; then
        WEBSITE_URL="https://${S3_BUCKET_NAME}.s3-website.${S3_REGION}.scw.cloud"
      else
        WEBSITE_URL="http://${S3_BUCKET_NAME}.s3-website-${S3_REGION}.amazonaws.com"
      fi
      echo "✅ Website successfully deployed on ${WEBSITE_URL}"
+1 −1

File changed.

Contains only whitespace changes.

+1 −1

File changed.

Contains only whitespace changes.

Loading