Merge branch '416-new-job-aws-s3-sync' into 'latest' (dd91a9d6) · Commits · mirror-bot / Hub

.spell_check/dico.txt

+4 −0

Original line number	Diff line number	Diff line
		@@ -11,7 +11,10 @@ args
		arror
		aspell
		aws
		acl
		ACL
		behaviour
		bitnami
		bom
		bulary
		bundler
		@@ -365,6 +368,7 @@ ISORT
		LWC
		MYPY
		SCALAFIX
		Scaleway
		SFDX
		SQLFLUFF
		SWIFTLINT

jobs/aws_s3_sync/README.md

0 → 100644

+43 −0

Original line number	Diff line number	Diff line
		## Objective

		This job uses the AWS CLI (v1 for now) in order to sync files between a directory and a target S3 bucket. It's compatible with all s3 object storage (not only AWS).

		[AWS CLI](https://aws.amazon.com/cli/){:target="_blank"} is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts.

		## How to use it
		1. Add this job URL inside the `include` list of your `.gitlab-ci.yml` file (see the [quick setup](/use-the-hub/#quick-setup)). You can specify [a fixed version](#changelog) instead of `latest`.
		```yaml
		- remote: 'https://jobs.r2devops.io/latest/aws_s3_sync.yml'
		```
		1. Set your crendentials variables in the Gitlab CI/CD variables section of your project (see the [S3 documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html){:target="_blank"} about credentials) .

		??? summary "Need a custom Endpoint ⚓ ?"
		Just type your custom url in the variable `AWS_ENDPOINT`
		For example, you can find the custom endpoint url for Scaleway [here](https://www.scaleway.com/en/docs/storage/object/api-cli/object-storage-aws-cli/){:target="_blank"}.

		1. If you need to customize the job (stage, variables, ...) 👉 check the [jobs
		customization](/use-the-hub/#jobs-customization)
		1. Well done, your job is ready to work ! 😀

		## Job details
		* Job name: `aws_s3_sync`
		* Docker image: [bitnami/aws-cli:2.2.41](https://hub.docker.com/r/bitnami/aws-cli){:target="blank"}
		* Default stage: `deploy`

		### Variables

		\| Name \| Description \| Default \|
		\| ---- \| ----------- \| ------- \|
		\| `AWS_ACCESS_KEY_ID` \| Access key \| ` ` \|
		\| `AWS_SECRET_ACCESS_KEY` \| Secret key \| ` ` \|
		\| `AWS_DEFAULT_REGION` \| Region used \| ` ` \|
		\| `AWS_ENDPOINT` \| Custom endpoint if needed \| ` ` \|
		\| `AWS_ACL` \| If you want to add an ACL (e.g. `public-read`) \| ` ` \|
		\| `SYNC_DIR` \| Directory to sync \| `build` \|
		\| `BUCKET_NAME`\| The name of the bucket \| ` ` \|
		\| `S3_SIGNATURE_VERSION` \| S3 signature version \| `s3v4` \|
		\| `S3_MAX_CONCURRENT_REQUESTS` \| The maximum concurrent requests' number \| `100` \|
		\| `S3_MAX_QUEUE_SIZE` \| The maximum queue's size \| `1000` \|
		\| `S3_MULTIPART_THRESHOLD` \| The multipart threshold (MB at end) \| `50MB` \|
		\| `S3_MULTIPART_CHUNCKSIZE` \| The multipart chuncksize (MB at end) \| `10MB` \|
		\| `S3API_ENDPOINTURL` \| The S3 endpoint URL \| ` ` \|
		No newline at end of file

jobs/aws_s3_sync/aws_s3_sync.yml

0 → 100644

+59 −0

Original line number	Diff line number	Diff line
		# Job from R2Devops hub --> r2devops.io

		stages:
		- deploy

		aws_s3_sync:
		stage: deploy
		image:
		name: bitnami/aws-cli:2.2.41
		entrypoint: [""]
		variables:
		AWS_ACCESS_KEY_ID: " "
		AWS_SECRET_ACCESS_KEY: " "
		AWS_DEFAULT_REGION: ""
		AWS_ENDPOINT: " "
		SYNC_DIR: "build"
		BUCKET_NAME: " "
		AWS_ACL: " "

		S3_SIGNATURE_VERSION: "s3v4"
		S3_MAX_CONCURRENT_REQUESTS: "100"
		S3_MAX_QUEUE_SIZE: "1000"
		S3_MULTIPART_THRESHOLD: "50MB"
		S3_MULTIPART_CHUNCKSIZE: "10MB"
		S3API_ENDPOINTURL: " "

		script:

		- pip3 install awscli-plugin-endpoint

		- touch new_config_aws
		- export AWS_CONFIG_FILE=new_config_aws

		- aws configure set plugins.endpoint awscli_plugin_endpoint

		- aws configure set region $AWS_DEFAULT_REGION
		# set custom endpoint if populated
		- if [ ! -z ${AWS_ENDPOINT} ]; then
		- aws configure set default.s3.endpoint_url ${AWS_ENDPOINT}
		- fi

		- aws configure set default.s3.signature_version ${S3_SIGNATURE_VERSION}
		- aws configure set default.s3.max_concurrent_requests ${S3_MAX_CONCURRENT_REQUESTS}
		- aws configure set default.s3.max_queue_size ${S3_MAX_QUEUE_SIZE}
		- aws configure set default.s3.multipart_threshold ${S3_MULTIPART_THRESHOLD}
		- aws configure set default.s3.multipart_chunksize ${S3_MULTIPART_CHUNCKSIZE}

		- if [ ! -z ${S3API_ENDPOINTURL} ]; then
		- aws configure set default.s3api.endpointurl ${S3API_ENDPOINTURL}
		- fi


		#update the ACL
		- if [ ! -z "$AWS_ACL" ]; then
		- aws s3api put-bucket-acl $AWS_ACL
		- fi

		#sync to the specified bucket
		- aws s3 sync $SYNC_DIR s3://${BUCKET_NAME}
		No newline at end of file

jobs/aws_s3_sync/job.yml

0 → 100644

+15 −0

Original line number	Diff line number	Diff line
		name: aws_s3_sync
		description: A ready-to-use AWS S3 job to sync a local directory into a bucket. This Job is compatible with all S3 Object storage.
		default_stage: deploy
		icon: 🪣
		maintainer: GridexX
		license: MIT
		images:
		"bitnami/aws-cli": '2.2.41'
		tools:
		AWS CLI: "1.20.44"
		Python: "3.7.12"
		labels:
		- GitLab
		- AWS
		- S3

jobs/aws_s3_sync/versions/0.1.0.md

0 → 100644

+1 −0

Original line number	Diff line number	Diff line
		* Initial version