Using Bash to Build an AWS S3 Tool

devopsawss3

Aug 19

Introduction

Amazon S3, a reliable and scalable object storage service by AWS, is extensively used to store and manage data. In this blog post, we will walk through creating a simple bash script that allows you to list objects in an S3 bucket and download them using arguments passed through the command line.

1. How to Access the S3 Bucket Using a Shell Script?

To access an S3 bucket via a shell script, you need to have the AWS CLI installed and configured. Here’s how you can set it up:

Step 1: Install the AWS CLI using the following command:

$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
$ unzip awscliv2.zip
$ sudo ./aws/install

Step 2: Configure AWS CLI by entering your credentials:

$ aws configure

This command will prompt you for your Access Key, Secret Key, default region, and output format.

2. How to Download S3 Files from the Command Line?

To download files from an S3 bucket through the command line, you use the aws s3 cp command. Example:

$ aws s3 cp s3://my-bucket/file.txt /local/path/file.txt

3. How Do I Get a List of Files from an S3 Bucket?

To list the files (or objects) in an S3 bucket, use the aws s3 ls command. Example:

$ aws s3 ls s3://my-bucket/

4. How Do I Download Data from AWS S3?

To download data from S3, you use the aws s3 cp command, as mentioned earlier. To download an entire directory, use the --recursive flag. Example:

$ aws s3 cp s3://my-bucket/ /local/path/ --recursive

Writing the Bash Script

Now, let’s combine these commands into a simple bash script that takes command line arguments. Save this script as s3_operations.sh:

#!/bin/bash

ACTION=$1
BUCKET=$2
SOURCE=$3
DEST=$4

case $ACTION in
  list)
    aws s3 ls s3://$BUCKET/
    ;;
  download)
    aws s3 cp s3://$BUCKET/$SOURCE $DEST
    ;;
  *)
    echo "Usage: $0 {list|download} <bucket> <source> <destination>"
    exit 1
esac

Usage Examples:

To list objects in a bucket:

$ ./s3_operations.sh list my-bucket

To download a file:

$ ./s3_operations.sh download my-bucket file.txt /local/path/file.txt

Conclusion

With a simple bash script, we can make it much easier and more efficient to interact with Amazon S3. This script is flexible and can be extended to perform other operations as well, such as uploading files, deleting files, and more. Try it out, and feel free to modify and extend it for your use cases.