9 AWS S3 commands and examples for managing buckets and data

Controlling and managing data can be a daunting task. These AWS S3 commands help you manage your AWS S3 buckets and data quickly and efficiently.

AWS S3 is an object storage service provided by AWS. This is AWS’s most widely used storage service that can hold virtually unlimited amounts of data. It is highly available, durable, and easily integrated with several other AWS services.

AWS S3 can be used by users with any requirement, including mobile/web application storage, big data storage, machine learning data storage, and static website hosting.

If you’ve ever used S3 in a project, you know that managing hundreds of buckets and terabytes of data within those buckets can be a daunting task, given the sheer amount of storage capacity. You know. We have a list of AWS S3 commands with examples that you can use to efficiently manage your AWS S3 buckets and data.

Setting up the AWS CLI

After you successfully download and install the AWS CLI, you need to configure your AWS credentials to access your AWS account and services. A quick guide to configuring the AWS CLI.

The first step is to create a user with programmatic access to your AWS account. Be sure to check this box when creating a user for the AWS CLI.

Create a user with permissions. On the final screen after this user is successfully created, copy this user’s Access Key ID and Secret Access Key. Use these credentials to log in via the AWS CLI.

Then go to your terminal of choice and run the following command:

 aws configure  

When prompted, enter your access key ID and secret access key. Choose your AWS Region and command output format. I personally prefer using JSON format. This is no big deal and you can always change these values ​​later.

You can now run any AWS CLI commands in the console. Next, let’s take a look at the AWS S3 commands.

C.P.

The cp command simply copies data to and from your S3 bucket. You can use it to copy files from local to S3, S3 to local, and between two S3 buckets. There are many other parameters that can be specified with the command.

For example, the -dryrun parameter to test your commands, the –storage-class parameter to specify the storage class of your data in S3, and other parameters to configure encryption. The cp command gives you complete control over how data security is configured in S3.

Usage

 aws s3 cp   [--options] 

example

Copy data from local to S3

 aws s3 cp file_name.txt s3://bucket_name/file_name_2.txt 

Copy data locally from S3

 aws s3 cp s3://bucket_name/file_name_2.txt file_name.txt 

Copy data between S3 buckets

 aws s3 cp s3://bucket_name/file_name.txt s3://bucket_name_2/file_name_2.txt 

Copy data from local to S3 – IA

 aws s3 cp file_name.txt s3://bucket_name/file_name_2.txt --storage-class STANDARD_IA  

Copy all data from local folders to S3.

 aws s3 cp ./local_folder s3://bucket_name --recursive 

ls

The ls command is used to list buckets or their contents. So, if you simply want to view information about a bucket or the data within a bucket, you can use the ls command.

Usage:

 aws s3 ls NONE or  [--options] 

example

List all buckets in your account

 aws s3 ls

Output:
2022-02-02 18:20:14 BUCKET_NAME_1
2022-03-20 13:12:43 BUCKET_NAME_2
2022-03-29 10:52:33 BUCKET_NAME_3 

This command lists all buckets in your account along with the bucket creation date.

List all top-level objects in a bucket

 aws s3 ls BUCKET_NAME_1 or s3://BUCKET_NAME_1 

Output:
                           PRE samplePrefix/
2021-12-09 12:23:20       8754 file_1.png
2021-12-09 12:23:21       1290 file_2.json
2021-12-09 12:23:21       3088 file_3.html
 

This command lists all top-level objects in your S3 bucket. Note that objects with the prefix samplePrefix/ are not the only top-level objects shown here.

List all objects in a bucket

 aws s3 ls BUCKET_NAME_1 or s3://BUCKET_NAME_1 --recursive

Output:
2021-12-09 12:23:20       8754 file_1.png
2021-12-09 12:23:21       1290 file_2.json
2021-12-09 12:23:21       3088 file_3.html
2021-12-09 12:23:20      16328 samplePrefix/file_1.txt
2021-12-09 12:23:20      29325 samplePrefix/sampleSubPrefix/file_1.css 

This command lists all objects in your S3 bucket. Note that objects with the prefix samplePrefix/ and all subprefixes are also displayed here.

M.B.

The mb command is simply used to create a new S3 bucket. This is a very simple command, but to create a new bucket, the name of the new bucket must be unique among all S3 buckets.

Usage

 aws s3 mb  

example

Create a new bucket in a specific region

 aws s3 mb myUniqueBucketName --region eu-west-1 

MV

The mv command simply moves data to and from your S3 bucket. Similar to the cp command, the mv command is used to move data from local to S3, S3 to local, or between two S3 buckets.

The only difference between the mv and cp commands is that the mv command removes files from the source. AWS moves this file to your destination. There are many options that can be specified with the command.

Usage

 aws s3 mv   [--options] 

example

Move data from local to S3

 aws s3 mv file_name.txt s3://bucket_name/file_name_2.txt 

Move data from S3 locally

 aws s3 mv s3://bucket_name/file_name_2.txt file_name.txt 

Move data between S3 buckets

 aws s3 mv s3://bucket_name/file_name.txt s3://bucket_name_2/file_name_2.txt 

Move data from local to S3 – IA

 aws s3 mv file_name.txt s3://bucket_name/file_name_2.txt --storage-class STANDARD_IA  

Move all data from the S3 prefix to a local folder.

 aws s3 mv s3://bucket_name/somePrefix ./localFolder --recursive 

sign

The presign command generates a signed URL for a key in your S3 bucket. You can use this command to generate a URL that other users can use to access files in the specified S3 bucket key.

Usage

aws s3 presign <OBJECT_KEY> –expires-in <TIME_IN_SECONDS>

example

Generates signed URLs for objects in your bucket that are valid for one hour.

 aws s3 presign s3://bucket_name/samplePrefix/file_name.png --expires-in 3600

Output:
https://s3.ap-south-1.amazonaws.com/bucket_name/samplePrefix/file_name.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA4MCZT73PAX7ZMVFW%2F20220314%2Fap-south-1%2Fs3%2Faws4_request&X-Amz-Date=20220314T054113Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=f14608bbf3e1f9f8d215eb5b439b87e167b1055bcd7a45c13a33debd3db1be96 

R.B.

The rb command is simply used to delete an S3 bucket.

Usage

 aws rb  

example

Delete an S3 bucket.

 aws s3 mb myBucketName
# This command fails if there is any data in this bucket. 

Delete the S3 bucket along with the data in the S3 bucket.

 aws s3 mb myBucketName --force 

rm

The rm command is only used to delete objects in an S3 bucket.

Usage

 aws s3 rm  

example

Delete a single file from your S3 bucket.

 aws s3 rm s3://bucket_name/sample_prefix/file_name_2.txt 

Delete all files with a specific prefix in an S3 bucket.

 aws s3 rm s3://bucket_name/sample_prefix --recursive 

Delete all files in your S3 bucket.

 aws s3 rm s3://bucket_name --recursive 

synchronize

The sync command copies and updates files from a source to a destination, similar to the cp command. It’s important to understand the difference between the cp and sync commands. cp copies data from the source to the destination, even if the data already exists at the destination.

Also, even if a file is deleted from the source, it is not deleted from the destination. However, Sync checks the destination before copying data and only copies new and updated files. The sync command is similar to committing and pushing changes to a remote branch in git. The sync command provides many options to customize the command.

Usage

 aws s3 sync   [--options] 

example

Sync local folders to S3

 aws s3 sync ./local_folder s3://bucket_name 

Sync S3 data to local folders

 aws s3 sync s3://bucket_name ./local_folder 

Synchronize data between two S3 buckets

 aws s3 sync s3://bucket_name s3://bucket_name_2 

Move data between two S3 buckets excluding all .txt files

 aws s3 sync s3://bucket_name s3://bucket_name_2 --exclude "*.txt 

Website

You can use an S3 bucket to host static websites. The website command is used to configure S3 static website hosting for your bucket.

After you specify the index and error files, S3 provides a URL where you can view the files.

Usage

 aws s3 website  [--options] 

example:

Configure static hosting for an S3 bucket and specify index and error files

 aws s3 website s3://bucket_name --index-document index.html --error-document error.html 

conclusion

We hope the above information has helped you understand some of the frequently used AWS S3 commands to manage buckets. If you want to learn more, check out our AWS certification details.