Introduction to AWS S3 client command line

In this short article i will go over the basic command used to work with AWS S3 storage command line client. For this you will need to install the AWS client first and and have an AWS account ready for use(Is Free and comes with a good amount of free EC2 hours to use). To install the AWS client follow this article, is easy as 1,2,3 :). So the S3 utility that comes with the AWs client is a powerful tool that will enable you to manage your AWs S3 buckets for the command line and also you can implement it using scripts and create automated tasks.   I will start by creating a new bucket and from here execute some basic tasks and commands.  Add a new bucket

  • use the mb(make bucket) option followed by the {URI}:://name_of_bucket
[root@aodba~]# aws s3 mb s3://testbkt-aodba
make_bucket: s3://testbkt-aodba/
List Bucket details
  • you will use the ls option followed by the bucket name
[root@aodba~]# aws s3 ls s3://testbkt-aodba --summarize

Total Objects: 0
   Total Size: 0
Remove Bucket
  • use the rb(remove bucket) option followed by the {URI}:://name_of_bucket
[root@aodba~]# aws s3 rb s3://testbkt-aodba
remove_bucket: s3://testbkt-aodba/
Remove a bucket that contains files
  • use the rb(make bucket) option followed by the {URI}:://name_of_bucket  and the --force option.
-if you try to remove a bucket that contains files you will get the following error. 
[root@aodba]# aws s3 ls testbkt-aodba2
2016-02-08 11:11:15          0 bla
2016-02-08 11:11:15          0 bla_1
2016-02-08 11:11:15          0 bla_2

[root@aodba]# aws s3 rb s3://testbkt-aodba2
remove_bucket failed: s3://testbkt-aodba2/ A client error (BucketNotEmpty) occurred when calling the DeleteBucket operation: The bucket you tried to delete is not empty


[root@aodba]# aws s3 rb s3://testbkt-aodba2 --force
delete: s3://testbkt-aodba2/bla2
delete: s3://testbkt-aodba2/bla_1
delete: s3://testbkt-aodba2/bla
remove_bucket: s3://testbkt-aodba2/
Add new objects to a bucket: Adding a new object to a S3 bucket is quite easy and similar to the copy command from Linux (cp). Syntax: aws s3 cp source destination option Examples: 
[root@aodba]# aws s3 cp /tmp/bla s3://testbkt-aodba
upload: ./bla to s3://testbkt-aodba/bla

-- list content
[root@aodba]# aws s3 ls s3://testbkt-aodba --summarize
2016-02-08 11:02:10         66 bla

Total Objects: 1
   Total Size: 66
Copy an object from one bucket to another 
[root@aodba]# aws s3 cp s3://testbkt-aodba/bla s3://testbkt-aodba2/bla2
copy: s3://testbkt-aodba/bla to s3://testbkt-aodba2/bla2

-- list bucket contents
[root@aodba]# aws s3 ls s3://testbkt-aodba
2016-02-08 11:02:10         66 bla
[root@aodba]# aws s3 ls s3://testbkt-aodba2
2016-02-08 11:03:55         66 bla2
Copy an object from one bucket to the same bucket 
[root@aodba]# aws s3 cp s3://testbkt-aodba/bla s3://testbkt-aodba/bla_1
copy: s3://testbkt-aodba/bla to s3://testbkt-aodba/bla_1
[root@aodba]# aws s3 ls s3://testbkt-aodba
2016-02-08 11:02:10         66 bla
2016-02-08 11:07:18         66 bla_1
Copy from a S3 bucket to a local folder 
[root@aodba]# aws s3 ls testbkt-aodba
2016-02-08 11:11:15          0 bla
2016-02-08 11:11:15          0 bla_1
2016-02-08 11:11:15          0 bla_2

[root@aodba]# ls -la /tmp/localdir/
total 8
drwxr-xr-x   2 root root 4096 Feb  8 13:55 .
drwxrwxrwt. 25 root root 4096 Feb  8 12:42 ..

[root@aodba]# aws s3 cp s3://testbkt-aodba/bla_* .
A client error (404) occurred when calling the HeadObject operation: Key "bla_*" does not exist
Completed 1 part(s) with ... file(s) remaining

[root@prodvert01 localdir]# aws s3 cp s3://testbkt-aodba/bla_1 /tmp/localdir/
download: s3://testbkt-aodba/bla_1 to ./bla_1

[root@prodvert01 localdir]# ls -la
total 8
drwxr-xr-x   2 root root 4096 Feb  8 13:56 .
drwxrwxrwt. 25 root root 4096 Feb  8 12:42 ..
-rw-r--r--   1 root root    0 Feb  8 11:11 bla_1

[root@aodba]# aws s3 cp s3://testbkt-aodba/bla_2 .
download: s3://testbkt-aodba/bla_2 to ./bla_2

[root@aodba]# ls -la
total 8
drwxr-xr-x   2 root root 4096 Feb  8 13:56 .
drwxrwxrwt. 25 root root 4096 Feb  8 12:42 ..
-rw-r--r--   1 root root    0 Feb  8 11:11 bla_1
-rw-r--r--   1 root root    0 Feb  8 11:11 bla_2
Sync the content of two buckets
  • the sync S3 option recursively copies new and updated files from the source directory to the destination.
Syntax:  aws s3 sync {source} {destination} {option} Examples: 
[root@AODBA]# aws s3 sync s3://testbkt-aodba s3://testbkt-aodba2
copy: s3://testbkt-aodba/bla to s3://testbkt-aodba2/bla
copy: s3://testbkt-aodba/bla_1 to s3://testbkt-aodba2/bla_1
Sync the content of one local folder and a S3 bucket 
[root@aodba]# pwd
/tmp/localdir
[root@aodba]# ll
total 0
-rw-r--r-- 1 root root 0 Feb  8 11:10 bla

-rw-r--r-- 1 root root 0 Feb  8 11:10 bla_1

-rw-r--r-- 1 root root 0 Feb  8 11:10 bla_2

[root@aodba]# aws s3 ls s3://testbkt-aodba
2016-02-08 11:02:10         66 bla
2016-02-08 11:07:18         66 bla_1

-- sync

[root@aodba]# aws s3 sync /tmp/localdir s3://testbkt-aodba
upload: ./bla_2 to s3://testbkt-aodba/bla_2
upload: ./bla_1 to s3://testbkt-aodba/bla_1
upload: ./bla to s3://testbkt-aodba/bla

-- list content

[root@aodba]# aws s3 ls s3://testbkt-aodba
2016-02-08 11:11:15          0 bla
2016-02-08 11:11:15          0 bla_1
2016-02-08 11:11:15          0 bla_2
I will post more advanced AWS S3 command line examples in the future articles.

Copyright 2020. All Rights Reserved