How to use AWS S3 CLI to dump files to stdout in BASH?
BashAmazon Web-ServicesAmazon S3Aws CliBash Problem Overview
I'm starting a bash script which will take a path in S3 (as specified to the ls command) and dump the contents of all of the file objects to stdout. Essentially I'd like to replicate cat /path/to/files/* except for S3, e.g. s3cat '/bucket/path/to/files/*'. My first inclination looking at the options is to use the cp command to a temporary file and then cat that.
Has anyone tried this or similar or is there already a command I'm not finding which does it?
Bash Solutions
Solution 1 - Bash
> dump the contents of all of the file objects to stdout.
You can accomplish this if you pass - for destination of aws s3 cp command.
For example, $ aws s3 cp s3://mybucket/stream.txt -.
What you're trying to do is something like this? ::
#!/bin/bash
BUCKET=YOUR-BUCKET-NAME
for key in `aws s3api list-objects --bucket $BUCKET --prefix bucket/path/to/files/ | jq -r '.Contents[].Key'`
do
echo $key
aws s3 cp s3://$BUCKET/$key - | md5sum
done
Solution 2 - Bash
If you are using a version of the AWS CLI that doesn't support copying to "-" you can also use /dev/stdout:
$ aws s3 cp --quiet s3://mybucket/stream.txt /dev/stdout
You also may want the --quiet flag to prevent a summary line like the following from being appended to your output:
> download: s3://mybucket/stream.txt to ../../dev/stdout
Solution 3 - Bash
You can try using s3streamcat, it supports bzip, gzip and xz formats as well.
Install with
sudo pip install s3streamcat
Usage:
s3streamcat s3://bucketname/dir/file_path
s3streamcat s3://bucketname/dir/file_path | more
s3streamcat s3://bucketname/dir/file_path | grep something
Solution 4 - Bash
Ah ha!
https://pypi.python.org/pypi/s3cat/1.0.8
I'm writing more characters to satisfy the length requirement.
Solution 5 - Bash
If you wish to accomplish this using BASH, you'll have to call-out to an external app such as the AWS Command-Line Interface (CLI). It does not have a CAT equivalent, so you would need to copy the file locally and then CAT it.
Alternatively, you could use/write an app that directly calls the AWS SDK, which is available for languages such as Python, PHP, Java. By using the SDK, file contents can be retrieved in-memory and then sent to stdout.