How can I see what&#39;s inside a bucket in S3 with `boto3`? (i.e. do an `&quot;ls&quot;`)?

Doing the following:

    import boto3
    s3 = boto3.resource(&#39;s3&#39;)
    my_bucket = s3.Bucket(&#39;some/path/&#39;)

returns:

    s3.Bucket(name=&#39;some/path/&#39;)

How do I see its contents?

Listing contents of a bucket with boto3

I have the following code

    import matplotlib.pyplot as plt
    import matplotlib.image as mpimg
    import numpy as np
    import boto3
    s3 = boto3.resource(&#39;s3&#39;, region_name=&#39;us-east-2&#39;)
    bucket = s3.Bucket(&#39;sentinel-s2-l1c&#39;)
    object = bucket.Object(&#39;tiles/10/S/DG/2015/12/7/0/B01.jp2&#39;)
    object.download_file(&#39;B01.jp2&#39;)
    img=mpimg.imread(&#39;B01.jp2&#39;)
    imgplot = plt.imshow(img)
    plt.show(imgplot)

and it works. But the problem it downloads file into current directory first. Is it possible to read file and decode it as image directly in RAM?


How to read image file from S3 bucket directly into memory?

I am trying to download a file from Amazon S3 bucket to my local using the below code but I get an error saying &quot;Unable to locate credentials&quot;

Given below is the code I have written:

    from boto3.session import Session
    import boto3

    ACCESS_KEY = &#39;ABC&#39;
    SECRET_KEY = &#39;XYZ&#39;

    session = Session(aws_access_key_id=ACCESS_KEY,
                  aws_secret_access_key=SECRET_KEY)
    s3 = session.resource(&#39;s3&#39;)
    your_bucket = s3.Bucket(&#39;bucket_name&#39;)

    for s3_file in your_bucket.objects.all():
        print(s3_file.key) # prints the contents of bucket
    
    s3 = boto3.client (&#39;s3&#39;)
  
    s3.download_file(&#39;your_bucket&#39;,&#39;k.png&#39;,&#39;/Users/username/Desktop/k.png&#39;)

Could anyone help me on this. Thanks.

Download file from AWS S3 using Python

For example, I have this code:

    import boto3

    s3 = boto3.resource(&#39;s3&#39;)

    bucket = s3.Bucket(&#39;my-bucket-name&#39;)

    # Does it exist???

How can I easily determine if a Boto 3 S3 bucket resource exists?

Is there a way at all to query on the `global secondary index` of dynamodb using `boto3`. I dont find any online tutorials or resources. 

How do we query on a secondary index of dynamodb using boto3?

This is the table partition key setting
[![enter image description here][1]][1]

The table content
[![enter image description here][2]][2]

When I tried to get an item from the table, it prints this error 

&gt; botocore.exceptions.ClientError: An error occurred
&gt; (ValidationException) when calling the GetItem operation: The provided
&gt; key element does not match the schema

This is my code

    dynamodb = boto3.resource(&#39;dynamodb&#39;)
    table = dynamodb.Table(&#39;testDynamodb&#39;)
    response = table.get_item(Key={&#39;userId&#39;: &quot;user2873&quot;})
    item = response[&#39;Item&#39;]
    print(item)

Any ideas? thanks.

  [1]: https://i.stack.imgur.com/RPnjj.png
  [2]: https://i.stack.imgur.com/kbKPU.png

&quot;The provided key element does not match the schema&quot; error when getting an item from DynamoDB

Boto3 Mavens,

What is the functional difference, if any, between Clients and Resources?

Are they functionally equivalent?

Under what conditions would you elect to invoke a Boto3 Resource vs. a Client (and vice-versa)?

Although I&#39;ve endeavored to answer this question by RTM...regrets, understanding the functional difference between the two eludes me.

Your thoughts?

Many, *many* thanks!

*Plane Wryter*





Are Boto3 Resources and Clients Equivalent? When Use One or Other?

I would like to know if a key exists in boto3. I can loop the bucket contents and check the key if it matches.

But that seems longer and an overkill. Boto3 official docs explicitly state how to do this. 

May be I am missing the obvious. Can anybody point me how I can achieve this.

check if a key exists in a bucket in s3 using boto3

I am trying to figure how to do proper error handling with boto3.

I am trying to create an IAM user:

    def create_user(username, iam_conn):
        try:
            user = iam_conn.create_user(UserName=username)
            return user
        except Exception as e:
            return e

When the call to create_user succeeds, I get a neat object that contains the http status code of the API call and the data of the newly created user.

Example:

    {&#39;ResponseMetadata&#39;: 
          {&#39;HTTPStatusCode&#39;: 200, 
           &#39;RequestId&#39;: &#39;omitted&#39;
          },
     u&#39;User&#39;: {u&#39;Arn&#39;: &#39;arn:aws:iam::omitted:user/omitted&#39;,
               u&#39;CreateDate&#39;: datetime.datetime(2015, 10, 11, 17, 13, 5, 882000, tzinfo=tzutc()),
               u&#39;Path&#39;: &#39;/&#39;,
               u&#39;UserId&#39;: &#39;omitted&#39;,
               u&#39;UserName&#39;: &#39;omitted&#39;
              }
    }

 This works great. But when this fails (like if the user already exists), I just get an object of type botocore.exceptions.ClientError with only text to tell me what went wrong.

Example:
ClientError(&#39;An error occurred (EntityAlreadyExists) when calling the CreateUser operation: User with name omitted already exists.&#39;,)

This (AFAIK) makes error handling very hard because I can&#39;t just switch on the resulting http status code (409 for user already exists according to the AWS API docs for IAM). This makes me think that I must be doing something the wrong way. The optimal way would be for boto3 to never throw exceptions, but juts always return an object that reflects how the API call went.

Can anyone enlighten me on this issue or point me in the right direction?

How to handle errors with boto3?


I have a boto3 client :

    boto3.client(&#39;kms&#39;)

But it happens on new machines, They open and close dynamically. 


        if endpoint is None:
            if region_name is None:
                # Raise a more specific error message that will give
                # better guidance to the user what needs to happen.
                raise NoRegionError()


Why is this happening? and why only part of the time?



boto3 client NoRegionError: You must specify a region error only sometimes

I am using the Boto 3 python library, and want to connect to AWS CloudFront.
I need to specify the correct AWS Profile (AWS Credentials), but looking at the official documentation, I see no way to specify it.

I am initializing the client using the code:
`client = boto3.client(&#39;cloudfront&#39;)`

However, this results in it using the default profile to connect.
I couldn&#39;t find a method where I can specify which profile to use.


How to choose an AWS profile when using boto3 to connect to CloudFront

I&#39;m aware that with Boto 2 it&#39;s possible to open an S3 object as a string with: [`get_contents_as_string()`](http://boto.readthedocs.org/en/latest/ref/file.html?highlight=contents%20string#boto.file.key.Key.get_contents_as_string)


Is there an equivalent function in boto3 ? 

Open S3 object as a string with Boto3

When I simply run the following code, I always gets this error.

   

    s3 = boto3.resource(&#39;s3&#39;)
    bucket_name = &quot;python-sdk-sample-%s&quot; % uuid.uuid4()
    print(&quot;Creating new bucket with name:&quot;, bucket_name)
    s3.create_bucket(Bucket=bucket_name)


I have saved my credential file in 

 `C:\Users\myname\.aws\credentials`, from where Boto should read my credentials.

Is my setting wrong?

Here is the output from `boto3.set_stream_logger(&#39;botocore&#39;, level=&#39;DEBUG&#39;)`.


    2015-10-24 14:22:28,761 botocore.credentials [DEBUG] Skipping environment variable credential check because profile name was explicitly set.
    2015-10-24 14:22:28,761 botocore.credentials [DEBUG] Looking for credentials via: env
    2015-10-24 14:22:28,773 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
    2015-10-24 14:22:28,774 botocore.credentials [DEBUG] Looking for credentials via: config-file
    2015-10-24 14:22:28,774 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file
    2015-10-24 14:22:28,774 botocore.credentials [DEBUG] Looking for credentials via: boto-config
    2015-10-24 14:22:28,774 botocore.credentials [DEBUG] Looking for credentials via: iam-role

Boto3 Error: botocore.exceptions.NoCredentialsError: Unable to locate credentials

I have a pandas DataFrame that I want to upload to a new CSV file. The problem is that I don&#39;t want to save the file locally before transferring it to s3. Is there any method like to_csv for writing the dataframe to s3 directly? I am using boto3.  
Here is what I have so far:
    
    import boto3
    s3 = boto3.client(&#39;s3&#39;, aws_access_key_id=&#39;key&#39;, aws_secret_access_key=&#39;secret_key&#39;)
    read_file = s3.get_object(Bucket, Key)
    df = pd.read_csv(read_file[&#39;Body&#39;])

    # Make alterations to DataFrame

    # Then export DataFrame to CSV through direct transfer to s3


Save Dataframe to csv directly to s3 Python

In boto 2, you can write to an S3 object using these methods:

- [Key.set_contents_from_string()](https://boto.readthedocs.io/en/latest/ref/s3.html#boto.s3.key.Key.set_contents_from_string)
- [Key.set_contents_from_file()](https://boto.readthedocs.io/en/latest/ref/s3.html#boto.s3.key.Key.set_contents_from_file)
- [Key.set_contents_from_filename()](https://boto.readthedocs.io/en/latest/ref/s3.html#boto.s3.key.Key.set_contents_from_filename)
- [Key.set_contents_from_stream()](https://boto.readthedocs.io/en/latest/ref/s3.html#boto.s3.key.Key.set_contents_from_stream)

Is there a boto 3 equivalent? What is the boto3 method for saving data to an object stored on S3?



How to write a file or data to an S3 object using boto3

I&#39;m trying to do a &quot;hello world&quot; with new [boto3][1] client for AWS.

The use-case I have is fairly simple: get object from S3 and save it to the file.

In boto 2.X I would do it like this:

    import boto
    key = boto.connect_s3().get_bucket(&#39;foo&#39;).get_key(&#39;foo&#39;)
    key.get_contents_to_filename(&#39;/tmp/foo&#39;)

In boto 3 . I can&#39;t find a clean way to do the same thing, so I&#39;m manually iterating over the &quot;Streaming&quot; object:

    import boto3
    key = boto3.resource(&#39;s3&#39;).Object(&#39;fooo&#39;, &#39;docker/my-image.tar.gz&#39;).get()
    with open(&#39;/tmp/my-image.tar.gz&#39;, &#39;w&#39;) as f:
        chunk = key[&#39;Body&#39;].read(1024*8)
        while chunk:
            f.write(chunk)
            chunk = key[&#39;Body&#39;].read(1024*8)

or

    import boto3
    key = boto3.resource(&#39;s3&#39;).Object(&#39;fooo&#39;, &#39;docker/my-image.tar.gz&#39;).get()
    with open(&#39;/tmp/my-image.tar.gz&#39;, &#39;w&#39;) as f:
        for chunk in iter(lambda: key[&#39;Body&#39;].read(4096), b&#39;&#39;):
            f.write(chunk)


And it works fine. I was wondering is there any &quot;native&quot; boto3 function that will do the same task?
 

  [1]: https://github.com/boto/boto3

How to save S3 object to a file using boto3

On boto I used to specify my credentials when connecting to S3 in such a way:

    import boto
    from boto.s3.connection import Key, S3Connection
    S3 = S3Connection( settings.AWS_SERVER_PUBLIC_KEY, settings.AWS_SERVER_SECRET_KEY )

I could then use S3 to perform my operations (in my case deleting an object from a bucket).

With boto3 all the examples I found are such:

    import boto3
    S3 = boto3.resource( &#39;s3&#39; )
    S3.Object( bucket_name, key_name ).delete()


I couldn&#39;t specify my credentials and thus all attempts fail with `InvalidAccessKeyId` error.

How can I specify credentials with boto3?


How to specify credentials when connecting to boto3 S3?

I&#39;m new to AWS using Python and I&#39;m trying to learn the boto API however I noticed that there are two major versions/packages for Python.  That would be boto and boto3.

What is the difference between the AWS boto and boto3 libraries?

What is the difference between the AWS boto and boto3

Using boto3, I can access my AWS S3 bucket:

    s3 = boto3.resource(&#39;s3&#39;)
    bucket = s3.Bucket(&#39;my-bucket-name&#39;)

Now, the bucket contains folder `first-level`, which itself contains several sub-folders named with a timestamp, for instance `1456753904534`. 
I need to know the name of these sub-folders for another job I&#39;m doing and I wonder whether I could have boto3 retrieve those for me.

So I tried:

    objs = bucket.meta.client.list_objects(Bucket=&#39;my-bucket-name&#39;)

which gives a dictionary, whose key &#39;Contents&#39; gives me all the third-level files instead of the second-level timestamp directories, in fact I get a list containing things as 

&gt; {u&#39;ETag&#39;: &#39;&quot;etag&quot;&#39;,   u&#39;Key&#39;: first-level/1456753904534/part-00014&#39;,   u&#39;LastModified&#39;:
&gt; datetime.datetime(2016, 2, 29, 13, 52, 24, tzinfo=tzutc()),  
&gt; u&#39;Owner&#39;: {u&#39;DisplayName&#39;: &#39;owner&#39;,    u&#39;ID&#39;:
&gt; &#39;id&#39;},  
&gt; u&#39;Size&#39;: size,   u&#39;StorageClass&#39;: &#39;storageclass&#39;}

you can see that the specific files, in this case `part-00014` are retrieved, while I&#39;d like to get the name of the directory alone. 
In principle I could strip out the directory name from all the paths but it&#39;s ugly and expensive to retrieve everything at third level to get the second level!

I also tried something reported [here][1]:

    for o in bucket.objects.filter(Delimiter=&#39;/&#39;):
        print(o.key)

but I do not get the folders at the desired level.

Is there a way to solve this?

  [1]: https://github.com/boto/boto3/issues/134

Retrieving subfolders names in S3 bucket from boto3

I have a text file saved on S3 which is a tab delimited table. I want to load it into pandas but cannot save it first because I am running on a heroku server. Here is what I have so far.

    import io
    import boto3
    import os
    import pandas as pd
    
    os.environ[&quot;AWS_ACCESS_KEY_ID&quot;] = &quot;xxxxxxxx&quot;
    os.environ[&quot;AWS_SECRET_ACCESS_KEY&quot;] = &quot;xxxxxxxx&quot;
    
    s3_client = boto3.client(&#39;s3&#39;)
    response = s3_client.get_object(Bucket=&quot;my_bucket&quot;,Key=&quot;filename.txt&quot;)
    file = response[&quot;Body&quot;]
    
    
    pd.read_csv(file, header=14, delimiter=&quot;\t&quot;, low_memory=False)

the error is

    OSError: Expected file path name or file-like object, got &lt;class &#39;bytes&#39;&gt; type

How do I convert the response body into a format pandas will accept?

    pd.read_csv(io.StringIO(file), header=14, delimiter=&quot;\t&quot;, low_memory=False)

    returns

    TypeError: initial_value must be str or None, not StreamingBody

    pd.read_csv(io.BytesIO(file), header=14, delimiter=&quot;\t&quot;, low_memory=False)

    returns

    TypeError: &#39;StreamingBody&#39; does not support the buffer interface


UPDATE - Using the following worked

    file = response[&quot;Body&quot;].read()

and 
    
    pd.read_csv(io.BytesIO(file), header=14, delimiter=&quot;\t&quot;, low_memory=False)


How to import a text file on AWS S3 into pandas without writing to disk

I&#39;m using boto3 to get files from s3 bucket. I need a similar functionality like `aws s3 sync`

My current code is

    #!/usr/bin/python
    import boto3
    s3=boto3.client(&#39;s3&#39;)
    list=s3.list_objects(Bucket=&#39;my_bucket_name&#39;)[&#39;Contents&#39;]
    for key in list:
        s3.download_file(&#39;my_bucket_name&#39;, key[&#39;Key&#39;], key[&#39;Key&#39;])

This is working fine, as  long as the bucket has only files.
If a folder is present inside the bucket, its throwing an error

    Traceback (most recent call last):
      File &quot;./test&quot;, line 6, in &lt;module&gt;
        s3.download_file(&#39;my_bucket_name&#39;, key[&#39;Key&#39;], key[&#39;Key&#39;])
      File &quot;/usr/local/lib/python2.7/dist-packages/boto3/s3/inject.py&quot;, line 58, in download_file
        extra_args=ExtraArgs, callback=Callback)
      File &quot;/usr/local/lib/python2.7/dist-packages/boto3/s3/transfer.py&quot;, line 651, in download_file
        extra_args, callback)
      File &quot;/usr/local/lib/python2.7/dist-packages/boto3/s3/transfer.py&quot;, line 666, in _download_file
        self._get_object(bucket, key, filename, extra_args, callback)
      File &quot;/usr/local/lib/python2.7/dist-packages/boto3/s3/transfer.py&quot;, line 690, in _get_object
        extra_args, callback)
      File &quot;/usr/local/lib/python2.7/dist-packages/boto3/s3/transfer.py&quot;, line 707, in _do_get_object
        with self._osutil.open(filename, &#39;wb&#39;) as f:
      File &quot;/usr/local/lib/python2.7/dist-packages/boto3/s3/transfer.py&quot;, line 323, in open
        return open(filename, mode)
    IOError: [Errno 2] No such file or directory: &#39;my_folder/.8Df54234&#39;


Is this a proper way to download a complete s3 bucket using boto3. How to download folders.

Boto3 to download all files from a S3 Bucket

I&#39;m trying to mock a singluar method from the boto3 s3 client object to throw an exception. But I need all other methods for this class to work as normal.

This is so I can test a singular Exception test when and error occurs performing a [upload_part_copy][1]

**1st Attempt**

    import boto3
    from mock import patch

    with patch(&#39;botocore.client.S3.upload_part_copy&#39;, side_effect=Exception(&#39;Error Uploading&#39;)) as mock:
        client = boto3.client(&#39;s3&#39;)
        # Should return actual result
        o = client.get_object(Bucket=&#39;my-bucket&#39;, Key=&#39;my-key&#39;)
        # Should return mocked exception
        e = client.upload_part_copy()

However this gives the following error:

    ImportError: No module named S3

**2nd Attempt**

After looking at the botocore.client.py source code I found that it is doing something clever and the method `upload_part_copy` does not exist. I found that it seems to call `BaseClient._make_api_call` instead so I tried to mock that

    import boto3
    from mock import patch

    with patch(&#39;botocore.client.BaseClient._make_api_call&#39;, side_effect=Exception(&#39;Error Uploading&#39;)) as mock:
        client = boto3.client(&#39;s3&#39;)
        # Should return actual result
        o = client.get_object(Bucket=&#39;my-bucket&#39;, Key=&#39;my-key&#39;)
        # Should return mocked exception
        e = client.upload_part_copy()

This throws an exception... but on the `get_object` which I want to avoid.

Any ideas about how I can only throw the exception on the `upload_part_copy` method?

  [1]: http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.upload_part_copy

Mocking boto3 S3 client method Python

I read the filenames in my S3 bucket by doing 

    objs = boto3.client.list_objects(Bucket=&#39;my_bucket&#39;)
        while &#39;Contents&#39; in objs.keys():
            objs_contents = objs[&#39;Contents&#39;]
            for i in range(len(objs_contents)):
                filename = objs_contents[i][&#39;Key&#39;]

Now, I need to get the actual content of the file, similarly to a `open(filename).readlines()`. What is the best way?

Read file content from S3 bucket with boto3

I am trying to understand when I should use a [Resource][1] and when I should use a [Client][2].

The definitions provided in boto3 docs don&#39;t really make it clear when it is preferable to use one or the other.

  [1]: http://boto3.readthedocs.io/en/latest/guide/resources.html
  [2]: http://boto3.readthedocs.io/en/latest/guide/clients.html

When to use a boto3 client and when to use a boto3 resource?

I&#39;m seeing the below error from my lambda function when I drop a file.csv into an S3 bucket. The file is not large and I even added a 60 second sleep prior to opening the file for reading, but for some reason the file has the extra &quot;.6CEdFe7C&quot; appended to it. Why is that?

    [Errno 30] Read-only file system: u&#39;/file.csv.6CEdFe7C&#39;: IOError
    Traceback (most recent call last):
    File &quot;/var/task/lambda_function.py&quot;, line 75, in lambda_handler
    s3.download_file(bucket, key, filepath)
    File &quot;/var/runtime/boto3/s3/inject.py&quot;, line 104, in download_file
    extra_args=ExtraArgs, callback=Callback)
    File &quot;/var/runtime/boto3/s3/transfer.py&quot;, line 670, in download_file
    extra_args, callback)
    File &quot;/var/runtime/boto3/s3/transfer.py&quot;, line 685, in _download_file
    self._get_object(bucket, key, filename, extra_args, callback)
    File &quot;/var/runtime/boto3/s3/transfer.py&quot;, line 709, in _get_object
    extra_args, callback)
    File &quot;/var/runtime/boto3/s3/transfer.py&quot;, line 723, in _do_get_object
    with self._osutil.open(filename, &#39;wb&#39;) as f:
    File &quot;/var/runtime/boto3/s3/transfer.py&quot;, line 332, in open
    return open(filename, mode)
    IOError: [Errno 30] Read-only file system: u&#39;/file.csv.6CEdFe7C&#39;


Code:

    def lambda_handler(event, context):

        s3_response = {}
        counter = 0
        event_records = event.get(&quot;Records&quot;, [])

        s3_items = []
        for event_record in event_records:
            if &quot;s3&quot; in event_record:
                bucket = event_record[&quot;s3&quot;][&quot;bucket&quot;][&quot;name&quot;]
                key = event_record[&quot;s3&quot;][&quot;object&quot;][&quot;key&quot;]
                filepath = &#39;/&#39; + key
                print(bucket)
                print(key)
                print(filepath)
                s3.download_file(bucket, key, filepath)

The result of the above is:

    mytestbucket
    file.csv
    /file.csv
    [Errno 30] Read-only file system: u&#39;/file.csv.6CEdFe7C&#39;

If the key/file is &quot;file.csv&quot;, then why does the s3.download_file method try to download &quot;file.csv.6CEdFe7C&quot;? I&#39;m guessing when the function is triggered, the file is file.csv.xxxxx but by the time it gets to line 75, the file is renamed to file.csv?



Error &quot;Read-only file system&quot; in AWS Lambda when downloading a file from S3

I&#39;m trying to rename a file in my s3 bucket using python boto3, I couldn&#39;t clearly understand the arguments. can someone help me here?

What I&#39;m planing is to copy object to a new object, and then delete the actual object.

I found similar questions here, but I need a solution using boto3.

Boto3/S3: Renaming an object using copy_object

I kept following JSON in S3 bucket &#39;test&#39;

    {
      &#39;Details&#39; : &quot;Something&quot; 
    }
I am using following code to read this JSON and printing the key &#39;Details&#39;

    s3 = boto3.resource(&#39;s3&#39;,
                        aws_access_key_id=&lt;access_key&gt;,
                        aws_secret_access_key=&lt;secret_key&gt;
                        )
    content_object = s3.Object(&#39;test&#39;, &#39;sample_json.txt&#39;)
    file_content = content_object.get()[&#39;Body&#39;].read().decode(&#39;utf-8&#39;)
    json_content = json.loads(repr(file_content))
    print(json_content[&#39;Details&#39;])
And i am getting error as **&#39;string indices must be integers&#39;**
I don&#39;t want to download the file from S3 and then reading..



Reading an JSON file from S3 using Python boto3

I have a variable which has the aws s3 url 

    s3://bucket_name/folder1/folder2/file1.json

I want to get the bucket_name in a variables and rest i.e /folder1/folder2/file1.json in another variable. I tried the regular expressions and could get the bucket_name like below, not sure if there is a better way.

    m = re.search(&#39;(?&lt;=s3:\/\/)[^\/]+&#39;, &#39;s3://bucket_name/folder1/folder2/file1.json&#39;)
    print(m.group(0))

How do I get the rest i.e  - folder1/folder2/file1.json ?

I have checked if there is a boto3 feature to extract the bucket_name and key from the url, but couldn&#39;t find it.

 

s3 urls - get bucket name and path

I have more than 500,000 objects on s3. I am trying to get the size of each object. I am using the following python code for that

```python
import boto3

bucket = &#39;bucket&#39;
prefix = &#39;prefix&#39;

contents = boto3.client(&#39;s3&#39;).list_objects_v2(Bucket=bucket,  MaxKeys=1000, Prefix=prefix)[&quot;Contents&quot;]

for c in contents:
    print(c[&quot;Size&quot;])
```

But it just gave me the size of the top 1000 objects. Based on the documentation we can&#39;t get more than 1000. Is there any way I can get more than that? 

How to get more than 1000 objects from S3 by using list_objects_v2?

I am able to upload an image file using:

    s3 = session.resource(&#39;s3&#39;)
    bucket = s3.Bucket(S3_BUCKET)
    bucket.upload_file(file, key)

However, I want to make the file public too. I tried looking up for some functions to set ACL for the file but seems like boto3 have changes their API and removed some functions. Is there a way to do it in the latest release of boto3?

How to upload a file to S3 and make it public using boto3?

My table is around 220mb with 250k records within it. I&#39;m trying to pull all of this data into python.  I realize this needs to be a chunked batch process and looped through, but I&#39;m not sure how I can set the batches to start where the previous left off.

Is there some way to filter my scan? From what I read that filtering occurs after loading and the loading stops at 1mb so I wouldn&#39;t actually be able to scan in new objects. 

Any assistance would be appreciated.

    import boto3
    dynamodb = boto3.resource(&#39;dynamodb&#39;,
        aws_session_token = aws_session_token,
        aws_access_key_id = aws_access_key_id,
        aws_secret_access_key = aws_secret_access_key,
        region_name = region
        )

    table = dynamodb.Table(&#39;widgetsTableName&#39;)

    data = table.scan()

 







Complete scan of dynamoDb with boto3

I&#39;m trying to use the `update_item` functionality for DynamoDB in boto3. 

I&#39;m struggling right now to update lists for items. I would like to create a new list if the list does not exist yet and otherwise append to the existing list. 

Using an `UpdateExpression` of the form `SET my_list = list_append(my_list, :my_value)` returns an error *&quot;The provided expression refers to an attribute that does not exist in the item&quot;* if the list does not exist yet.

Any idea how I would have to modify my `UpdateExpression`?

Is it possible to combine if_not_exists and list_append in update_item

I need to get the [account-id][1] of the &#39;current user&#39; in a [`boto3`][2] script. Up to now my best solution is to parse the current user [arn][3]:

    &gt;&gt;&gt; import boto3
    &gt;&gt;&gt; account_id = boto3.resource(&#39;iam&#39;).CurrentUser().arn.split(&#39;:&#39;)[4]

but I was wondering if there is a more &#39;lightweight&#39; approach. In fact

    &gt;&gt;&gt; timeit.timeit(&quot;boto3.resource(&#39;iam&#39;).CurrentUser().arn&quot;,
    ... &#39;import boto3&#39;, number=10)
    4.8895583080002325

and I actually do not need the [`CurrentUser`][4] resource in my script.


  [1]: http://docs.aws.amazon.com/general/latest/gr/acct-identifiers.html
  [2]: http://boto3.readthedocs.org/en/latest/index.html
  [3]: http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#genref-arns
  [4]: https://boto3.readthedocs.org/en/latest/reference/services/iam.html#currentuser

getting the current user account-id in boto3

I have trouble installing boto3 inside a virtual environment. 

I have done what the document says. First I activated virtual environment. then I did a:

    Sudo pip install boto3

Now I enter python

    &gt;&gt; import boto3
    ImportError: No module named boto3

But if I import boto, it works

    &gt;&gt; import boto
    &gt;&gt; boto.Version
    &#39;2.38.0&#39;

Why does it install boto 2.38 when I installed boto3.
I tried closing the terminal and re-opened it.
Should I restart the Ubuntu machine?

Unable to install boto3

I&#39;m a new user in boto3 and i&#39;m using `DynamoDB`.

I went through over the DynamoDB api and I couldn&#39;t find any method which tell me if a table is already exists.

What is the best approach dealing this issue? 

Should I try to create a new table and wrap it using try catch ?

How to check if DynamoDB table exists?

I have two AWS account - lets say A and B.

In account B, I have a role defined that allow access to another role from account A. Lets call it Role-B

&lt;!-- language: lang-json --&gt;

    {
      &quot;Version&quot;: &quot;2012-10-17&quot;,
      &quot;Statement&quot;: [
      {
         &quot;Effect&quot;: &quot;Allow&quot;,
         &quot;Principal&quot;: {
             &quot;AWS&quot;: &quot;arn:aws:iam::********:role/RoleA&quot;
         },
        &quot;Action&quot;: &quot;sts:AssumeRole&quot;
      }]
    }


In account A, I have defined a role that allows the root user to assume role. Lets call it Role-A

&lt;!-- language: lang-json --&gt;

    {
      &quot;Version&quot;: &quot;2012-10-17&quot;,
      &quot;Statement&quot;: [
      {
         &quot;Effect&quot;: &quot;Allow&quot;,
         &quot;Principal&quot;: {
             &quot;AWS&quot;: &quot;arn:aws:iam::********:root&quot;
         },
        &quot;Action&quot;: &quot;sts:AssumeRole&quot;
      }]
    }

Role A has the following policy attached to it

&lt;!-- language: lang-json --&gt;

     {
        &quot;Version&quot;: &quot;2012-10-17&quot;,
        &quot;Statement&quot;: [
            {
                &quot;Action&quot;: &quot;sts:AssumeRole&quot;,
                &quot;Resource&quot;: &quot;arn:aws:iam::****:role/RoleB&quot;,
                &quot;Effect&quot;: &quot;Allow&quot;
           }]
     }

As a user in account A, I assumed the Role-A. Now using this temporary credential, I want to assume the Role-B and access the resource owned by account B. I have the below code

&lt;!-- language: lang-py --&gt;

    client = boto3.client(&#39;sts&#39;)
    
    firewall_role_object = client.assume_role(
        RoleArn=INTERMEDIARY_IAM_ROLE_ARN,
        RoleSessionName=str(&quot;default&quot;),
        DurationSeconds=3600)
     
    firewall_credentials = firewall_role_object[&#39;Credentials&#39;]

    firewall_client = boto3.client(
        &#39;sts&#39;,
        aws_access_key_id=firewall_credentials[&#39;AccessKeyId&#39;],
        aws_secret_access_key=firewall_credentials[&#39;SecretAccessKey&#39;],
        aws_session_token=firewall_credentials[&#39;SessionToken&#39;], )

    optimizely_role_object = firewall_client.assume_role(
        RoleArn=CUSTOMER_IAM_ROLE_ARN,
        RoleSessionName=str(&quot;default&quot;),
        DurationSeconds=3600)

    print(optimizely_role_object[&#39;Credentials&#39;])

This code works for the set of roles I got from my client but is not working for the roles I defined between two of the AWS account I have access to.

How to assume an AWS role from another AWS role?

I have a hacky way of achieving this using `boto3` (1.4.4), `pyarrow` (0.4.1) and `pandas` (0.20.3).

First, I can read a single parquet file locally like this:

    import pyarrow.parquet as pq

    path = &#39;parquet/part-r-00000-1e638be4-e31f-498a-a359-47d017a0059c.gz.parquet&#39;
    table = pq.read_table(path)
    df = table.to_pandas()

I can also read a directory of parquet files locally like this:

    import pyarrow.parquet as pq

    dataset = pq.ParquetDataset(&#39;parquet/&#39;)
    table = dataset.read()
    df = table.to_pandas()

Both work like a charm. Now I want to achieve the same remotely with files stored in a S3 bucket. I was hoping that something like this would work:

    dataset = pq.ParquetDataset(&#39;s3n://dsn/to/my/bucket&#39;)

But it does not:

`OSError: Passed non-file path: s3n://dsn/to/my/bucket`

After reading [pyarrow&#39;s documentation][1] thoroughly, this does not seem possible [at the moment][2]. So I came out with the following solution:

Reading a single file from S3 and getting a pandas dataframe:

    import io
    import boto3
    import pyarrow.parquet as pq

    buffer = io.BytesIO()
    s3 = boto3.resource(&#39;s3&#39;)
    s3_object = s3.Object(&#39;bucket-name&#39;, &#39;key/to/parquet/file.gz.parquet&#39;)
    s3_object.download_fileobj(buffer)
    table = pq.read_table(buffer)
    df = table.to_pandas()


And here my hacky, not-so-optimized, solution to create a pandas dataframe from a S3 folder path:

    import io
    import boto3
    import pandas as pd
    import pyarrow.parquet as pq

    bucket_name = &#39;bucket-name&#39;
    def download_s3_parquet_file(s3, bucket, key):
        buffer = io.BytesIO()
        s3.Object(bucket, key).download_fileobj(buffer)
        return buffer

    client = boto3.client(&#39;s3&#39;)
    s3 = boto3.resource(&#39;s3&#39;)
    objects_dict = client.list_objects_v2(Bucket=bucket_name, Prefix=&#39;my/folder/prefix&#39;)
    s3_keys = [item[&#39;Key&#39;] for item in objects_dict[&#39;Contents&#39;] if item[&#39;Key&#39;].endswith(&#39;.parquet&#39;)]
    buffers = [download_s3_parquet_file(s3, bucket_name, key) for key in s3_keys]
    dfs = [pq.read_table(buffer).to_pandas() for buffer in buffers]
    df = pd.concat(dfs, ignore_index=True)

Is there a better way to achieve this? Maybe some kind of connector for pandas using pyarrow? I would like to avoid using `pyspark`, but if there is no other solution, then I would take it.
    

  [1]: https://arrow.apache.org/docs/python/
  [2]: http://markmail.org/message/uo4kvzrt6ce6yruw

How to read a list of parquet files from S3 as a pandas dataframe using pyarrow?

I am using Python 2.7.12 in Ubuntu 16.04 LTS. I&#39;m learning how to use boto3 from the following link: https://boto3.readthedocs.io/en/latest/guide/quickstart.html#using-boto-3. My doubt is when to use resource, client, or session, and their respective functionality.

Difference in boto3 between resource, client, and session?

Following [the documentation][1], I&#39;m trying to create an update statement that will update or add if not exists only one attribute in a dynamodb table.

I&#39;m trying this

    response = table.update_item(
        Key={&#39;ReleaseNumber&#39;: &#39;1.0.179&#39;},
        UpdateExpression=&#39;SET&#39;,
        ConditionExpression=&#39;Attr(\&#39;ReleaseNumber\&#39;).eq(\&#39;1.0.179\&#39;)&#39;,
        ExpressionAttributeNames={&#39;attr1&#39;: &#39;val1&#39;},
        ExpressionAttributeValues={&#39;val1&#39;: &#39;false&#39;}
    )


The error I&#39;m getting is:

`botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the UpdateItem operation: ExpressionAttributeNames contains invalid key: Syntax error; key: &quot;attr1&quot;`

If anyone has done anything similar to what I&#39;m trying to achieve please share example.


  [1]: http://boto3.readthedocs.org/en/latest/reference/services/dynamodb.html#DynamoDB.Table.update_item

Example of update_item in dynamodb boto3

I need to update the timestamp attribute in my dynamodb table using boto3 but the attribute name &quot;timestamp&quot; is a reserved word so it&#39;s throwing an error on the SET command.

```python
table.update_item(
    Key={
        &#39;id&#39;: item_id
    },
    UpdateExpression=&#39;SET timestamp = :val1&#39;, # this is the line giving the problem
    ExpressionAttributeValues={
        &quot;:val1&quot;: new_timestamp
    }
)
```

&quot;errorMessage&quot;: &quot;An error occurred (ValidationException) when calling the UpdateItem operation: Invalid UpdateExpression: Attribute name is a reserved keyword; reserved keyword: timestamp&quot;,



All Boto3 Solutions on Gang of Coders

Total of 40 Boto3 Solutions