How can I see what&#39;s inside a bucket in S3 with `boto3`? (i.e. do an `&quot;ls&quot;`)?

Doing the following:

    import boto3
    s3 = boto3.resource(&#39;s3&#39;)
    my_bucket = s3.Bucket(&#39;some/path/&#39;)

returns:

    s3.Bucket(name=&#39;some/path/&#39;)

How do I see its contents?

Listing contents of a bucket with boto3

I&#39;m trying to get django to upload static files to S3, but istead I&#39;m getting a 403 forbidden error, and I&#39;m not sure why.

**Full Stacktrace:**

    Traceback (most recent call last):
      File &quot;manage.py&quot;, line 14, in &lt;module&gt;
        execute_manager(settings)
      File &quot;/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/__init__.py&quot;, line 438, in execute_manager
        utility.execute()
      File &quot;/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/__init__.py&quot;, line 379, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File &quot;/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/base.py&quot;, line 191, in run_from_argv
        self.execute(*args, **options.__dict__)
      File &quot;/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/base.py&quot;, line 220, in execute
        output = self.handle(*args, **options)
      File &quot;/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/base.py&quot;, line 351, in handle
        return self.handle_noargs(**options)
      File &quot;/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py&quot;, line 89, in handle_noargs
        self.copy_file(path, prefixed_path, storage, **options)
      File &quot;/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py&quot;, line 184, in copy_file
        if not self.delete_file(path, prefixed_path, source_storage, **options):
      File &quot;/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py&quot;, line 115, in delete_file
        if self.storage.exists(prefixed_path):
      File &quot;/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/storages/backends/s3boto.py&quot;, line 209, in exists
        return k.exists()
      File &quot;/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py&quot;, line 391, in exists
        return bool(self.bucket.lookup(self.name))
      File &quot;/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py&quot;, line 143, in lookup
        return self.get_key(key_name, headers=headers)
      File &quot;/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py&quot;, line 208, in get_key
        response.status, response.reason, &#39;&#39;)
    boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden

**Contents of settings.py:**

    import os
    DIRNAME = os.path.dirname(__file__)
    # Django settings for DoneBox project.
    
    DEBUG = True
    TEMPLATE_DEBUG = DEBUG
    
    ADMINS = (
        # (&#39;Your Name&#39;, &#39;your_email@example.com&#39;),
    )
    
    MANAGERS = ADMINS
    
    DATABASES = {
        &#39;default&#39;: {
            &#39;ENGINE&#39;: &#39;django.db.backends.sqlite3&#39;, # Add &#39;postgresql_psycopg2&#39;, &#39;postgresql&#39;, &#39;mysql&#39;, &#39;sqlite3&#39; or &#39;oracle&#39;.
            &#39;NAME&#39;: os.path.join(DIRNAME, &quot;box.sqlite&quot;),                      # Or path to database file if using sqlite3.
            &#39;USER&#39;: &#39;&#39;,                      # Not used with sqlite3.
            &#39;PASSWORD&#39;: &#39;&#39;,                  # Not used with sqlite3.
            &#39;HOST&#39;: &#39;&#39;,                      # Set to empty string for localhost. Not used with sqlite3.
            &#39;PORT&#39;: &#39;&#39;,                      # Set to empty string for default. Not used with sqlite3.
        }
    }
    
    # Local time zone for this installation. Choices can be found here:
    # http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
    # although not all choices may be available on all operating systems.
    # On Unix systems, a value of None will cause Django to use the same
    # timezone as the operating system.
    # If running in a Windows environment this must be set to the same as your
    # system time zone.
    TIME_ZONE = &#39;America/Denver&#39;
    
    # Language code for this installation. All choices can be found here:
    # http://www.i18nguy.com/unicode/language-identifiers.html
    LANGUAGE_CODE = &#39;en-us&#39;
    
    SITE_ID = 1
    
    # If you set this to False, Django will make some optimizations so as not
    # to load the internationalization machinery.
    USE_I18N = True
    
    # If you set this to False, Django will not format dates, numbers and
    # calendars according to the current locale
    USE_L10N = True
    
    # Absolute filesystem path to the directory that will hold user-uploaded files.
    # Example: &quot;/home/media/media.lawrence.com/media/&quot;
    MEDIA_ROOT = &#39;&#39;
    
    # URL that handles the media served from MEDIA_ROOT. Make sure to use a
    # trailing slash.
    # Examples: &quot;http://media.lawrence.com/media/&quot;, &quot;http://example.com/media/&quot;
    MEDIA_URL = &quot;d1eyn4cjl5vzx0.cloudfront.net&quot;
    
    # Absolute path to the directory static files should be collected to.
    # Don&#39;t put anything in this directory yourself; store your static files
    # in apps&#39; &quot;static/&quot; subdirectories and in STATICFILES_DIRS.
    # Example: &quot;/home/media/media.lawrence.com/static/&quot;
    STATIC_ROOT = os.path.join(DIRNAME, &quot;static&quot;)
    
    # URL prefix for static files.
    # Example: &quot;http://media.lawrence.com/static/&quot;
    STATIC_URL = &quot;d280kzug7l5rug.cloudfront.net&quot;
    
    # URL prefix for admin static files -- CSS, JavaScript and images.
    # Make sure to use a trailing slash.
    # Examples: &quot;http://foo.com/static/admin/&quot;, &quot;/static/admin/&quot;.
    ADMIN_MEDIA_PREFIX = &#39;/static/admin/&#39;
    
    # Additional locations of static files
    STATICFILES_DIRS = (
        # Put strings here, like &quot;/home/html/static&quot; or &quot;C:/www/django/static&quot;.
        # Always use forward slashes, even on Windows.
        # Don&#39;t forget to use absolute paths, not relative paths.
        os.path.join(DIRNAME, &quot;main&quot;, &quot;static&quot;),
    )
    
    # List of finder classes that know how to find static files in
    # various locations.
    STATICFILES_FINDERS = (
        &#39;django.contrib.staticfiles.finders.FileSystemFinder&#39;,
        &#39;django.contrib.staticfiles.finders.AppDirectoriesFinder&#39;,
        &#39;django.contrib.staticfiles.finders.DefaultStorageFinder&#39;,
    )
    
    # Make this unique, and don&#39;t share it with anybody.
    SECRET_KEY = &#39;&lt;snip&gt;&#39;
    
    # List of callables that know how to import templates from various sources.
    TEMPLATE_LOADERS = (
        &#39;django.template.loaders.filesystem.Loader&#39;,
        &#39;django.template.loaders.app_directories.Loader&#39;,
        &#39;django.template.loaders.eggs.Loader&#39;,
    )
    
    MIDDLEWARE_CLASSES = (
        &#39;django.middleware.common.CommonMiddleware&#39;,
        &#39;django.contrib.sessions.middleware.SessionMiddleware&#39;,
        &#39;django.middleware.csrf.CsrfViewMiddleware&#39;,
        &#39;django.contrib.auth.middleware.AuthenticationMiddleware&#39;,
        &#39;django.contrib.messages.middleware.MessageMiddleware&#39;,
    )
    
    ROOT_URLCONF = &#39;DoneBox.urls&#39;
    
    TEMPLATE_DIRS = (
        # Put strings here, like &quot;/home/html/django_templates&quot; or &quot;C:/www/django/templates&quot;.
        # Always use forward slashes, even on Windows.
        # Don&#39;t forget to use absolute paths, not relative paths.
        os.path.join(DIRNAME, &quot;main&quot;, &quot;templates&quot;),
        os.path.join(DIRNAME, &quot;templates&quot;),
        os.path.join(DIRNAME, &quot;basic&quot;, &quot;blog&quot;, &quot;templates&quot;),
    )
    
    INSTALLED_APPS = (
        &#39;django.contrib.auth&#39;,
        &#39;django.contrib.contenttypes&#39;,
        &#39;django.contrib.sessions&#39;,
        &#39;django.contrib.sites&#39;,
        &#39;django.contrib.messages&#39;,
        &#39;django.contrib.staticfiles&#39;,
        &#39;django.contrib.sitemaps&#39;,
        # Uncomment the next line to enable the admin:
        &#39;django.contrib.admin&#39;,
        # Uncomment the next line to enable admin documentation:
        &#39;storages&#39;,
        &#39;django.contrib.admindocs&#39;,
        &#39;main&#39;,
        &#39;contacts&#39;,
        &#39;piston&#39;,
        &#39;registration&#39;,
    #    &#39;contact_form&#39;,
        &#39;basic&#39;,
        &#39;basic.blog&#39;,
    )
    
    # A sample logging configuration. The only tangible logging
    # performed by this configuration is to send an email to
    # the site admins on every HTTP 500 error.
    # See http://docs.djangoproject.com/en/dev/topics/logging for
    # more details on how to customize your logging configuration.
    LOGGING = {
        &#39;version&#39;: 1,
        &#39;disable_existing_loggers&#39;: False,
        &#39;handlers&#39;: {
            &#39;mail_admins&#39;: {
                &#39;level&#39;: &#39;ERROR&#39;,
                &#39;class&#39;: &#39;django.utils.log.AdminEmailHandler&#39;
            }
        },
        &#39;loggers&#39;: {
            &#39;django.request&#39;: {
                &#39;handlers&#39;: [&#39;mail_admins&#39;],
                &#39;level&#39;: &#39;DEBUG&#39;,
                &#39;propagate&#39;: True,
            },
            &#39;django.db.backends&#39;: {
                &#39;handlers&#39;: [&#39;mail_admins&#39;],
                &#39;level&#39;: &#39;DEBUG&#39;,
                &#39;propagate&#39;: True,
            }
        }
    }
    
    DEFAULT_FILE_STORAGE = &#39;storages.backends.s3boto.S3BotoStorage&#39;
    AWS_ACCESS_KEY_ID = &#39;&lt;snip&gt;&#39;
    AWS_SECRET_ACCESS_KEY = &#39;&lt;snip&gt;&#39;
    STATICFILES_STORAGE = &#39;storages.backends.s3boto.S3BotoStorage&#39;
    AWS_STORAGE_BUCKET_NAME = &quot;donebox-static&quot;
    STATIC_FILES_BUCKET = &quot;donebox-static&quot;
    MEDIA_FILES_BUCKET = &quot;donebox-media&quot;
    ACCOUNT_ACTIVATION_DAYS = 7
    
    EMAIL_HOST = &quot;email-smtp.us-east-1.amazonaws.com&quot;
    EMAIL_HOST_USER = &#39;&lt;snip&gt;&#39;
    EMAIL_HOST_PASSWORD = &#39;&lt;snip&gt;&#39;
    EMAIL_PORT = 587
    EMAIL_USE_TLS = True
    TEMPLATE_CONTEXT_PROCESSORS = (
        &quot;django.contrib.auth.context_processors.auth&quot;,
         &quot;django.core.context_processors.debug&quot;,
         &quot;django.core.context_processors.i18n&quot;,
         &quot;django.core.context_processors.media&quot;,
         &quot;django.core.context_processors.static&quot;,
         &quot;django.contrib.messages.context_processors.messages&quot;,
         &quot;DoneBox.main.context_processors_PandC&quot;,
         )

**Contents of requirements.pip:**

    django==1.3
    django-storages==1.1.4
    django-registration==0.8
    django-piston==0.2.3
    django-tagging==0.3.1
    django-extensions==0.8
    BeautifulSoup==3.2.1
    boto==2.4.1
    mysql-python==1.2.3
    tweepy==1.9
    feedparser==5.1.2
    pycrypto==2.6

A google search for this exception doesn&#39;t turn up anything interesting. I suspect I mis-configured things, although I&#39;m not sure. Could someone point me in the right direction? Thank you for your time and consideration.

boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden

Problem:

I&#39;m trying to get the region of the authenticated user from boto3.

Use case:

I&#39;m working on adding cache to https://github.com/pmazurek/aws-fuzzy-finder. I would prefer to cache the result on *per-region* basis.

This package uses boto to get user authentication data (keys and the region). The problem is the region is never passed explicitly by the user, its being taken from one of the many murky places that boto reads so I don&#39;t really have a way of getting it. 

I have tried searching through boto3 api and googling, but couldn&#39;t find anything like a `get_region` or `get_user_data` method. Is it possible?

How to get the region of the current user from boto?

I am saving a key to a bucket with:

    	key = bucket.new_key(fileName)
		key.set_contents_from_string(base64.b64decode(data))
		key.set_metadata(&#39;Content-Type&#39;, &#39;image/jpeg&#39;)
		key.set_acl(&#39;public-read&#39;)

After the save is successful, how can I access the URL of the newly created file? 

Using Amazon s3 boto library, how can I get the URL of a saved key?

So that I am able to work with it within my python scripts?

How do I install boto?

I have my $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY environment variables set properly, and I run this code:

    import boto
    conn = boto.connect_s3()

and get this error:

    boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. [&#39;HmacAuthV1Handler&#39;]

What&#39;s happening? I don&#39;t know where to start debugging.


---

It seems boto isn&#39;t taking the values from my environment variables. If I pass in the key id and secret key as arguments to the connection constructor, this works fine.

Why are no Amazon S3 authentication handlers ready?

Is there a way to get an item depending on a field that is not the hashkey?

Example

My Table Users: `id (HashKey), name, email`

And I want to retrieve the user having email as &#39;test@mail.com&#39;

How this can be done?

I try this with boto:

    user = users.get_item(email=&#39;john.doe@gmail.com&#39;)

I get the following error: 

    &#39;The provided key element does not match the schema&#39;



DynamoDB : The provided key element does not match the schema

I am thinking about redeploying my static website to Amazon S3. I need to automate the deployment so I was looking for an API for such tasks. I&#39;m a bit confused over the different options.

**Question**: What is the difference between s3cmd, the Python library boto and AWS CLI?

Difference between s3cmd, boto and AWS CLI

I have the next code:

    users_table = Table(users_table_name, connection=Core.aws_dynamodb_connection)
    users_table.put_item(data={
      &quot;login&quot;: login,
      &quot;password&quot;: hashlib.sha256(password.encode(&quot;utf-8&quot;)).hexdigest(),
      &quot;profile&quot;: profile,
      &quot;registration_date&quot;: datetime.now() # PROBLEM IS HERE
    })

But when I run it, it fails with error:

&gt; TypeError: Unsupported type &quot;&lt; type &#39;datetime.datetime&#39; &gt;&quot; for value &quot;2015-01-12 05:02:57.053131&quot;

I&#39;ve tried a lot of ways, but it seems that it isn&#39;t possible to save `datetime` to DynamoDB. Btw it works fine in MongoDB.

Is there any solution?

Is it possible to save datetime to DynamoDB?

I am trying to figure how to do proper error handling with boto3.

I am trying to create an IAM user:

    def create_user(username, iam_conn):
        try:
            user = iam_conn.create_user(UserName=username)
            return user
        except Exception as e:
            return e

When the call to create_user succeeds, I get a neat object that contains the http status code of the API call and the data of the newly created user.

Example:

    {&#39;ResponseMetadata&#39;: 
          {&#39;HTTPStatusCode&#39;: 200, 
           &#39;RequestId&#39;: &#39;omitted&#39;
          },
     u&#39;User&#39;: {u&#39;Arn&#39;: &#39;arn:aws:iam::omitted:user/omitted&#39;,
               u&#39;CreateDate&#39;: datetime.datetime(2015, 10, 11, 17, 13, 5, 882000, tzinfo=tzutc()),
               u&#39;Path&#39;: &#39;/&#39;,
               u&#39;UserId&#39;: &#39;omitted&#39;,
               u&#39;UserName&#39;: &#39;omitted&#39;
              }
    }

 This works great. But when this fails (like if the user already exists), I just get an object of type botocore.exceptions.ClientError with only text to tell me what went wrong.

Example:
ClientError(&#39;An error occurred (EntityAlreadyExists) when calling the CreateUser operation: User with name omitted already exists.&#39;,)

This (AFAIK) makes error handling very hard because I can&#39;t just switch on the resulting http status code (409 for user already exists according to the AWS API docs for IAM). This makes me think that I must be doing something the wrong way. The optimal way would be for boto3 to never throw exceptions, but juts always return an object that reflects how the API call went.

Can anyone enlighten me on this issue or point me in the right direction?

How to handle errors with boto3?

I&#39;m aware that with Boto 2 it&#39;s possible to open an S3 object as a string with: [`get_contents_as_string()`](http://boto.readthedocs.org/en/latest/ref/file.html?highlight=contents%20string#boto.file.key.Key.get_contents_as_string)


Is there an equivalent function in boto3 ? 

Open S3 object as a string with Boto3

When I simply run the following code, I always gets this error.

   

    s3 = boto3.resource(&#39;s3&#39;)
    bucket_name = &quot;python-sdk-sample-%s&quot; % uuid.uuid4()
    print(&quot;Creating new bucket with name:&quot;, bucket_name)
    s3.create_bucket(Bucket=bucket_name)


I have saved my credential file in 

 `C:\Users\myname\.aws\credentials`, from where Boto should read my credentials.

Is my setting wrong?

Here is the output from `boto3.set_stream_logger(&#39;botocore&#39;, level=&#39;DEBUG&#39;)`.


    2015-10-24 14:22:28,761 botocore.credentials [DEBUG] Skipping environment variable credential check because profile name was explicitly set.
    2015-10-24 14:22:28,761 botocore.credentials [DEBUG] Looking for credentials via: env
    2015-10-24 14:22:28,773 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
    2015-10-24 14:22:28,774 botocore.credentials [DEBUG] Looking for credentials via: config-file
    2015-10-24 14:22:28,774 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file
    2015-10-24 14:22:28,774 botocore.credentials [DEBUG] Looking for credentials via: boto-config
    2015-10-24 14:22:28,774 botocore.credentials [DEBUG] Looking for credentials via: iam-role

Boto3 Error: botocore.exceptions.NoCredentialsError: Unable to locate credentials

In boto 2, you can write to an S3 object using these methods:

- [Key.set_contents_from_string()](https://boto.readthedocs.io/en/latest/ref/s3.html#boto.s3.key.Key.set_contents_from_string)
- [Key.set_contents_from_file()](https://boto.readthedocs.io/en/latest/ref/s3.html#boto.s3.key.Key.set_contents_from_file)
- [Key.set_contents_from_filename()](https://boto.readthedocs.io/en/latest/ref/s3.html#boto.s3.key.Key.set_contents_from_filename)
- [Key.set_contents_from_stream()](https://boto.readthedocs.io/en/latest/ref/s3.html#boto.s3.key.Key.set_contents_from_stream)

Is there a boto 3 equivalent? What is the boto3 method for saving data to an object stored on S3?



How to write a file or data to an S3 object using boto3

I&#39;m trying to do a &quot;hello world&quot; with new [boto3][1] client for AWS.

The use-case I have is fairly simple: get object from S3 and save it to the file.

In boto 2.X I would do it like this:

    import boto
    key = boto.connect_s3().get_bucket(&#39;foo&#39;).get_key(&#39;foo&#39;)
    key.get_contents_to_filename(&#39;/tmp/foo&#39;)

In boto 3 . I can&#39;t find a clean way to do the same thing, so I&#39;m manually iterating over the &quot;Streaming&quot; object:

    import boto3
    key = boto3.resource(&#39;s3&#39;).Object(&#39;fooo&#39;, &#39;docker/my-image.tar.gz&#39;).get()
    with open(&#39;/tmp/my-image.tar.gz&#39;, &#39;w&#39;) as f:
        chunk = key[&#39;Body&#39;].read(1024*8)
        while chunk:
            f.write(chunk)
            chunk = key[&#39;Body&#39;].read(1024*8)

or

    import boto3
    key = boto3.resource(&#39;s3&#39;).Object(&#39;fooo&#39;, &#39;docker/my-image.tar.gz&#39;).get()
    with open(&#39;/tmp/my-image.tar.gz&#39;, &#39;w&#39;) as f:
        for chunk in iter(lambda: key[&#39;Body&#39;].read(4096), b&#39;&#39;):
            f.write(chunk)


And it works fine. I was wondering is there any &quot;native&quot; boto3 function that will do the same task?
 

  [1]: https://github.com/boto/boto3

How to save S3 object to a file using boto3

I&#39;m new to AWS using Python and I&#39;m trying to learn the boto API however I noticed that there are two major versions/packages for Python.  That would be boto and boto3.

What is the difference between the AWS boto and boto3 libraries?

What is the difference between the AWS boto and boto3

How can I create a folder under a bucket using `boto` library for Amazon s3?  

I followed the manual, and created the keys with permission, metadata etc, but no where in the boto&#39;s documentation it describes how to create folders under a bucket, or create a folder under folders in bucket.

Amazon S3 boto - how to create a folder?

I want to copy a file in s3 bucket using python.

Ex : I have bucket name = test. And in the bucket, I have 2 folders name &quot;dump&quot; &amp; &quot;input&quot;. Now I want to copy a file from local directory to S3 &quot;dump&quot; folder using python... Can anyone help me?


How to upload a file to directory in S3 bucket using boto

I created a folder in s3 named &quot;test&quot; and I pushed &quot;test_1.jpg&quot;, &quot;test_2.jpg&quot; into &quot;test&quot;.

How can I use boto to delete folder &quot;test&quot;?

Amazon S3 boto - how to delete folder?

I&#39;m trying to mock a singluar method from the boto3 s3 client object to throw an exception. But I need all other methods for this class to work as normal.

This is so I can test a singular Exception test when and error occurs performing a [upload_part_copy][1]

**1st Attempt**

    import boto3
    from mock import patch

    with patch(&#39;botocore.client.S3.upload_part_copy&#39;, side_effect=Exception(&#39;Error Uploading&#39;)) as mock:
        client = boto3.client(&#39;s3&#39;)
        # Should return actual result
        o = client.get_object(Bucket=&#39;my-bucket&#39;, Key=&#39;my-key&#39;)
        # Should return mocked exception
        e = client.upload_part_copy()

However this gives the following error:

    ImportError: No module named S3

**2nd Attempt**

After looking at the botocore.client.py source code I found that it is doing something clever and the method `upload_part_copy` does not exist. I found that it seems to call `BaseClient._make_api_call` instead so I tried to mock that

    import boto3
    from mock import patch

    with patch(&#39;botocore.client.BaseClient._make_api_call&#39;, side_effect=Exception(&#39;Error Uploading&#39;)) as mock:
        client = boto3.client(&#39;s3&#39;)
        # Should return actual result
        o = client.get_object(Bucket=&#39;my-bucket&#39;, Key=&#39;my-key&#39;)
        # Should return mocked exception
        e = client.upload_part_copy()

This throws an exception... but on the `get_object` which I want to avoid.

Any ideas about how I can only throw the exception on the `upload_part_copy` method?

  [1]: http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.upload_part_copy

Mocking boto3 S3 client method Python

I am able to upload an image file using:

    s3 = session.resource(&#39;s3&#39;)
    bucket = s3.Bucket(S3_BUCKET)
    bucket.upload_file(file, key)

However, I want to make the file public too. I tried looking up for some functions to set ACL for the file but seems like boto3 have changes their API and removed some functions. Is there a way to do it in the latest release of boto3?

How to upload a file to S3 and make it public using boto3?

Is it better to have multiple s3 buckets per category of uploads or one bucket with sub folders OR a linked s3 bucket? I know for sure there will be more user-images than there will be profille-pics and that there is a 5TB limit per bucket and 100 buckets per account. I&#39;m doing this using aws boto library and https://github.com/amol-/depot

Which is the  structure my folders in which of the following manner?


	/app_bucket
	    /profile-pic-folder
	    /user-images-folder

	OR

	profile-pic-bucket
	user-images-bucket


    OR
 

    /app_bucket_1
    /app_bucket_2


The last one implies that its really a 10TB bucket where a new bucket is created when the files within bucket_1 exceeds 5TB. But all uploads will be read as if in one bucket. Or is there a better way of doing what I&#39;m trying to do? Many thanks!

I&#39;m not sure if this is correct... 100 buckets per account? 

https://www.reddit.com/r/aws/comments/28vbjs/requesting_increase_in_number_of_s3_buckets/

Is it better to have multiple s3 buckets or one bucket with sub folders?

How do you rename a S3 key in a bucket with boto?

Amazon S3 boto: How do you rename a file in a bucket?

I&#39;m getting the error &quot;The role defined for the function cannot be assumed by Lambda&quot; when I&#39;m trying to create a lambda function with create-function command.

&gt;    aws lambda create-function  
&gt; --region us-west-2  
&gt; --function-name HelloPython  
&gt; --zip-file fileb://hello_python.zip  
&gt; --role arn:aws:iam::my-acc-account-id:role/default   
&gt; --handler hello_python.my_handler  
&gt; --runtime python2.7  
&gt; --timeout 15  
&gt; --memory-size 512

The role defined for the function cannot be assumed by Lambda

I am using the Boto library to talk to AWS. I want to disable logging. (Or redirect to /dev/null or other file). I can&#39;t find an obvious way to do this. I tried this, but that doesn&#39;t seem to help:

    import boto
    boto.set_file_logger(&#39;boto&#39;, &#39;logs/boto.log&#39;)


This says it is possible, http://developer.amazonwebservices.com/connect/thread.jspa?messageID=52727&amp;#52727 but as far as I know the documentation doesn&#39;t tell how.

Disable boto logging without modifying the boto files

There must be an easy way to get the file size (key size) without pulling over a whole file.  I can see it in the Properties of the AWS S3 browser.  And I think I can get it off the &quot;Content-length&quot; header of a &quot;HEAD&quot; request.  But I&#39;m not connecting the dots about how to do this with boto.  Extra kudos if you post a link to some more comprehensive examples than are in the [standard boto docs][1].

EDIT: So the following seems to do the trick (though from looking at source code I&#39;m not completely sure.):

    bk = conn.get_bucket(&#39;my_bucket_name&#39;)
    ky = boto.s3.key.Key(bk)
    ky.open_read()  ## This sends a GET request. 
    print ky.size

For now I&#39;ll leave the question open for comments, better solutions, or pointers to examples.

  [1]: http://boto.cloudhackers.com/

How do I get the file / key size in boto S3?

How can I upload a file to [Google Cloud Storage](https://cloud.google.com/storage/) from Python 3? Eventually Python 2, if it&#39;s infeasible from Python 3.

I&#39;ve looked and looked, but haven&#39;t found a solution that actually works. I tried [boto](https://github.com/boto/boto), but when I try to generate the necessary .boto file through `gsutil config -e`, it keeps saying that I need to configure authentication through `gcloud auth login`. However, I have done the latter a number of times, without it helping.


How to upload a file to Google Cloud Storage on Python 3?

I have S3 access only to a specific directory in an S3 bucket.

For example, with the `s3cmd` command if I try to list the whole bucket:

        $ s3cmd ls s3://bucket-name

I get an error: `Access to bucket &#39;my-bucket-url&#39; was denied`

But if I try access a specific directory in the bucket, I can see the contents:

        $ s3cmd ls s3://bucket-name/dir-in-bucket

Now I want to connect to the S3 bucket with python boto. Similary with:

        bucket = conn.get_bucket(&#39;bucket-name&#39;)

I get an error: `boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden`

But if I try:

        bucket = conn.get_bucket(&#39;bucket-name/dir-in-bucket&#39;)

The script stalls for about 10 seconds, and prints out an error afterwards. Bellow is the full trace. Any idea how to proceed with this?

Note question is about the boto version 2 module, not boto3.

	Traceback (most recent call last):
	  File &quot;test_s3.py&quot;, line 7, in &lt;module&gt;
		bucket = conn.get_bucket(&#39;bucket-name/dir-name&#39;)
	  File &quot;/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py&quot;, line 471, in get_bucket
		return self.head_bucket(bucket_name, headers=headers)
	  File &quot;/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py&quot;, line 490, in head_bucket
		response = self.make_request(&#39;HEAD&#39;, bucket_name, headers=headers)
	  File &quot;/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py&quot;, line 633, in make_request
		retry_handler=retry_handler
	  File &quot;/usr/local/lib/python2.7/dist-packages/boto/connection.py&quot;, line 1046, in make_request
		retry_handler=retry_handler)
	  File &quot;/usr/local/lib/python2.7/dist-packages/boto/connection.py&quot;, line 922, in _mexe
		request.body, request.headers)
	  File &quot;/usr/lib/python2.7/httplib.py&quot;, line 958, in request
		self._send_request(method, url, body, headers)
	  File &quot;/usr/lib/python2.7/httplib.py&quot;, line 992, in _send_request
		self.endheaders(body)
	  File &quot;/usr/lib/python2.7/httplib.py&quot;, line 954, in endheaders
		self._send_output(message_body)
	  File &quot;/usr/lib/python2.7/httplib.py&quot;, line 814, in _send_output
		self.send(msg)
	  File &quot;/usr/lib/python2.7/httplib.py&quot;, line 776, in send
		self.connect()
	  File &quot;/usr/lib/python2.7/httplib.py&quot;, line 1157, in connect
		self.timeout, self.source_address)
	  File &quot;/usr/lib/python2.7/socket.py&quot;, line 553, in create_connection
		for res in getaddrinfo(host, port, 0, SOCK_STREAM):
	socket.gaierror: [Errno -2] Name or service not known

Python boto, list contents of specific dir in bucket

I need to create a CSV and upload it to an S3 bucket. Since I&#39;m creating the file on the fly, it would be better if I could write it directly to S3 bucket as it is being created rather than writing the whole file locally, and then uploading the file at the end.

Is there a way to do this? My project is in Python and I&#39;m fairly new to the language. Here is what I tried so far:

    import csv
    import csv
    import io
    import boto
    from boto.s3.key import Key


    conn = boto.connect_s3()
    bucket = conn.get_bucket(&#39;dev-vs&#39;)
    k = Key(bucket)
    k.key = &#39;foo/foobar&#39;

    fieldnames = [&#39;first_name&#39;, &#39;last_name&#39;]
    writer = csv.DictWriter(io.StringIO(), fieldnames=fieldnames)
    k.set_contents_from_stream(writer.writeheader())

I received this error: BotoClientError: s3 does not support chunked transfer

**UPDATE: I found a way to write directly to S3, but I can&#39;t find a way to clear the buffer without actually deleting the lines I already wrote. So, for example:**

    conn = boto.connect_s3()
    bucket = conn.get_bucket(&#39;dev-vs&#39;)
    k = Key(bucket)
    k.key = &#39;foo/foobar&#39;

    testDict = [{
        &quot;fieldA&quot;: &quot;8&quot;,
        &quot;fieldB&quot;: None,
        &quot;fieldC&quot;: &quot;888888888888&quot;},
        {
        &quot;fieldA&quot;: &quot;9&quot;,
        &quot;fieldB&quot;: None,
        &quot;fieldC&quot;: &quot;99999999999&quot;}]

    f = io.StringIO()
    fieldnames = [&#39;fieldA&#39;, &#39;fieldB&#39;, &#39;fieldC&#39;]
    writer = csv.DictWriter(f, fieldnames=fieldnames)
    writer.writeheader()
    k.set_contents_from_string(f.getvalue())

    for row in testDict:
        writer.writerow(row)
        k.set_contents_from_string(f.getvalue())

    f.close()

Writes 3 lines to the file, however I&#39;m unable to release memory to write a big file. If I add:

    f.seek(0)
    f.truncate(0)
to the loop, then only the last line of the file is written. Is there any way to release resources without deleting lines from the file?

All Boto Solutions on Gang of Coders

Total of 28 Boto Solutions