I found another solution

    s3 = boto3.resource(&#39;s3&#39;)
    s3.Object(&#39;my_bucket&#39;,&#39;new_file_key&#39;).copy_from(CopySource=&#39;my_bucket/old_file_key&#39;)
    s3.Object(&#39;my_bucket&#39;,&#39;old_file_key&#39;).delete()
    

You cannot rename objects in S3, so as you indicated, you need to copy it to a new name and then deleted the old one:

    client.copy_object(Bucket=&quot;BucketName&quot;, CopySource=&quot;BucketName/OriginalName&quot;, Key=&quot;NewName&quot;)
    client.delete_object(Bucket=&quot;BucketName&quot;, Key=&quot;OriginalName&quot;)

Following examples from updated Boto3 documentation for the [copy()][1] method, which also works with copy_object() and appears to be the required syntax now:

    copy_source = {&#39;Bucket&#39;: &#39;source__bucket&#39;, &#39;Key&#39;: &#39;my_folder/my_file&#39;}
    s3.copy_object(CopySource = copy_source, Bucket = &#39;dest_bucket&#39;, Key = &#39;new_folder/my_file&#39;)
    s3.delete_object(Bucket = &#39;source_bucket&#39;, Key = &#39;my_folder/my_file&#39;)

Note from documentation linked above:

&gt; CopySource (dict) -- The name of the source bucket, key name of the source object, and optional version ID of the source object. The dictionary format is: {&#39;Bucket&#39;: &#39;bucket&#39;, &#39;Key&#39;: &#39;key&#39;, &#39;VersionId&#39;: &#39;id&#39;}. Note that the VersionId key is optional and may be omitted.

  [1]: http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.copy

I&#39;ve got two lists that I want to merge into a single array and finally put it in a csv file. I&#39;m a newbie with Python arrays and I don&#39;t understand how I can avoid this error :

    def fill_csv(self, array_urls, array_dates, csv_file_path):
        result_array = []
        array_length = str(len(array_dates))
    
        # We fill the CSV file
        file = open(csv_file_path, &quot;w&quot;)
        csv_file = csv.writer(file, delimiter=&#39;;&#39;, lineterminator=&#39;\n&#39;)
    
        # We merge the two arrays in one
    
        for i in array_length:
            result_array[i][0].append(array_urls[i])
            result_array[i][1].append(array_dates[i])
            i += 1
    
        csv_file.writerows(result_array)

And got :

      File &quot;C:\Users\--\gcscan.py&quot;, line 63, in fill_csv
        result_array[i][0].append(array_urls[i])
    TypeError: list indices must be integers or slices, not str

How can my count work ?

TypeError: list indices must be integers or slices, not str

I&#39;m puzzled by how the haskell compiler sometimes infers types that are less
polymorphic than what I&#39;d expect, for example when using point-free definitions.

It seems like the issue is the &quot;monomorphism restriction&quot;, which is on by default on
older versions of the compiler.

Consider the following haskell program:

    {-# LANGUAGE MonomorphismRestriction #-}

    import Data.List(sortBy)

    plus = (+)
    plus&#39; x = (+ x)

    sort = sortBy compare

    main = do
      print $ plus&#39; 1.0 2.0
      print $ plus 1.0 2.0
      print $ sort [3, 1, 2]


If I compile this with `ghc` I obtain no erros and the output of the executable is:

    3.0
    3.0
    [1,2,3]

If I change the `main` body to:

    main = do
      print $ plus&#39; 1.0 2.0
      print $ plus (1 :: Int) 2
      print $ sort [3, 1, 2]

I get no compile time errors and the output becomes:

    3.0
    3
    [1,2,3]

as expected. However if I try to change it to:

    main = do
      print $ plus&#39; 1.0 2.0
      print $ plus (1 :: Int) 2
      print $ plus 1.0 2.0
      print $ sort [3, 1, 2]

I get a type error:

&lt;!-- language: lang-none --&gt;

    test.hs:13:16:
        No instance for (Fractional Int) arising from the literal ‘1.0’
        In the first argument of ‘plus’, namely ‘1.0’
        In the second argument of ‘($)’, namely ‘plus 1.0 2.0’
        In a stmt of a &#39;do&#39; block: print $ plus 1.0 2.0

The same happens when trying to call `sort` twice with different types:

    main = do
      print $ plus&#39; 1.0 2.0
      print $ plus 1.0 2.0
      print $ sort [3, 1, 2]
      print $ sort &quot;cba&quot;

produces the following error:

&lt;!-- language: lang-none --&gt;

    test.hs:14:17:
        No instance for (Num Char) arising from the literal ‘3’
        In the expression: 3
        In the first argument of ‘sort’, namely ‘[3, 1, 2]’
        In the second argument of ‘($)’, namely ‘sort [3, 1, 2]’


 - Why does `ghc` suddenly think that `plus` isn&#39;t polymorphic and requires an `Int` argument?
   The only reference to `Int` is in *an application* of `plus`, how can that matter
   when the definition is clearly polymorphic?
 - Why does `ghc` suddenly think that `sort` requires a `Num Char` instance?

Moreover if I try to place the function definitions into their own module, as in:

    {-# LANGUAGE MonomorphismRestriction #-}

    module TestMono where

    import Data.List(sortBy)

    plus = (+)
    plus&#39; x = (+ x)

    sort = sortBy compare

I get the following error when compiling:

&lt;!-- language: lang-none --&gt;

    TestMono.hs:10:15:
        No instance for (Ord a0) arising from a use of ‘compare’
        The type variable ‘a0’ is ambiguous
        Relevant bindings include
          sort :: [a0] -&gt; [a0] (bound at TestMono.hs:10:1)
        Note: there are several potential instances:
          instance Integral a =&gt; Ord (GHC.Real.Ratio a)
            -- Defined in ‘GHC.Real’
          instance Ord () -- Defined in ‘GHC.Classes’
          instance (Ord a, Ord b) =&gt; Ord (a, b) -- Defined in ‘GHC.Classes’
          ...plus 23 others
        In the first argument of ‘sortBy’, namely ‘compare’
        In the expression: sortBy compare
        In an equation for ‘sort’: sort = sortBy compare

 - Why isn&#39;t `ghc` able to use the polymorphic type `Ord a =&gt; [a] -&gt; [a]` for `sort`?
 - And why does `ghc` treat `plus` and `plus&#39;` differently? `plus` should have the
  polymorphic type `Num a =&gt; a -&gt; a -&gt; a` and I don&#39;t really see how this is different
  from the type of `sort` and yet only `sort` raises an error.

Last thing: if I comment the definition of `sort` the file compiles. However
if I try to load it into `ghci` and check the types I get:

&lt;!-- language: lang-none --&gt;

    *TestMono&gt; :t plus
    plus :: Integer -&gt; Integer -&gt; Integer
    *TestMono&gt; :t plus&#39;
    plus&#39; :: Num a =&gt; a -&gt; a -&gt; a

Why isn&#39;t the type for `plus` polymorphic?


----

&lt;sup&gt;
This is the canonical question about monomorphism restriction in Haskell
as discussed in [the meta question](https://meta.stackoverflow.com/questions/294053/can-we-provide-a-canonical-questionanswer-for-haskells-monomorphism-restrictio).
&lt;/sup&gt;

What is the monomorphism restriction?

I&#39;m trying to rename a file in my s3 bucket using python boto3, I couldn&#39;t clearly understand the arguments. can someone help me here?

What I&#39;m planing is to copy object to a new object, and then delete the actual object.

I found similar questions here, but I need a solution using boto3.

Boto3/S3: Renaming an object using copy_object

I'm trying to rename a file in my s3 bucket using python boto3, I couldn't clearly understand the arguments. can someone help me here?
What I'm planing is to copy object to a new object, and then delete the actual object.
I found similar questions here, but I need a solution using boto3.

What is the difference between python functions `datetime.now()` and `datetime.today()` ?

&gt; In [1]: from datetime import datetime
&gt; 
&gt; In [2]: datetime.now()
&gt;
&gt; Out[2]: datetime.datetime(2015, 9, 11, 12, 8,28, 909842)
&gt; 
&gt; In [3]: datetime.today()
&gt;
&gt; Out[3]: datetime.datetime(2015, 9, 11, 12,8, 45, 175839)


Thanks in advance.



What is the difference between python functions `datetime.now()` and `datetime.today()`?

Can anyone explain why passing a generator as the only positional argument to a function seems to have special rules?

If we have: 

    def f(*args):
        print &quot;Success!&quot;
        print args

1. This works, as expected.
        
       &gt;&gt;&gt; f(1, *[2])
       Success!
       (1, 2)

2. This does not work, as expected.

       &gt;&gt;&gt; f(*[2], 1)
         File &quot;&lt;stdin&gt;&quot;, line 1
       SyntaxError: only named arguments may follow *expression

3. This works, as expected

       &gt;&gt;&gt; f(1 for x in [1], *[2])
       Success! 
       (generator object &lt;genexpr&gt; at 0x7effe06bdcd0&gt;, 2)

4. This works, but I don&#39;t understand why. Shouldn&#39;t it fail in the same way as 2)

       &gt;&gt;&gt; f(*[2], 1 for x in [1])
       Success!
       (generator object &lt;genexpr&gt; at 0x7effe06bdcd0&gt;, 2)

Generator as function argument

I am trying to get unique IDs for my Django objects.  In Django 1.8 they have the UUIDField.  I am unsure how to use this field in order to generate unique IDs for each object in my model.

Here is what I have for the UUIDField

    import uuid
    from django.db import models

    class MyUUIDModel(models.Model):
        id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)

    class Person(models.Model):
        ...
        unique_id = MyUUIDModel()

I can reproduce the id for the UUID model, but everytime I do I get the exact same id. For Example: 

    person = Person.objects.get(some_field = some_thing)
    id = person.unique_id.id

id then gives me the same id every time.  What is wrong, how do I fix this?  



How to use UUID

Trying to create a super user for my database:

    manage.py createsuperuser

Getting a sad recursive message:

*Superuser creation skipped due to not running in a TTY. You can run `manage.py createsuperuser` in your project to create one manually.*

Seriously Django? Seriously?


The only information I found for this was the one listed above but it didn&#39;t work:
https://stackoverflow.com/questions/26980003/unable-to-create-superuser-in-django-due-to-not-working-in-tty

And this other one here, which is basically the same:
https://stackoverflow.com/questions/14059573/cant-create-super-user-django



Not able to create super user with Django manage.py

I am working on Caffe framework and using PyCaffe interface. I am using a Python script obtained from converting the IPython Notebook **00-classification.ipynb** for testing the classification by a trained model for ImageNet. But any **get_ipython()** statement in the script is giving the following error:

    $ python python/my_test_imagenet.py 
    Traceback (most recent call last):
      File &quot;python/my_test_imagenet.py&quot;, line 23, in &lt;module&gt;
        get_ipython().magic(u&#39;matplotlib inline&#39;)

In the script, I&#39;m importing the following:

    import numpy as np
    import matplotlib.pyplot as plt
    
    get_ipython().magic(u&#39;matplotlib inline&#39;)
    
    # Make sure that caffe is on the python path:
    caffe_root = &#39;/path/to/caffe/&#39;
    import sys
    sys.path.insert(0, caffe_root + &#39;python&#39;)
    
    import caffe
    
    plt.rcParams[&#39;figure.figsize&#39;] = (10, 10)
    plt.rcParams[&#39;image.interpolation&#39;] = &#39;nearest&#39;
    plt.rcParams[&#39;image.cmap&#39;] = &#39;gray&#39;
    
    import os
    
    # ... Rest of the code...

Can someone please help me to resolve this error?


NameError: name &#39;get_ipython&#39; is not defined

I am trying to create a table to store invoice line items in DynamoDB. Let&#39;s say the item is defined by `CompanyCode`, `InvoiceNumber` and `LineItemId`, amount and other line item details. 

A unique item is defined by the combination of the first 3 attributes. Any 2 of those attributes can be same for the different items. What should I select as the Hash Attribute and the Range Attribute? 

3 fields composite primary key (unique item) in Dynamodb

Is it better to have multiple s3 buckets per category of uploads or one bucket with sub folders OR a linked s3 bucket? I know for sure there will be more user-images than there will be profille-pics and that there is a 5TB limit per bucket and 100 buckets per account. I&#39;m doing this using aws boto library and https://github.com/amol-/depot

Which is the  structure my folders in which of the following manner?


	/app_bucket
	    /profile-pic-folder
	    /user-images-folder

	OR

	profile-pic-bucket
	user-images-bucket


    OR
 

    /app_bucket_1
    /app_bucket_2


The last one implies that its really a 10TB bucket where a new bucket is created when the files within bucket_1 exceeds 5TB. But all uploads will be read as if in one bucket. Or is there a better way of doing what I&#39;m trying to do? Many thanks!

I&#39;m not sure if this is correct... 100 buckets per account? 

https://www.reddit.com/r/aws/comments/28vbjs/requesting_increase_in_number_of_s3_buckets/

Is it better to have multiple s3 buckets or one bucket with sub folders?

I&#39;m using **AWS Lambda** to scan data from a **DynamoDB** table. This is what I get in return:

    {
      &quot;videos&quot;: [
        {
          &quot;file&quot;: {
            &quot;S&quot;: &quot;file1.mp4&quot;
          },
          &quot;id&quot;: {
            &quot;S&quot;: &quot;1&quot;
          },
          &quot;canvas&quot;: {
            &quot;S&quot;: &quot;This is Canvas1&quot;
          }
        },
        {
          &quot;file&quot;: {
            &quot;S&quot;: &quot;main.mp4&quot;
          },
          &quot;id&quot;: {
            &quot;S&quot;: &quot;0&quot;
          },
          &quot;canvas&quot;: {
            &quot;S&quot;: &quot;this is a canvas&quot;
          }
        }
      ]
    }

My front-end application is using **Ember Data Rest Adapter** which does not accepts such response. Is there any way I can get normal JSON format? There is this NPM module called `dynamodb-marshaler` to convert DynamoDB data to normal JSON. I&#39;m looking for a **native solution** if possible. 


Formatting DynamoDB data to normal JSON in AWS Lambda

I&#39;ve noticed some Elastic Beanstalk is drawing about $20 to $30 every month on my AWS account and it turns out it is an empty S3 bucket.

I tried deleting it but it returns &quot;Access Denied&quot; error.

[![Access Denied][1]][1]


I also tried to use life cycle to remove it, but it also didn&#39;t work. Here is how I set up the life cycle:

[![Life Cycle][2]][2]

I set it to remove 1 day after creation

[![enter image description here][3]][3]


  [1]: http://i.stack.imgur.com/M5a5i.png
  [2]: http://i.stack.imgur.com/F8C3T.png
  [3]: http://i.stack.imgur.com/4u7k8.png

How to fix &#39;Access Denied&#39; while deleting empty S3 Elastic Beanstalk?

I&#39;ve recently started using the new Amazon Elasticsearch Service and I can&#39;t seem to figure out the access policy I need so that I can only access the services from my EC2 instances that have a specific IAM role assigned to them.

Here&#39;s an example of the access policy I currently have assigned for the ES domain:

    {
      &quot;Version&quot;: &quot;2012-10-17&quot;,
      &quot;Statement&quot;: [
        {
          &quot;Sid&quot;: &quot;&quot;,
          &quot;Effect&quot;: &quot;Allow&quot;,
          &quot;Principal&quot;: {
            &quot;AWS&quot;: [
              &quot;arn:aws:iam::[ACCOUNT_ID]:role/my_es_role&quot;,
            ]
          },
          &quot;Action&quot;: &quot;es:*&quot;,
          &quot;Resource&quot;: &quot;arn:aws:es:us-east-1:[ACCOUNT_ID]:domain/[ES_DOMAIN]/*&quot;
        }
      ]
    }

But as I said, this doesn&#39;t work.  I log into the EC2 instance (which has the `my_es_role` role attached to it) and attempt to run a simple curl call on the &quot;https://*.es.amazonaws.com&quot; end point, I get the following error:

&gt; {&quot;Message&quot;:&quot;User: anonymous is not authorized to perform: es:ESHttpGet on resource: arn:aws:es:us-east-1:[ACCOUNT_ID]:domain/[ES_DOMAIN]/“}

Does anyone know what I have to change in the access policy in order for this to work?

Proper access policy for Amazon Elastic Search Cluster

I&#39;m aware of the existence of https://wiki.apache.org/hadoop/AmazonS3 and the following words:

&gt; S3 Native FileSystem (URI scheme: s3n) A native filesystem for reading and writing regular files on S3. The advantage of this filesystem is that you can access files on S3 that were written with other tools. Conversely, other tools can access files written using Hadoop. The disadvantage is the 5GB limit on file size imposed by S3.
&gt;
&gt; S3A (URI scheme: s3a) A successor to the S3 Native, s3n fs, the S3a: system uses Amazon&#39;s libraries to interact with S3. This allows S3a to support larger files (no more 5GB limit), higher performance operations and more. The filesystem is intended to be a replacement for/successor to S3 Native: all objects accessible from s3n:// URLs should also be accessible from s3a simply by replacing the URL schema.
&gt;
&gt; S3 Block FileSystem (URI scheme: s3) A block-based filesystem backed by S3. Files are stored as blocks, just like they are in HDFS. This permits efficient implementation of renames. This filesystem requires you to dedicate a bucket for the filesystem - you should not use an existing bucket containing files, or write other files to the same bucket. The files stored by this filesystem can be larger than 5GB, but they are not interoperable with other S3 tools.


Why a letter change on the URI could make such difference? For example 

    val data = sc.textFile(&quot;s3n://bucket-name/key&quot;)

to 

    val data = sc.textFile(&quot;s3a://bucket-name/key&quot;)

What is the technical difference underlying this change? Are there any good articles that I can read on this?





Technically what is the difference between s3n, s3a and s3?

I would like to know if a key exists in boto3. I can loop the bucket contents and check the key if it matches.

But that seems longer and an overkill. Boto3 official docs explicitly state how to do this. 

May be I am missing the obvious. Can anybody point me how I can achieve this.

check if a key exists in a bucket in s3 using boto3

I&#39;ve just deployed a static website to Amazon S3, which can currently be viewed here: http://www.rdegges.com.s3-website-us-east-1.amazonaws.com/

If you click any of the article links, you&#39;ll notice the following error:

[![S3 Error][1]][1]

S3 is complaining the file doesn&#39;t exist. Now, here&#39;s what&#39;s weird about this -- I&#39;m using CloudFront on my domain. So when you click that article link, it&#39;s sending the request to CloudFront which then tries to fetch the file back from the S3 bucket.

However, if you visit that same URL from S3 directly, eg: http://www.rdegges.com.s3-website-us-east-1.amazonaws.com/2015/building-a-heroku-addon-planning/ the page will load just fine.

It appears that something is getting lost in translation here.

Anyone got a suggestion of what I can do to fix my settings?

  [1]: http://i.stack.imgur.com/c715j.png

CloudFront + S3 Website: &quot;The specified key does not exist&quot; when an implicit index document should be displayed

I am trying to figure how to do proper error handling with boto3.

I am trying to create an IAM user:

    def create_user(username, iam_conn):
        try:
            user = iam_conn.create_user(UserName=username)
            return user
        except Exception as e:
            return e

When the call to create_user succeeds, I get a neat object that contains the http status code of the API call and the data of the newly created user.

Example:

    {&#39;ResponseMetadata&#39;: 
          {&#39;HTTPStatusCode&#39;: 200, 
           &#39;RequestId&#39;: &#39;omitted&#39;
          },
     u&#39;User&#39;: {u&#39;Arn&#39;: &#39;arn:aws:iam::omitted:user/omitted&#39;,
               u&#39;CreateDate&#39;: datetime.datetime(2015, 10, 11, 17, 13, 5, 882000, tzinfo=tzutc()),
               u&#39;Path&#39;: &#39;/&#39;,
               u&#39;UserId&#39;: &#39;omitted&#39;,
               u&#39;UserName&#39;: &#39;omitted&#39;
              }
    }

 This works great. But when this fails (like if the user already exists), I just get an object of type botocore.exceptions.ClientError with only text to tell me what went wrong.

Example:
ClientError(&#39;An error occurred (EntityAlreadyExists) when calling the CreateUser operation: User with name omitted already exists.&#39;,)

This (AFAIK) makes error handling very hard because I can&#39;t just switch on the resulting http status code (409 for user already exists according to the AWS API docs for IAM). This makes me think that I must be doing something the wrong way. The optimal way would be for boto3 to never throw exceptions, but juts always return an object that reflects how the API call went.

Can anyone enlighten me on this issue or point me in the right direction?

How to handle errors with boto3?

When I simply run the following code, I always gets this error.

   

    s3 = boto3.resource(&#39;s3&#39;)
    bucket_name = &quot;python-sdk-sample-%s&quot; % uuid.uuid4()
    print(&quot;Creating new bucket with name:&quot;, bucket_name)
    s3.create_bucket(Bucket=bucket_name)


I have saved my credential file in 

 `C:\Users\myname\.aws\credentials`, from where Boto should read my credentials.

Is my setting wrong?

Here is the output from `boto3.set_stream_logger(&#39;botocore&#39;, level=&#39;DEBUG&#39;)`.


    2015-10-24 14:22:28,761 botocore.credentials [DEBUG] Skipping environment variable credential check because profile name was explicitly set.
    2015-10-24 14:22:28,761 botocore.credentials [DEBUG] Looking for credentials via: env
    2015-10-24 14:22:28,773 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
    2015-10-24 14:22:28,774 botocore.credentials [DEBUG] Looking for credentials via: config-file
    2015-10-24 14:22:28,774 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file
    2015-10-24 14:22:28,774 botocore.credentials [DEBUG] Looking for credentials via: boto-config
    2015-10-24 14:22:28,774 botocore.credentials [DEBUG] Looking for credentials via: iam-role

Boto3 Error: botocore.exceptions.NoCredentialsError: Unable to locate credentials

I need to get the [account-id][1] of the &#39;current user&#39; in a [`boto3`][2] script. Up to now my best solution is to parse the current user [arn][3]:

    &gt;&gt;&gt; import boto3
    &gt;&gt;&gt; account_id = boto3.resource(&#39;iam&#39;).CurrentUser().arn.split(&#39;:&#39;)[4]

but I was wondering if there is a more &#39;lightweight&#39; approach. In fact

    &gt;&gt;&gt; timeit.timeit(&quot;boto3.resource(&#39;iam&#39;).CurrentUser().arn&quot;,
    ... &#39;import boto3&#39;, number=10)
    4.8895583080002325

and I actually do not need the [`CurrentUser`][4] resource in my script.


  [1]: http://docs.aws.amazon.com/general/latest/gr/acct-identifiers.html
  [2]: http://boto3.readthedocs.org/en/latest/index.html
  [3]: http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#genref-arns
  [4]: https://boto3.readthedocs.org/en/latest/reference/services/iam.html#currentuser

getting the current user account-id in boto3

I am using the Boto 3 python library, and want to connect to AWS CloudFront.
I need to specify the correct AWS Profile (AWS Credentials), but looking at the official documentation, I see no way to specify it.

I am initializing the client using the code:
`client = boto3.client(&#39;cloudfront&#39;)`

However, this results in it using the default profile to connect.
I couldn&#39;t find a method where I can specify which profile to use.


How to choose an AWS profile when using boto3 to connect to CloudFront

I have trouble installing boto3 inside a virtual environment. 

I have done what the document says. First I activated virtual environment. then I did a:

    Sudo pip install boto3

Now I enter python

    &gt;&gt; import boto3
    ImportError: No module named boto3

But if I import boto, it works

    &gt;&gt; import boto
    &gt;&gt; boto.Version
    &#39;2.38.0&#39;

Why does it install boto 2.38 when I installed boto3.
I tried closing the terminal and re-opened it.
Should I restart the Ubuntu machine?

Content Type	Original Author	Original Content on Stackoverflow
Question	MikA	View Question on Stackoverflow
Solution 1 - Python	MikA	View Answer on Stackoverflow
Solution 2 - Python	Boto User	View Answer on Stackoverflow
Solution 3 - Python	jpgard	View Answer on Stackoverflow

Boto3/S3: Renaming an object using copy_object

Python Problem Overview

Python Solutions

Solution 1 - Python

Solution 2 - Python

Solution 3 - Python

What is the monomorphism restriction?

TypeError: list indices must be integers or slices, not str

Attributions