I had to add a new profile to this file `~/.aws/config`. Example of the file with 2 profiles:

    [profile eb-cli]
    aws_access_key_id = XXXXXXXXXXXXX
    aws_secret_access_key = XXXXXXXXXXXX

    [profile eb-cli2]
    aws_access_key_id = XXXXXXXXXX
    aws_secret_access_key = XXXXXXXXXXXX

Also make sure to update the profile value in your application .elasticbeanstalk/config.yml



Bpn&#39;s answer is correct just add the next step to avoid spending time as I did..

After updating the `~/.aws/config` file, just run:

    eb init --profile &lt;profilename&gt;
in order to switch between the account.

(in this case profilename = eb-cli2)

  

Using `export AWS_EB_PROFILE=&quot;default&quot;` I was able to use my key from my default profile from `~/.aws/credentials`.

For more details: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3-configuration.html

The `awsebcli` tool automatically creates a file named `~/.aws/config` and you can see your keys in there under the `[profile eb-cli]` category. Just change them there.

    [profile eb-cli]
    aws_access_key_id = ...
    aws_secret_access_key = ...

Specify the profiles in ~/.aws/credentials:  

    [myprofile2] 
    aws_access_key_id = ...
    aws_secret_access_key = ...  

Like other has indicated. Then use it in eb cli like this:  

    eb init --profile myprofile2 

Basically append &quot;--profile myprofile2&quot; in all eb commands:  

    eb deploy --profile myprofile2 
    eb open --profile myprofile2



I am looking for a solution to be able to log additional data when executing code on Apache Spark Nodes that could help investigate later some issues that might appear during execution. Trying to use a traditional solution like for example `com.typesafe.scalalogging.LazyLogging` fails because the log instance cannot be serialized on a distributed environment like Apache Spark.

I&#39;ve investigated this problem and for now the solution that I found was to use the `org.apache.spark.Logging` trait like this :

    class SparkExample with Logging {
      val someRDD = ...
      someRDD.map {
        rddElement =&gt; logInfo(s&quot;$rddElement will be processed.&quot;)
        doSomething(rddElement)
      }
    }

However it looks like the Logging trait is not a permanent solution for Apache Spark because it&#39;s marked as `@DeveloperApi` and the class documentation mentions: 
&gt;This will likely be changed or removed in future releases.

I am wondering - are they any known logging solution that I can use and will allow me to log data when the RDDs are executed on Apache Spark nodes ?

**@Later Edit** : Some of the comments from below suggest to use Log4J. I&#39;ve tried using Log4J but I&#39;m still having issues when using logger from a Scala class (and not a Scala object).
Here is my full code :

    import org.apache.log4j.Logger
    import org.apache.spark._

    object Main {
     def main(args: Array[String]) {
      new LoggingTestWithRDD().doTest()
     }
    }

    class LoggingTestWithRDD extends Serializable {

      val log = Logger.getLogger(getClass.getName)

      def doTest(): Unit = {
       val conf = new SparkConf().setMaster(&quot;local[4]&quot;).setAppName(&quot;LogTest&quot;)
       val spark = new SparkContext(conf)

       val someRdd = spark.parallelize(List(1, 2, 3))
       someRdd.map {
         element =&gt;
           log.info(s&quot;$element will be processed&quot;)
           element + 1
        }
       spark.stop()
     }

   }

The exception that I&#39;m seeing is :
&gt; Exception in thread &quot;main&quot; org.apache.spark.SparkException: Task not serializable -&gt; Caused by: java.io.NotSerializableException: org.apache.log4j.Logger

Apache Spark logging within Scala

**2020 update**, about 11 years after the question was posted and later closed, preventing newer answers.

**Almost everything written here is obsolete. Once upon a time sqlite was limited to the memory capacity or to 2 GB of storage (32 bits) or other popular numbers... well, that was a long time ago.**

**[Official limitations are listed here][1]. Practically sqlite is likely to work as long as there is storage available**. It works well with dataset larger than memory, it was originally created when memory was thin and it was a very important point from the start.

There is absolutely no issue with storing 100 GB of data. It could probably store a TB just fine but eventually that&#39;s the point where you need to question whether SQLite is the best tool for the job and you probably want features from a full fledged database (remote clients, concurrent writes, read-only replicas, sharding, etc...).

---

Original:

I know that sqlite doesn&#39;t perform well with extremely large database files even when they are supported (there used to be a comment on the sqlite website stating that if you need file sizes above 1GB you may want to consider using an enterprise rdbms. Can&#39;t find it anymore, might be related to an older version of sqlite).

However, for my purposes I&#39;d like to get an idea of how bad it really is before I consider other solutions.

I&#39;m talking about sqlite data files in the multi-gigabyte range, from 2GB onwards.
Anyone have any experience with this? Any tips/ideas?


  [1]: https://www.sqlite.org/limits.html

What are the performance characteristics of sqlite with very large database files?

I deployed an app using elastic beanstalk to my personal AWS account..Now I want to change the AWS credentials so the eb cli knows to deploy to a different account. But it does not ask me for the AWS keys when I type &quot;eb init&quot;. Where do I specify this?

How to change the AWS account using the Elastic Beanstalk CLI

<p>I deployed an app using elastic beanstalk to my personal AWS account..Now I want to change the AWS credentials so the eb cli knows to deploy to a different account. But it does not ask me for the AWS keys when I type "eb init". Where do I specify this?</p>


I&#39;m building a monitoring tool based on AWS Lambda. Given a set of metrics, the Lambdas should be able to send SMS using [Twilio][1] API. To be able to use the API, Twilio provide an account SID and an auth token. 

How and where should I store these secrets?

I&#39;m currently thinking to use [AWS KMS][2] but there might be other better solutions.


  [1]: https://www.twilio.com
  [2]: http://aws.amazon.com/kms/

AWS Lambda: How to store secret to external API?

I&#39;m trying to do a &quot;hello world&quot; with new [boto3][1] client for AWS.

The use-case I have is fairly simple: get object from S3 and save it to the file.

In boto 2.X I would do it like this:

    import boto
    key = boto.connect_s3().get_bucket(&#39;foo&#39;).get_key(&#39;foo&#39;)
    key.get_contents_to_filename(&#39;/tmp/foo&#39;)

In boto 3 . I can&#39;t find a clean way to do the same thing, so I&#39;m manually iterating over the &quot;Streaming&quot; object:

    import boto3
    key = boto3.resource(&#39;s3&#39;).Object(&#39;fooo&#39;, &#39;docker/my-image.tar.gz&#39;).get()
    with open(&#39;/tmp/my-image.tar.gz&#39;, &#39;w&#39;) as f:
        chunk = key[&#39;Body&#39;].read(1024*8)
        while chunk:
            f.write(chunk)
            chunk = key[&#39;Body&#39;].read(1024*8)

or

    import boto3
    key = boto3.resource(&#39;s3&#39;).Object(&#39;fooo&#39;, &#39;docker/my-image.tar.gz&#39;).get()
    with open(&#39;/tmp/my-image.tar.gz&#39;, &#39;w&#39;) as f:
        for chunk in iter(lambda: key[&#39;Body&#39;].read(4096), b&#39;&#39;):
            f.write(chunk)


And it works fine. I was wondering is there any &quot;native&quot; boto3 function that will do the same task?
 

  [1]: https://github.com/boto/boto3

How to save S3 object to a file using boto3

I am trying to access an environment variable that I have defined in the AWS Beanstalk configuration. I need to access it within a config file in `.ebextensions` _or_ in a file that is copied in place in a config file. I have tried the following:

    container_commands:
      update_nginx_config:
        command: &quot;cp .ebextensions/files/nginx/nginx.conf /etc/nginx/nginx.conf&quot;

And in my `nginx.conf` file, I have tried to access `$MYVAR`, `${MYVAR}` and `{$MYVAR}`, some of which was suggested [here][1] and [here][2] (the latter being directly within a config file).

    files:
      &quot;/etc/nginx/nginx.conf&quot; :
        mode: &quot;000644&quot;
        owner: root
        group: root
        content: |
          $MYVAR ${MYVAR} {$MYVAR}

This does not work either. In all cases, the variable names are just output such as `$MYVAR`, so Beanstalk does not recognize my variables. I found the below in the [AWS documentation][3] about `container_commands`:

&gt; They also have access to environment variables such as your AWS
&gt; security credentials.

This is great, but it does not say _how_.

How can I access an environment variable with ebextensions, be it within a config file itself or in a separate file that is copied in place?

Thank you in advance!


  [1]: https://forums.aws.amazon.com/thread.jspa?messageID=529504
  [2]: https://stackoverflow.com/questions/16585898/referencing-env-variables-from-elastic-beanstalk-ebextensions-config-files
  [3]: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html#customize-containers-format-container_commands

Accessing environment variables in AWS Beanstalk ebextensions

As per the title of this question, what are the practical differences between AWS EFS, EBS and S3?

My understanding of each:

- S3 is a storage facility accessible any where
- EBS is a device you can mount onto EC2
- EFS is a file system you can mount onto EC2

So why would I use EBS over EFS? Seem like they have the same use cases but minor semantic differences? Although EFS is replicated across AZs where as EBS is just a mounted device. I guess my understanding of EBS is lacking hence I&#39;m unable to distinguish. 

Why choose S3 over EFS? They both store files, scale and are replicated. I guess with S3 you have to use the SDK where as with EFS being a file system you can you standard I/O methods from your programming language of choice to create files. But is that the only real difference?

AWS EFS vs EBS vs S3 (differences &amp; when to use?)

I&#39;ve developed a Docker based application comprised of multiple microservices. It has to consume Amazon SQS messages and processes them. At first I wanted to use AWS Elastic Beanstalk, but then I fell over the EC2 Container Service. Now I don&#39;t know which one to choose.

As of now, Elastic Beanstalk supports Multi-Container-Environments. That&#39;s great because every microservice has its own application server inside a docker container. The next problem is scaling:

I don&#39;t know how the scaling mechanism works. For example: I have 5 docker containers in my Elastic Beanstalk Environment. Now only the fifth docker container is under heavy load, because it has a huge amount of SQS messages to process, the other four are nearly idle, because they don&#39;t need much CPU or maybe don&#39;t have a lot of SQS messages. Let&#39;s assume the 5th container runs a JBoss application server. As far as i know, the server only can consume a limited amount of parallel requests even if there is enough CPU/memory available.

If the JBoss Docker container isn&#39;t able to handle the amount of requests, but there is enough CPU/memory available, of course I want to automatically start a second Docker/JBoss container on the same instance. But what happens, if I don&#39;t have enough CPU/memory? Of course I want to spin on a second instance, which is configurable through a auto-scaling group in EB. Now a second instance spins up, but every container except of the 5th is nearly idle, of course I don&#39;t want them to spawn 4 unnecessary at the second instance too, which would be a waste of resources. Only the 5th should spawn and the others should scale like the 5th scale based on configurable parameters like e.g.: CPU/memory/SQS. 

I don&#39;t exactly know if Amazon ECS is doing that, or if it&#39;s possible at all, but I really can&#39;t find any source on the internet about this topic, which is in general said, scaling based on instances/containers.

Should I use AWS Elastic Beanstalk or the Amazon EC2 Container Service (ECS) to scale Docker containers?

I get the following error when deploying to EB:

&gt; ERROR: You cannot have more than 500 Application Versions. Either
&gt; remove some Application Versions or request a limit increase.

I went manually and deleted some versions.
I don&#39;t want deploys to fail because of this limit. 
Is there a way in Elastic Beanstalk to auto-evict unused versions? 

cannot deploy - ERROR: You cannot have more than 500 Application Versions

I want to shut down the app servers while I upgrade the database.  

Is there a way to pause or stop the app servers without terminating/destroying the environment?

Can I just go to the Elastic Beanstalk load balancer and change that temporarily without any issues or consequences to the Elastic Beanstalk configurations or the way it manages its servers?

Pause an Elastic Beanstalk app environment?

I&#39;ve noticed some Elastic Beanstalk is drawing about $20 to $30 every month on my AWS account and it turns out it is an empty S3 bucket.

I tried deleting it but it returns &quot;Access Denied&quot; error.

[![Access Denied][1]][1]


I also tried to use life cycle to remove it, but it also didn&#39;t work. Here is how I set up the life cycle:

[![Life Cycle][2]][2]

I set it to remove 1 day after creation

[![enter image description here][3]][3]


  [1]: http://i.stack.imgur.com/M5a5i.png
  [2]: http://i.stack.imgur.com/F8C3T.png
  [3]: http://i.stack.imgur.com/4u7k8.png

Content Type	Original Author	Original Content on Stackoverflow
Question	bpn	View Question on Stackoverflow
Solution 1 - Amazon Web-Services	bpn	View Answer on Stackoverflow
Solution 2 - Amazon Web-Services	Developeder	View Answer on Stackoverflow
Solution 3 - Amazon Web-Services	Joël	View Answer on Stackoverflow
Solution 4 - Amazon Web-Services	Dimitris	View Answer on Stackoverflow
Solution 5 - Amazon Web-Services	us_david	View Answer on Stackoverflow

How to change the AWS account using the Elastic Beanstalk CLI

Amazon Web-Services Problem Overview

Amazon Web-Services Solutions

Solution 1 - Amazon Web-Services

Solution 2 - Amazon Web-Services

Solution 3 - Amazon Web-Services

Solution 4 - Amazon Web-Services

Solution 5 - Amazon Web-Services

What are the performance characteristics of sqlite with very large database files?

Apache Spark logging within Scala

Attributions