Initial setup of terraform backend using terraform

I'm just getting started with terraform and I'd like to be able to use AWS S3 as my backend for storing the state of my projects.

terraform {
    backend "s3" {
      bucket = "tfstate"
      key = "app-state"
      region = "us-east-1"
    }
}

I feel like it is sensible to setup my S3 bucket, IAM groups and polices for the backend storage infrastructure with terraform as well.

If I setup my backend state before I apply my initial terraform infrastructure, it reasonably complains that the backend bucket is not yet created. So, my question becomes, how do I setup my terraform backend with terraform, while keeping my state for the backend tracked by terraform. Seems like a nested dolls problem.

I have some thoughts about how to script around this, for example, checking to see if the bucket exists or some state has been set, then bootstrapping terraform and finally copying the terraform tfstate up to s3 from the local file system after the first run. But before going down this laborious path, I thought I'd make sure I wasn't missing something obvious.

To set this up using terraform remote state, I usually have a separate folder called remote-state within my dev and prod terraform folder.

The following main.tf file will set up your remote state for what you posted:

provider "aws" {
  region = "us-east-1"
}

resource "aws_s3_bucket" "terraform_state" {
  bucket = "tfstate"

  versioning {
    enabled = true
  }

  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_dynamodb_table" "terraform_state_lock" {
  name           = "app-state"
  read_capacity  = 1
  write_capacity = 1
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

Then get into this folder using cd remote-state, and run terraform init && terraform apply - this should only need to be run once. You might add something to bucket and dynamodb table name to separate your different environments.

Building on the great contribution from Austin Davis, here is a variation that I use which includes a requirement for data encryption:

provider "aws" {
  region = "us-east-1"
}

resource "aws_s3_bucket" "terraform_state" {
  bucket = "tfstate"

  versioning {
    enabled = true
  }

  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_dynamodb_table" "terraform_state_lock" {
  name           = "app-state"
  read_capacity  = 1
  write_capacity = 1
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

resource "aws_s3_bucket_policy" "terraform_state" {
  bucket = "${aws_s3_bucket.terraform_state.id}"
  policy =<<EOF
{
  "Version": "2012-10-17",
  "Id": "RequireEncryption",
   "Statement": [
    {
      "Sid": "RequireEncryptedTransport",
      "Effect": "Deny",
      "Action": ["s3:*"],
      "Resource": ["arn:aws:s3:::${aws_s3_bucket.terraform_state.bucket}/*"],
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "false"
        }
      },
      "Principal": "*"
    },
    {
      "Sid": "RequireEncryptedStorage",
      "Effect": "Deny",
      "Action": ["s3:PutObject"],
      "Resource": ["arn:aws:s3:::${aws_s3_bucket.terraform_state.bucket}/*"],
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption": "AES256"
        }
      },
      "Principal": "*"
    }
  ]
}
EOF
}

As you've discovered, you can't use terraform to build the components terraform needs in the first place.

While I understand the inclination to have terraform "track everything", it is very difficult, and more headache than it's worth.

I generally handle this situation by creating a simple bootstrap shell script. It creates things like:

1. The s3 bucket for state storage
2. Adds versioning to said bucket
3. a terraform IAM user and group with certain policies I'll need for terraform builds

While you should only need to run this once (technically), I find that when I'm developing a new system, I spin up and tear things down repeatedly. So having those steps in one script makes that a lot simpler.

I generally build the script to be idempotent. This way, you can run it multiple times without concern that you're creating duplicate buckets, users, etc

I created a terraform module with a few bootstrap commands/instructions to solve this:

https://github.com/samstav/terraform-aws-backend

There are detailed instructions in the README, but the gist is:

# conf.tf

module "backend" {
  source         = "github.com/samstav/terraform-aws-backend"
  backend_bucket = "terraform-state-bucket"
}

Then, in your shell (make sure you haven't written your terraform {} block yet):

terraform get -update
terraform init -backend=false
terraform plan -out=backend.plan -target=module.backend
terraform apply backend.plan

Now write your terraform {} block:

# conf.tf

terraform {
  backend "s3" {
    bucket         = "terraform-state-bucket"
    key            = "states/terraform.tfstate"
    dynamodb_table = "terraform-lock"
  }
}

And then you can re-init:

terraform init -reconfigure

Setting up a Terraform backend leveraging an AWS s3 bucket is relatively easy.

First, create a bucket in the region of your choice (eu-west-1 for the example), named terraform-backend-store (remember to choose a unique name.)

To do so, open your terminal and run the following command, assuming that you have properly set up the AWS CLI (otherwise, follow the instructions at the official documentation):

aws s3api create-bucket --bucket terraform-backend-store \
    --region eu-west-1 \
    --create-bucket-configuration \
    LocationConstraint=eu-west-1
# Output:
{
    "Location": "http://terraform-backend-store.s3.amazonaws.com/"
}

The command should be self-explanatory; to learn more check the documentation here.

Once the bucket is in place, it needs a proper configuration for security and reliability. For a bucket that holds the Terraform state, it’s common-sense enabling the server-side encryption. Keeping it simple, try first AES256 method (although I recommend to use KMS and implement a proper key rotation):

aws s3api put-bucket-encryption \
    --bucket terraform-backend-store \
    --server-side-encryption-configuration={\"Rules\":[{\"ApplyServerSideEncryptionByDefault\":{\"SSEAlgorithm\":\"AES256\"}}]}
# Output: expect none when the command is executed successfully

Next, it’s crucial restricting the access to the bucket; create an unprivileged IAM user as follows:

aws iam create-user --user-name terraform-deployer
# Output:
{
    "User": {
        "UserName": "terraform-deployer",
        "Path": "/",
        "CreateDate": "2019-01-27T03:20:41.270Z",
        "UserId": "AIDAIOSFODNN7EXAMPLE",
        "Arn": "arn:aws:iam::123456789012:user/terraform-deployer"
    }
}

Take note of the Arn from the command’s output (it looks like: “Arn”: “arn:aws:iam::123456789012:user/terraform-deployer”).

To correctly interact with the s3 service and DynamoDB at a later stage to implement the locking, our IAM user must hold a sufficient set of permissions. It is recommended to have severe restrictions in place for production environments, though, for the sake of simplicity, start assigning AmazonS3FullAccess and AmazonDynamoDBFullAccess:

aws iam attach-user-policy --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess --user-name terraform-deployer
# Output: expect none when the command execution is successful

aws iam attach-user-policy --policy-arn arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess --user-name terraform-deployer
# Output: expect none when the command execution is successful

The freshly created IAM user must be enabled to execute the required actions against your s3 bucket. You can do this by creating and applying the right policy, as follows:

cat <<-EOF >> policy.json
{
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:user/terraform-deployer"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::terraform-remote-store"
        }
    ]
}
EOF

This basic policy file grants the principal with arn “arn:aws:iam::123456789012:user/terraform-deployer”, to execute all the available actions (“Action”: “s3:*") against the bucket with arn “arn:aws:s3:::terraform-remote-store”. Again, in production is desired to force way stricter policies. For reference, have a look at the AWS Policy Generator.

Back to the terminal and run the command as shown below, to enforce the policy in your bucket:

aws s3api put-bucket-policy --bucket terraform-remote-store --policy file://policy.json
# Output: none

As the last step, enable the bucket’s versioning:

aws s3api put-bucket-versioning --bucket terraform-remote-store --versioning-configuration Status=Enabled

It allows saving different versions of the infrastructure’s state and rollback easily to a previous stage without struggling.

The AWS s3 bucket is ready, time to integrate it with Terraform. Listed below, is the minimal configuration required to set up this remote backend:

# terraform.tf

provider "aws" {
  region                  = "${var.aws_region}"
  shared_credentials_file = "~/.aws/credentials"
  profile                 = "default"
}

terraform {  
    backend "s3" {
        bucket  = "terraform-remote-store"
        encrypt = true
        key     = "terraform.tfstate"    
        region  = "eu-west-1"  
    }
}

# the rest of your configuration and resources to deploy

Once in place, terraform must be initialized (again). terraform init The remote backend is ready for a ride, test it.

What about locking? Storing the state remotely brings a pitfall, especially when working in scenarios where several tasks, jobs, and team members have access to it. Under these circumstances, the risk of multiple concurrent attempts to make changes to the state is high. Here comes to help the lock, a feature that prevents opening the state file while already in use.

You can implement the lock creating an AWS DynamoDB Table, used by terraform to set and unset the locks. Provision the resource using terraform itself:

# create-dynamodb-lock-table.tf
resource "aws_dynamodb_table" "dynamodb-terraform-state-lock" {
  name           = "terraform-state-lock-dynamo"
  hash_key       = "LockID"
  read_capacity  = 20
  write_capacity = 20
attribute {
    name = "LockID"
    type = "S"
  }
tags {
    Name = "DynamoDB Terraform State Lock Table"
  }
}

and deploy it as shown: terraform plan -out "planfile" && terraform apply -input=false -auto-approve "planfile"

Once the command execution is completed, the locking mechanism must be added to your backend configuration as follow:

# terraform.tf

provider "aws" {
  region                  = "${var.aws_region}"
  shared_credentials_file = "~/.aws/credentials"
  profile                 = "default"
}

terraform {  
    backend "s3" {
        bucket         = "terraform-remote-store"
        encrypt        = true
        key            = "terraform.tfstate"    
        region         = "eu-west-1"
        dynamodb_table = "terraform-state-lock-dynamo"
    }
}

# the rest of your configuration and resources to deploy

All done. Remember to run again terraform init and enjoy your remote backend.

What I usually do is start without remote backend for creating initial infrastructure as you said , S3 , IAM roles and other essential stuff. Once I have that I just add backend configuration and run terraform init to migrate to S3.

It's not the best case but in most cases I don't rebuild my entire environment everyday so this semi automated approach is good enough. I also separate next "layers" (VPC, Subnets, IGW, NAT ,etc) of infrastructure to different states.

What I have been doing to address this is that, You can comment out the "backend" block for the initial run, and do a selected terraform apply on only the state bucket and any related resources(like bucket policies).

#  backend "s3" {
#    bucket         = "foo-bar-state-bucket"
#    key            = "core-terraform.tfstate"
#    region         = "eu-west-1"
#  }
#}
provider "aws" {
    region = "eu-west-1"
    profile = "terraform-iam-user"
    shared_credentials_file = "~/.aws/credentials"
  }

terraform apply --target aws_s3_bucket.foobar-terraform --target aws_s3_bucket_policy.foobar-terraform

This will provision your s3 state bucket, and will store .tfstate file locally in your working directory.

Later, Uncomment the "backend" block and reconfigure the backend terraform init --reconfigure , which will prompt you to copy your locally present .tfstate file, (tracking state of your backend s3 bucket) to the remote backend which is now available to be used by terraform for any subsequent runs.

Prompt for copying exisitng state to remote backend

There are some great answers here & I'd like to offer an alternative to managing your back end state;

1. Set up a Terraform Cloud Account (it's free for up to 5 users).
2. Create a workspace for your organization (Version control workflow is typical)
3. Select your VCS such as github or bitbucket (where you store your terraform plans and modules)
4. Terraform Cloud will give you the instructions needed for your new OAuth Connection
5. Once that's setup you'll have the option to set up an SSH keypair which is typically not needed & you can click the Skip & Finish button

Once your terraform cloud account is set up & connected to your VCS repos where you store your terraform plans & modules... Add your terraform module repos in terraform cloud, by clicking on the Registry tab. You will need to ensure that your terraform modules are versioned / tagged & follow proper naming convention. If you have a terraform module that creates a load balancer in AWS, you would name the terraform module repository (in github for example), like this: terraform-aws-loadbalancer. As long as it starts with terraform-aws- you're good. Then you add a version tag to it such as 1.0.0

So let's say you create a terraform plan that points to that load balancer module, this is how you point your backend config to terraform cloud & to the load balancer module:

backend-state.tf contents:

terraform {
  backend "remote" {
    hostname     = "app.terraform.io"
    organization = "YOUR-TERRAFORM-CLOUD-ORG"
    workspaces {
    # name = ""   ## For single workspace jobs
    # prefix = "" ## for multiple workspaces
    # you can use name instead of prefix
    prefix = "terraform-plan-name-"
    }
  }
}

terraform plan main.tf contents;

module "aws_alb" {
  source  = "app.terraform.io/YOUR-TERRAFORM-CLOUD-ORG/loadbalancer/aws"
  version = "1.0.0"
  
  name = "load-balancer-test"
  security_groups = [module.aws_sg.id]
  load_balancer_type = "application"
  internal = false
  subnets = [data.aws_subnet.public.id]
  idle_timeout = 1200
  # access_logs_enabled = true
  # access_logs_s3bucket = "BUCKET-NAME"
  tags = local.tags
}

Locally from your terminal (using Mac OSX as an example);

terraform init
# if you're using name instead of prefix in your backend set 
# up, no need to run terraform workspace cmd
terraform workspace new test
terraform plan
terraform apply

You'll see the apply happening in terraform cloud under your workspaces with this name: terraform-plan-name-test "test" is appended to your workspace prefix name which is defined in your backend-state.tf above. You end up with a GUI / Console full of your terraform plans within your workspace, the same way you can see your Cloudformation Stacks in AWS. I find devops that are used to Cloudformation and transitioning to Terraform, like this set up.

One advantage is, within Terraform Cloud you can easily set it up so that a plan (stack build) is triggered with a git commit or merge to the master branch.

1 reference: https://www.terraform.io/docs/language/settings/backends/remote.html#basic-configuration

The way I have overcome this issue is by creating the project remote state in the first init plan apply cycle and initializing the remote state in the second init plan apply cycle.

# first init plan apply cycle 
# Configure the AWS Provider
# https://www.terraform.io/docs/providers/aws/index.html
provider "aws" {
  version = "~> 2.0"
  region  = "us-east-1"
}

resource "aws_s3_bucket" "terraform_remote_state" {
  bucket = "terraform-remote-state"
  acl    = "private"

  tags = {
    Name        = "terraform-remote-state"
    Environment = "Dev"
  }
}

# add this sniped and execute the 
# the second init plan apply cycle
# https://www.terraform.io/docs/backends/types/s3.html

terraform {
  backend "s3" {
    bucket = "terraform-remote-state"
    key    = "path/to/my/key"
    region = "us-east-1"
  }
}

I would Highly recommend using Terragrunt to keep your Terraform code manageable and DRY (the Don't repeat yourself principle).

Terragrunt has many capabilities - for your specific case I would suggest following the Keep your remote state configuration DRY section.

I'll add a short and simplified summary below.

---

Problems with managing remote state with Terraform

Let's say you have the following Terraform infrastructure:

├── backend-app
│   ├── main.tf
│   └── other_resources.tf
│   └── variables.tf
├── frontend-app
│   ├── main.tf
│   └── other_resources.tf
│   └── variables.tf
├── mysql
│   ├── main.tf
│   └── other_resources.tf
│   └── variables.tf
└── mongo
    ├── main.tf
    └── other_resources.tf
    └── variables.tf

Each app is a terraform module that you'll want to store its Terraform state in a remote backend.

Without Terragrunt you'll have to write the backend configuration block for each application in order to save the current state in a remote state storage:

terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "frontend-app/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "my-lock-table"
  }
}

Managing a few modules like in the above example its not a burden to add this file for each one of them - but it won't last for real world scenarious.

Wouldn't it be better if we could do some kind of inheritance (like in Object oriented programming)?

This is made easy with Terragrunt.

---

Terragrunt to the rescue

Back to the modules structure.
With Terragrunt we just need add add a root terragrunt.hcl with all the configurations and for each module you add a child terragrunt.hcl which contains only on statement:

├── terragrunt.hcl       #<---- Root
├── backend-app
│   ├── main.tf
│   └── other_resources.tf
│   └── variables.tf
│   └── terragrunt.hcl   #<---- Child
├── frontend-app
│   ├── main.tf
│   └── other_resources.tf
│   └── variables.tf
│   └── terragrunt.hcl   #<---- Child
├── mysql
│   ├── main.tf
│   └── other_resources.tf
│   └── variables.tf
│   └── terragrunt.hcl   #<---- Child
└── mongo
    ├── main.tf
    └── other_resources.tf
    └── variables.tf
    └── terragrunt.hcl.  #<---- Child

The root terragrunt.hcl will keep your remote state configuration and the children will only have the following statement:

include {
  path = find_in_parent_folders()
}

This include block tells Terragrunt to use the exact same Terragrunt configuration from the root terragrunt.hcl file specified via the path parameter.

The next time you run terragrunt, it will automatically configure all the settings in the remote_state.config block, if they aren’t configured already, by calling terraform init.

The backend.tf file will be created automatically for you.

---

Summary

You can have hundreds of modules with nested hierarchy (for example divided into regions,tenants, applications etc') and still be able to maintain only one configuration of the remote state.

there is a version issue here within terraform, for me it is working for the mentioned version. also, it is good to have the terraform state on the bucket.

terraform {
    required_version = "~> 0.12.12"
    backend "gcs" {
        bucket = "bbucket-name"
        prefix = "terraform/state"
    }
}

As a word of caution, I would not create a terraform statefile with terraform in case someone inadvertently deletes it. So use scripts like aws-cli or boto3 which do not maintain state and keep those scripts limited to a variable for s3 bucket name. You will not really change the script for terraform state bucket in the long run except for creating additional folders inside the bucket which can be done outside terraform in the resource level.

All of the answers provided are very good. I just want to emphasize the "key" attribute. When you get into advanced applications of Terraform, you will eventually need to reference these S3 keys in order to pull remote state into a current project, or to leverage 'terraform move'.

It really helps to use intelligent key names when you plan your "terraform" stanza to define your backend.

I recommend the following as a base key name: account_name/{development:production}/region/module_name/terraform.tfstate

Revise to fit your needs, but going back and fixing all my key names as I expanded my use of Terraform across many accounts and regions was not fun at all.

Assuming that you are running terraform locally and not on some virtual server and that you want to store terraform state in S3 bucket that doesn't exist. This is how I would approach it,

> Create terraform script, that provisions S3 bucket > > Create terraform script that provisions your infrastructure > > At the end of your terraform script to provision bucket to be used by second terraform script for storing state files, include code to provision null resource. > > In the code block for the null resource using local-exec provisioner run command to go into the directory where your second terraform script exist followed by usual terraform init to initialize the backend then terraform plan, then terraform apply