Storing passwords in Chef?

What is the best practice for storing password and API keys with Chef? It's really tempting to store database passwords, AWS api keys, and other sensitive credentials as Chef Server Attributes for use in recipes -- but what about security considerations? What's the best practice for this?

From the #chef IRC channel, many people store this kind of data in a data bag on the chef server.

For example, a data bag might be 'aws', with an item 'main', referring to the primary AWS account. Separate keys in the item would be for each particular value. E.g.:

{
  "id": "main",
  "aws_secret_key": "The secret access key",
  "aws_access_key": "The access key"
}

You may also be interested in encrypted data bags. I wrote about them in more detail for managing postfix SASL authentication.

Update: I've written blog posts about Chef Vault on my blog and sysadvent.

This question is old and has no accepted answer, however, the correct answer to this question is that Chef allows the use of Encrypted Data Bags for storing sensitive data in Data Bags.

I think Hashicorp's Vault is really promising as a way to dynamically retrieve encrypted information and leave behind some of the oddities of Chef workflow in this area.

This is an interesting post that starts to touch the subject. https://www.hashicorp.com/blog/using-hashicorp-vault-with-chef.html

The best practice is to keep keys and passwords in chef data_bags. A data bag contains databag items. Individual data_bag item are in json format.

For exmaple:

{
  /* This is a supported comment style */
  // This style is also supported
  "id": "ITEM_NAME",
  "key": "value"
}

Encrypt Data Bag Item: data bag item may be encrypted using shared secret encryption. This allows each data bag item to store confidential information (such as a database password or ssh keys) or to be managed in a source control system (without plain-text data appearing in revision history). This can be done as follow:

Crete Secret Keys: Create a secret key called encrypted_data_bag_secret for example

$ openssl rand -base64 512 | tr -d '\r\n' > encrypted_data_bag_secret

where encrypted_data_bag_secret is the name of the file which will contain the secret key

Encrypt the data_bag: A data bag item is encrypted using a knife command similar to:

$ knife data bag create passwords mysql --secret-file /tmp/my_data_bag_key

where “passwords” is the name of the data bag, “mysql” is the name of the data bag item, and “/tmp/my_data_bag_key” is the path to the location in which the file that contains the secret-key is locate

Verify Encryption: When the contents of a data bag item are encrypted, they will not be readable until they are decrypted. Encryption can be verified with a knife command similar to:

$ knife data bag show passwords mysql

Decrypt data Bag: An encrypted data bag item is decrypted with a knife command similar to:

$ knife data bag show --secret-file /tmp/my_data_bag_key passwords mysql

Chef Encrypted data_bags is indeed a legitimate solution. Adding to that, you can also use a ruby Gem that allows you to encrypt a Chef Data Bag Item using the public keys of a list of chef nodes. This allows only those chef nodes to decrypt the encrypted values. cf. https://github.com/Nordstrom/chef-vault

Chef Vault can be a good choice. It provides simple interface for storing encrypted data on chef-server, access management. Upload, edit, update data with knife vault ... commands.

To get data from recipe use ChefVault::Item.load command

chef_gem "chef-vault"
require 'chef-vault'
item = ChefVault::Item.load("passwords", "root")
item["password"]

To set users, which can update data use knife vault_admins property.

knife[:vault_admins] = [ 'example-alice', 'example-bob', 'example-carol' ]

I've never tried databags, but that's probably because I find everything apart from chef-solo a little too complicated. Which is why I'm using chef recipies with a service called Scalarium.

So the issue with passwords, or e.g. private keys and all kinds of other credentials is a pretty tough one. I too have a bunch of recipes where passwords need to be created, or set correctly.

Usually what I do is, I specify what the scalarium folks call custom json. This json is similar to the node.json some people give to chef-solo using chef-solo -j node.json.

So e.g. in my custom json on Scalarium web interface, I have the following:

{"super_secure_password":"foobar"}

What this does is, my super secure password is available during my chef run in node[:super_secure_password] and I can use it in recipes or templates.

This works fine as long as I only deploy my server using Scalarium but we also use our recipes in local vagrant boxes for a development environment and easier testing. And when I use vagrant (or even chef-solo by itself), I don't have access to the custom json on Scalarium.

This is what I do to fix that, in my_recipe/attributes/default:

set_unless[:super_secure_password] = "test123"

This means that when my recipe is run outside of scalarium, the password is still available in node[:super_secure_password] and my recipes work and so on. When the recipe is executed in the scalarium context, it will not override what they provide.

Currently most widely used approach and for most cases secure enough is to use chef-vault.

It uses shared secret to encrypt your data (similarity to chef-encrypted databag). This shared secret is encrypted for every client and/or user that will use it(if You allow to use it).

Benefits:

• on test environment you can use unencrypted data
• One doesn't store shared secret as a plain text
• One may grant access only few out their servers to read and write some databags

Example

export EDITOR=vi #sets your favourite text editor 

knife vault create secret_data john_doe  --admins "admin" --search "*:*" --mode client 

Command above creates in secret_data databag item: john_doe that can modified by admin and used by all clients. After that command EDITOR will open so you can type o paste your secret data(in json ).

Search query can be: "role:basic" - Which means only servers with role basic can read this data knife vault need some extra installation

In Your Cookbook

chef_gem 'chef-vault' do
    compile_time true if respond_to?(:compile_time)
end

require 'chef-vault'

item = ChefVault::Item.load("secret_data", "john_doe")
item["password"]

and in metadata.rb: depends 'chef-vault', '1.3.0'

more info here: https://blog.chef.io/2016/01/21/chef-vault-what-is-it-and-what-can-it-do-for-you/

and here: https://github.com/chef/chef-vault

I would Suggest use IAM role with chef provisioning

require 'chef/provisioning/aws_driver'
iam = AWS::Core::CredentialProviders::EC2Provider.new
puts iam.credentials.inspect
with_driver(
  'aws:IAM:eu-west-1',
  :aws_credentials => { 'IAM' => iam.credentials }
)