Encrypted configuration in ASP.NET Core

With web.config going away, what is the preferred way to store sensitive info (passwords, tokens) in the configurations of a web app built using ASP.NET Core?

Is there a way to automatically get encrypted configuration sections in appsettings.json?

User secrets looks like a good solution for storing passwords, and, generally, application secrets, at least during development.

Check the official Microsoft documentation. You can also review this other SO question.

This is just a way to "hide" your secrets during development process and to avoid disclosing them into the source tree; the Secret Manager tool does not encrypt the stored secrets and should not be treated as a trusted store.

If you want to bring an encrypted appsettings.json to production, you can do so by building a custom configuration provider.

For example:

public class CustomConfigProvider : ConfigurationProvider, IConfigurationSource
{
    public CustomConfigProvider()
    {
    }

    public override void Load()
    {
        Data = UnencryptMyConfiguration();
    }

    private IDictionary<string, string> UnencryptMyConfiguration()
    {
        // do whatever you need to do here, for example load the file and unencrypt key by key
        //Like:
       var configValues = new Dictionary<string, string>
       {
            {"key1", "unencryptedValue1"},
            {"key2", "unencryptedValue2"}
       };
       return configValues;
    }

    private IDictionary<string, string> CreateAndSaveDefaultValues(IDictionary<string, string> defaultDictionary)
    {
        var configValues = new Dictionary<string, string>
        {
            {"key1", "encryptedValue1"},
            {"key2", "encryptedValue2"}
        };
        return configValues;                
    }

    public IConfigurationProvider Build(IConfigurationBuilder builder)
    {
       return new CustomConfigProvider();
    }
}

Define a static class for your extension method:

public static class CustomConfigProviderExtensions
{              
        public static IConfigurationBuilder AddEncryptedProvider(this IConfigurationBuilder builder)
        {
            return builder.Add(new CustomConfigProvider());
        }
}

And then you can activate it:

// Set up configuration sources.
var builder = new ConfigurationBuilder()
    .AddJsonFile("appsettings.json")
    .AddEncryptedProvider()
    .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true);

I agree with @CoderSteve that writing a whole new provider is too much work. It also doesn't build on the existing standard JSON architecture. Here is a solution that I come up with the builds on top of the standard JSON architecture, uses the preferred .Net Core encryption libraries, and is very DI friendly.

public static class IServiceCollectionExtensions
{
    public static IServiceCollection AddProtectedConfiguration(this IServiceCollection services)
    {
        services
            .AddDataProtection()
            .PersistKeysToFileSystem(new DirectoryInfo(@"c:\keys"))
            .ProtectKeysWithDpapi();

        return services;
    }

    public static IServiceCollection ConfigureProtected<TOptions>(this IServiceCollection services, IConfigurationSection section) where TOptions: class, new()
    {
        return services.AddSingleton(provider =>
        {
            var dataProtectionProvider = provider.GetRequiredService<IDataProtectionProvider>();
            section = new ProtectedConfigurationSection(dataProtectionProvider, section);
            
            var options = section.Get<TOptions>();
            return Options.Create(options);
        });
    }

    private class ProtectedConfigurationSection : IConfigurationSection
    {
        private readonly IDataProtectionProvider _dataProtectionProvider;
        private readonly IConfigurationSection _section;
        private readonly Lazy<IDataProtector> _protector;

        public ProtectedConfigurationSection(
            IDataProtectionProvider dataProtectionProvider,
            IConfigurationSection section)
        {
            _dataProtectionProvider = dataProtectionProvider;
            _section = section;

            _protector = new Lazy<IDataProtector>(() => dataProtectionProvider.CreateProtector(section.Path));
        }

        public IConfigurationSection GetSection(string key)
        {
            return new ProtectedConfigurationSection(_dataProtectionProvider, _section.GetSection(key));
        }

        public IEnumerable<IConfigurationSection> GetChildren()
        {
            return _section.GetChildren()
                .Select(x => new ProtectedConfigurationSection(_dataProtectionProvider, x));
        }

        public IChangeToken GetReloadToken()
        {
            return _section.GetReloadToken();
        }

        public string this[string key]
        {
            get => GetProtectedValue(_section[key]);
            set => _section[key] = _protector.Value.Protect(value);
        }

        public string Key => _section.Key;
        public string Path => _section.Path;

        public string Value
        {
            get => GetProtectedValue(_section.Value);
            set => _section.Value = _protector.Value.Protect(value);
        }

        private string GetProtectedValue(string value)
        {
            if (value == null)
                return null;

            return _protector.Value.Unprotect(value);
        }
    }
}

Wire up your protected config sections like this:

public void ConfigureServices(IServiceCollection services)
{
    services.AddMvc();

    // Configure normal config settings
    services.Configure<MySettings>(Configuration.GetSection("MySettings"));

    // Configure protected config settings
    services.AddProtectedConfiguration();
    services.ConfigureProtected<MyProtectedSettings>(Configuration.GetSection("MyProtectedSettings"));
}

You can easily create encrypted values for your config files using a controller like this:

[Route("encrypt"), HttpGet, HttpPost]
public string Encrypt(string section, string value)
{
    var protector = _dataProtectionProvider.CreateProtector(section);
    return protector.Protect(value);
}

Usage: http://localhost/cryptography/encrypt?section=SectionName:KeyName&value=PlainTextValue

I didn't want to write a custom provider – way too much work. I just wanted to tap into JsonConfigurationProvider, so I figured out a way that works for me, hope it helps someone.

public class JsonConfigurationProvider2 : JsonConfigurationProvider
{
    public JsonConfigurationProvider2(JsonConfigurationSource2 source) : base(source)
    {
    }

    public override void Load(Stream stream)
    {
        // Let the base class do the heavy lifting.
        base.Load(stream);

        // Do decryption here, you can tap into the Data property like so:

         Data["abc:password"] = MyEncryptionLibrary.Decrypt(Data["abc:password"]);

        // But you have to make your own MyEncryptionLibrary, not included here
    }
}

public class JsonConfigurationSource2 : JsonConfigurationSource
{
    public override IConfigurationProvider Build(IConfigurationBuilder builder)
    {
        EnsureDefaults(builder);
        return new JsonConfigurationProvider2(this);
    }
}

public static class JsonConfigurationExtensions2
{
    public static IConfigurationBuilder AddJsonFile2(this IConfigurationBuilder builder, string path, bool optional,
        bool reloadOnChange)
    {
        if (builder == null)
        {
            throw new ArgumentNullException(nameof(builder));
        }
        if (string.IsNullOrEmpty(path))
        {
            throw new ArgumentException("File path must be a non-empty string.");
        }

        var source = new JsonConfigurationSource2
        {
            FileProvider = null,
            Path = path,
            Optional = optional,
            ReloadOnChange = reloadOnChange
        };

        source.ResolveFileProvider();
        builder.Add(source);
        return builder;
    }
}

I managed to create a custom JSON configuration provider which uses DPAPI to encrypt and decrypt secrets. It basically uses simple regular expressions that you can define to specify what parts of the JSON needs to be encrypted.

The following steps are performed:

1. Json file is loaded
2. Determine whether the JSON parts that match the given regular expressions are already encrypted (or not). This is done by base-64 decoding of the JSON part and verify whether it starts with the expected prefix !ENC!)
3. If not encrypted, then encrypt the JSON part by first using DPAPI and secondly add the prefix !ENC! and encode to base-64
4. Overwrite the unencrypted JSON parts with the encrypted (base-64) values in the Json file

> Note that the base-64 does not bring better security, but only hides the prefix !ENC! for cosmetic reasons. This is just a matter of taste of course ;)

This solution consists of the following classes:

1. ProtectedJsonConfigurationProvider class (= custom JsonConfigurationProvider)
2. ProtectedJsonConfigurationSource class (= custom JsonConfigurationSource)
3. AddProtectedJsonFile() extension method on the IConfigurationBuilder in order to simple add the protected configuration

Assuming the following initial authentication.json file:

{
    "authentication": {
        "credentials": [
            {
                user: "john",
                password: "just a password"
            },
            {
                user: "jane",
                password: "just a password"
            }
        ]
    }
}

Which becomes (sort of) the following after loading

{
    "authentication": {
        "credentials": [
            {
                "user": "john",
                "password": "IUVOQyEBAAAA0Iyd3wEV0R=="
            },
            {
                "user": "jane",
                "password": "IUVOQyEBAAAA0Iyd3wEV0R=="
            }
        ]
    }
}

And assuming the following configuration class based on the json format

public class AuthenticationConfiguration
{
    [JsonProperty("credentials")]
    public Collection<CredentialConfiguration> Credentials { get; set; }
}

public class CredentialConfiguration
{
    [JsonProperty("user")]
    public string User { get; set; }
    [JsonProperty("password")]
    public string Password { get; set; }
}

Below the sample code:

//Note that the regular expression will cause the authentication.credentials.password path to be encrypted.
//Also note that the byte[] contains the entropy to increase security
var configurationBuilder = new ConfigurationBuilder()
    .AddProtectedJsonFile("authentication.json", true, new byte[] { 9, 4, 5, 6, 2, 8, 1 },
        new Regex("authentication:credentials:[0-9]*:password"));

var configuration = configurationBuilder.Build();
var authenticationConfiguration = configuration.GetSection("authentication").Get<AuthenticationConfiguration>();

//Get the decrypted password from the encrypted JSON file.
//Note that the ProtectedJsonConfigurationProvider.TryGet() method is called (I didn't expect that :D!)
var password = authenticationConfiguration.Credentials.First().Password

> Install the Microsoft.Extensions.Configuration.Binder package in order to get the configuration.GetSection("authentication").Get<T>() implementation

And finally the classes in which the magic happens :)

/// <summary>Represents a <see cref="ProtectedJsonConfigurationProvider"/> source</summary>
public class ProtectedJsonConfigurationSource : JsonConfigurationSource
{
    /// <summary>Gets the byte array to increse protection</summary>
    internal byte[] Entropy { get; private set; }

    /// <summary>Represents a <see cref="ProtectedJsonConfigurationProvider"/> source</summary>
    /// <param name="entropy">Byte array to increase protection</param>
    /// <exception cref="ArgumentNullException"/>
    public ProtectedJsonConfigurationSource(byte[] entropy)
    {
        this.Entropy = entropy ?? throw new ArgumentNullException(Localization.EntropyNotSpecifiedError);
    }

    /// <summary>Builds the configuration provider</summary>
    /// <param name="builder">Builder to build in</param>
    /// <returns>Returns the configuration provider</returns>
    public override IConfigurationProvider Build(IConfigurationBuilder builder)
    {
        EnsureDefaults(builder);
        return new ProtectedJsonConfigurationProvider(this);
    }

    /// <summary>Gets or sets the protection scope of the configuration provider. Default value is <see cref="DataProtectionScope.CurrentUser"/></summary>
    public DataProtectionScope Scope { get; set; }
    /// <summary>Gets or sets the regular expressions that must match the keys to encrypt</summary>
    public IEnumerable<Regex> EncryptedKeyExpressions { get; set; }
}

/// <summary>Represents a provider that protects a JSON configuration file</summary>
public partial class ProtectedJsonConfigurationProvider : JsonConfigurationProvider
{
    private readonly ProtectedJsonConfigurationSource protectedSource;
    private readonly HashSet<string> encryptedKeys = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
    private static readonly byte[] encryptedPrefixBytes = Encoding.UTF8.GetBytes("!ENC!");

    /// <summary>Checks whether the given text is encrypted</summary>
    /// <param name="text">Text to check</param>
    /// <returns>Returns true in case the text is encrypted</returns>
    private bool isEncrypted(string text)
    {
        if (text == null) { return false; }

        //Decode the data in order to verify whether the decoded data starts with the expected prefix
        byte[] decodedBytes;
        try { decodedBytes = Convert.FromBase64String(text); }
        catch (FormatException) { return false; }

        return decodedBytes.Length >= encryptedPrefixBytes.Length
            && decodedBytes.AsSpan(0, encryptedPrefixBytes.Length).SequenceEqual(encryptedPrefixBytes);
    }

    /// <summary>Converts the given key to the JSON token path equivalent</summary>
    /// <param name="key">Key to convert</param>
    /// <returns>Returns the JSON token path equivalent</returns>
    private string convertToTokenPath(string key)
    {
        var jsonStringBuilder = new StringBuilder();

        //Split the key by ':'
        var keyParts = key.Split(':');
        for (var keyPartIndex = 0; keyPartIndex < keyParts.Length; keyPartIndex++)
        {
            var keyPart = keyParts[keyPartIndex];

            if (keyPart.All(char.IsDigit)) { jsonStringBuilder.Append('[').Append(keyPart).Append(']'); }
            else if (keyPartIndex > 0) { jsonStringBuilder.Append('.').Append(keyPart); }
            else { jsonStringBuilder.Append(keyPart); }
        }

        return jsonStringBuilder.ToString();
    }

    /// <summary>Writes the given encrypted key/values to the JSON oconfiguration file</summary>
    /// <param name="encryptedKeyValues">Encrypted key/values to write</param>
    private void writeValues(IDictionary<string, string> encryptedKeyValues)
    {
        try
        {
            if (encryptedKeyValues == null || encryptedKeyValues.Count == 0) { return; }

            using (var stream = new FileStream(this.protectedSource.Path, FileMode.Open, FileAccess.ReadWrite))
            {
                JObject json;

                using (var streamReader = new StreamReader(stream, Encoding.UTF8, true, 4096, true))
                {
                    using (var jsonTextReader = new JsonTextReader(streamReader))
                    {
                        json = JObject.Load(jsonTextReader);

                        foreach (var encryptedKeyValue in encryptedKeyValues)
                        {
                            var tokenPath = this.convertToTokenPath(encryptedKeyValue.Key);
                            var value = json.SelectToken(tokenPath) as JValue;
                            if (value.Value != null) { value.Value = encryptedKeyValue.Value; }
                        }
                    }
                }

                stream.Seek(0, SeekOrigin.Begin);
                using (var streamWriter = new StreamWriter(stream))
                {
                    using (var jsonTextWriter = new JsonTextWriter(streamWriter) { Formatting = Formatting.Indented })
                    {
                        json.WriteTo(jsonTextWriter);
                    }
                }
            }
        }
        catch (Exception exception)
        {
            throw new Exception(string.Format(Localization.ProtectedJsonConfigurationWriteEncryptedValues, this.protectedSource.Path), exception);
        }
    }

    /// <summary>Represents a provider that protects a JSON configuration file</summary>
    /// <param name="source">Settings of the source</param>
    /// <see cref="ArgumentNullException"/>
    public ProtectedJsonConfigurationProvider(ProtectedJsonConfigurationSource source) : base(source)
    {
        this.protectedSource = source as ProtectedJsonConfigurationSource;
    }

    /// <summary>Loads the JSON data from the given <see cref="Stream"/></summary>
    /// <param name="stream"><see cref="Stream"/> to load</param>
    public override void Load(Stream stream)
    {
        //Call the base method first to ensure the data to be available
        base.Load(stream);
        
        var expressions = protectedSource.EncryptedKeyExpressions;
        if (expressions != null)
        {
            //Dictionary that contains the keys (and their encrypted value) that must be written to the JSON file
            var encryptedKeyValuesToWrite = new Dictionary<string, string>();

            //Iterate through the data in order to verify whether the keys that require to be encrypted, as indeed encrypted.
            //Copy the keys to a new string array in order to avoid a collection modified exception
            var keys = new string[this.Data.Keys.Count];
            this.Data.Keys.CopyTo(keys, 0);

            foreach (var key in keys)
            {
                //Iterate through each expression in order to check whether the current key must be encrypted and is encrypted.
                //If not then encrypt the value and overwrite the key
                var value = this.Data[key];
                if (!string.IsNullOrEmpty(value) && expressions.Any(e => e.IsMatch(key)))
                {
                    this.encryptedKeys.Add(key);

                    //Verify whether the value is encrypted
                    if (!this.isEncrypted(value))
                    {
                        var protectedValue = ProtectedData.Protect(Encoding.UTF8.GetBytes(value), protectedSource.Entropy, protectedSource.Scope);
                        var protectedValueWithPrefix = new List<byte>(encryptedPrefixBytes);
                        protectedValueWithPrefix.AddRange(protectedValue);

                        //Convert the protected value to a base-64 string in order to mask the prefix (for cosmetic purposes)
                        //and overwrite the key with the encrypted value
                        var protectedBase64Value = Convert.ToBase64String(protectedValueWithPrefix.ToArray());
                        encryptedKeyValuesToWrite.Add(key, protectedBase64Value);
                        this.Data[key] = protectedBase64Value;
                    }
                }
            }

            //Write the encrypted key/values to the JSON configuration file
            this.writeValues(encryptedKeyValuesToWrite);
        }
    }

    /// <summary>Attempts to get the value of the given key</summary>
    /// <param name="key">Key to get</param>
    /// <param name="value">Value of the key</param>
    /// <returns>Returns true in case the key has been found</returns>
    public override bool TryGet(string key, out string value)
    {
        if (!base.TryGet(key, out value)) { return false; }
        else if (!this.encryptedKeys.Contains(key)) { return true; }
        
        //Key is encrypted and must therefore be decrypted in order to return.
        //Note that the decoded base-64 bytes contains the encrypted prefix which must be excluded when unprotection
        var protectedValueWithPrefix = Convert.FromBase64String(value);
        var protectedValue = new byte[protectedValueWithPrefix.Length - encryptedPrefixBytes.Length];
        Buffer.BlockCopy(protectedValueWithPrefix, encryptedPrefixBytes.Length, protectedValue, 0, protectedValue.Length);

        var unprotectedValue = ProtectedData.Unprotect(protectedValue, this.protectedSource.Entropy, this.protectedSource.Scope);
        value = Encoding.UTF8.GetString(unprotectedValue);
        return true;
    }

/// <summary>Provides extensions concerning <see cref="ProtectedJsonConfigurationProvider"/></summary>
public static class ProtectedJsonConfigurationProviderExtensions
{
    /// <summary>Adds a protected JSON file</summary>
    /// <param name="configurationBuilder"><see cref="IConfigurationBuilder"/> in which to apply the JSON file</param>
    /// <param name="path">Path to the JSON file</param>
    /// <param name="optional">Specifies whether the JSON file is optional</param>
    /// <param name="entropy">Byte array to increase protection</param>
    /// <returns>Returns the <see cref="IConfigurationBuilder"/></returns>
    /// <exception cref="ArgumentNullException"/>
    public static IConfigurationBuilder AddProtectedJsonFile(this IConfigurationBuilder configurationBuilder, string path, bool optional, byte[] entropy, params Regex[] encryptedKeyExpressions)
    {
        var source = new ProtectedJsonConfigurationSource(entropy)
        {
            Path = path,
            Optional = optional,
            EncryptedKeyExpressions = encryptedKeyExpressions
        };

        return configurationBuilder.Add(source);
    }
}

public static IServiceCollection ConfigureProtected<TOptions>(this IServiceCollection services, IConfigurationSection section) where TOptions: class, new()
{
    return services.AddSingleton(provider =>
    {
        var dataProtectionProvider = provider.GetRequiredService<IDataProtectionProvider>();
        var protectedSection = new ProtectedConfigurationSection(dataProtectionProvider, section);

        var options = protectedSection.Get<TOptions>();
        return Options.Create(options);
    });
}

This method is correct

Just a few clarifications to help avoid problems. When you encrypt a value, it's using the section as 'Purpose' (https://docs.microsoft.com/en-us/aspnet/core/security/data-protection/consumer-apis/purpose-strings?view=aspnetcore-2.2) When you get a 'Payload not valid' or something similar, it's likely that the purpose you used to encrypt it, differs from the purpose use to decrypt it. So, let's say I have a first level section in my appsettings.json named 'SecureSettings' and within it a connection string:

{
"SecureSettings": 
  {
    "ConnectionString":"MyClearTextConnectionString"
  }
}

To encrypt the value, I'd call: http://localhost/cryptography/encrypt?section=SecureSettings:ConnectionString&value=MyClearTextConnectionString

You may not want to keep an Encrypt controller in the app itself btw.