Handling Application Settings in your DevOps Deployment Pipeline

Today we’ll look at handling application settings in build and deployment pipelines.

An application normally has several settings that are stored outside of main application codebase. Database Connection strings, external SaaS API Keys, credentials are examples of settings and other environment specific settings will be stored in appsettings.json for a DotNetCore app, web.xml for a Java app, or settings.yml for Ruby.

If you have hardcoded connection strings or parameters inside your application, it is highly recommended to extract these out to a file so that you can easily update these without having to store sensitive information inside the source code and stored as part of your version control repository.

AWS SSM Parameter Store (AWS Systems Manager Parameter Store) is a service and it’s importance should not be underestimated. It provides secure, hierarchical storage for configuration data management and secrets management. You can store data such as passwords, database strings, and license codes as parameter values.

Build Code and pull in Parameters from Parameter Store

AWS CodeBuild is regularly used to build software as part of deployment pipelines. AWS CodeBuild is a fully managed build service in the cloud. AWS CodeBuild compiles your source code, runs unit tests, and produces artifacts that are ready to deploy. During a build of your application, you can securely request parameters from your secure store for inclusion in your application.

In the below example, I’m using CodeBuild, the JQ tool and sponge to retrieve parameter values out of the parameter store, and write them into an appsettings.json configuration file that my lambda function uses.


$ENVIRONMENT = ‘production’

$PIPELINE_LAMBDA_BUCKET = ‘idea11-coolapp-pipeline’

version: 0.2

# Install dependencies
- apt-get update
- echo "Installing dependencies..."
- apt-get install -y moreutils jq
# Retrieve credentials
- set -x
- echo "Configuring AWS credentials"
- curl -qL -o aws_credentials.json$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI > aws_credentials.json
- aws configure set region $AWS_REGION
- aws configure set aws_access_key_id `jq -r '.AccessKeyId' aws_credentials.json`
- aws configure set aws_secret_access_key `jq -r '.SecretAccessKey' aws_credentials.json`
- aws configure set aws_session_token `jq -r '.Token' aws_credentials.json`
# Build Release
- dotnet restore
- dotnet build -c Release
- dotnet publish -c Release
- export WEB_PUBLISH_DIR=Idea11.CoolApp/bin/Release/netcoreapp2.1/publish
# Retrieve Application Settings from AWS Parameter Store
- export COGNITO_CLIENT_SECRET="`aws ssm get-parameter --name /idea11/$ENVIRONMENT/coolapp/AWSCognito/ClientSecret | jq -r .Parameter.Value`"
- export COGNITO_CLIENT_ID="`aws ssm get-parameter --name /idea11/$ENVIRONMENT/coolapp/AWSCognito/ClientId | jq -r .Parameter.Value`"
- export COGNITO_USERPOOL="`aws ssm get-parameter --name /idea11/$ENVIRONMENT/coolapp/AWSCognito/UserPool | jq -r .Parameter.Value`"
- export COGNITO_LOGOUT_URL="`aws ssm get-parameter --name /idea11/$ENVIRONMENT/coolapp/AWSCognito/LogOutUrl | jq -r .Parameter.Value`"
- export COGNITO_METADATA_ADDRESS="`aws ssm get-parameter --name /idea11/$ENVIRONMENT/coolapp/AWSCognito/MetaDataAddress | jq -r .Parameter.Value`"
- export BUCKET_NAME="`aws ssm get-parameter --name /idea11/$ENVIRONMENT/coolapp/BucketName | jq -r .Parameter.Value`"
- export SIGNED_COOKIE_CUSTOM_POLICY_RESOURCE="`aws ssm get-parameter --name /idea11/$ENVIRONMENT/coolapp/SignedCookieCustomPolicyResource | jq -r .Parameter.Value`"
- export BASE_URL="`aws ssm get-parameter --name /idea11/$ENVIRONMENT/coolapp/BaseUrl | jq -r .Parameter.Value`"
- export COOKIE_DOMAIN="`aws ssm get-parameter --name /idea11/$ENVIRONMENT/coolapp/CookieDomain | jq -r .Parameter.Value`"
- export CLOUDFRONT_KEY_ID="`aws ssm get-parameter --name /idea11/$ENVIRONMENT/coolapp/CloudFront/KeyId | jq -r .Parameter.Value`"
- export CLOUDFRONT_PRIVATE_KEY="`aws ssm get-parameter --name /idea11/$ENVIRONMENT/coolapp/CloudFront/PrivateKey --with-decryption | jq -r .Parameter.Value`"
# Write Retrieved Settings to appsettings.json
- jq ".AWSCognito.ClientSecret = \"$COGNITO_CLIENT_SECRET\"" $WEB_PUBLISH_DIR/appsettings.json | sponge $WEB_PUBLISH_DIR/appsettings.json
- jq ".AWSCognito.ClientId = \"$COGNITO_CLIENT_ID\"" $WEB_PUBLISH_DIR/appsettings.json | sponge $WEB_PUBLISH_DIR/appsettings.json
- jq ".AWSCognito.UserPool = \"$COGNITO_USERPOOL\"" $WEB_PUBLISH_DIR/appsettings.json | sponge $WEB_PUBLISH_DIR/appsettings.json
- jq ".AWSCognito.LogOutUrl = \"$COGNITO_LOGOUT_URL\"" $WEB_PUBLISH_DIR/appsettings.json | sponge $WEB_PUBLISH_DIR/appsettings.json
- jq ".AWSCognito.MetaDataAddress = \"$COGNITO_METADATA_ADDRESS\"" $WEB_PUBLISH_DIR/appsettings.json | sponge $WEB_PUBLISH_DIR/appsettings.json
- jq ".BucketName = \"$BUCKET_NAME\"" $WEB_PUBLISH_DIR/appsettings.json | sponge $WEB_PUBLISH_DIR/appsettings.json
- jq ".SignedCookieCustomPolicyResource = \"$SIGNED_COOKIE_CUSTOM_POLICY_RESOURCE\"" $WEB_PUBLISH_DIR/appsettings.json | sponge $WEB_PUBLISH_DIR/appsettings.json
- jq ".BaseUrl = \"$BASE_URL\"" $WEB_PUBLISH_DIR/appsettings.json | sponge $WEB_PUBLISH_DIR/appsettings.json
- jq ".CookieDomain = \"$COOKIE_DOMAIN\"" $WEB_PUBLISH_DIR/appsettings.json | sponge $WEB_PUBLISH_DIR/appsettings.json
- jq ".CloudFront.KeyId = \"$CLOUDFRONT_KEY_ID\"" $WEB_PUBLISH_DIR/appsettings.json | sponge $WEB_PUBLISH_DIR/appsettings.json
- jq ".CloudFront.PrivateKey = \"$CLOUDFRONT_PRIVATE_KEY\"" $WEB_PUBLISH_DIR/appsettings.json | sponge $WEB_PUBLISH_DIR/appsettings.json
- aws cloudformation package --template-file serverless.template --s3-bucket $PIPELINE_LAMBDA_BUCKET --kms-key-id $KMS_KEY_ARN --s3-prefix coolapp-$ENVIRONMENT/Packaged/ --output-template-file outputServerless.template

- outputServerless.template

Why don’t we just store our app settings as environment variables in the lambda function?

You can store app settings in environment variables – be aware that you’ll potentially see sensitive information right there in the AWS Console when you open the function with lambda:GetFunctionConfiguration permission.

Encryption of environment variables can’t be implemented within CloudFormation, and you shouldn’t store environment specific information within your source repository.

Why don’t we just give our application permission to access parameter store directly during runtime, rather than including the configuration at build time?

You can, but you may hit some nasty limits. Like this one:

ClientError: An error occurred (ThrottlingException) when calling the GetParameters operation (reached max retries: 4): Rate exceeded

If your application needs to evaluate some condition based on what is only available via parameter store, and your application continually goes back to SSM Parameter Store to retrieve the values, you may encounter errors due to throttling limits. This is especially true for serverless web applications, where every HTTP(s) request is an invocation of your lambda function.

Parameter Store has an undocumented hard throttle limit. If your site suddenly becomes popular, it may become unavailable, even though you’ve architected your site to serverless 🙂

What about using AWS Secrets Manager?

Secrets Manager does have some additional features and some higher throttling limits than SSM Parameter Store, but also costs quite a bit more. AWS SSM Parameter Store is Free whereas AWS Secrets Manager has a limit of 700 requests per second for DescribeSecret and GetSecretValue, but will also cost you $0.40 per secret/month and $0.05 per 1,000 API calls.