Secrets Manager / Parameter Store

Where do you put a DB password, an external API key, an OAuth client secret? Code and git are absolutely out, and plaintext in environment variables is dangerous too. AWS has two options — Secrets Manager and SSM Parameter Store.

This chapter puts together, all at once, the criteria for secret / configuration management: the difference between the two, automatic rotation, the pattern for fetching from code, integration with ECS / Lambda, IaC connection, and a cost comparison. If everything through the previous Chapter 19 EventBridge / SQS / SNS was about communication between components, this chapter deals with the secrets those components must hold safely. The pattern we set up here is used directly in the Task Definition secret injection of Chapter 15 ECS and Fargate, Part 4’s Chapter 22 ECS Fargate deployment skeleton, and Chapter 23 RDS integration.

Where secrets must not go #

This is an extension of Chapter 6 Security basics. The things you must absolutely never do are as follows.

• Plaintext in code (PASSWORD = "abc123")
• A .env in git (it remains in history forever, even after you delete it)
• Pasting into a README, messenger, wiki, or email
• Environment variables / build args in a Dockerfile (stamped into the image layers)
• Plaintext environment variables in an ECS Task Definition (exposed in the console / IaC / logs)

Use Secrets Manager or SSM Parameter Store instead.

The difference between the two #

A one-line decision guide #

DB password / external API key + need automatic rotation → Secrets Manager
   │
general configuration values (DB host, region, feature flag) → Parameter Store
   │
secret with no rotation needed (e.g., external API key) → Parameter Store SecureString (cheap)

Secrets Manager #

Creating #

aws secretsmanager create-secret \
  --name myapp/prod/db \
  --description "Postgres for myapp prod" \
  --secret-string '{
    "username": "myapp",
    "password": "very-strong-secret",
    "host": "myapp-prod.abc123.ap-northeast-2.rds.amazonaws.com",
    "port": 5432,
    "dbname": "myapp"
  }'

An ARN is returned.

arn:aws:secretsmanager:ap-northeast-2:123456789012:secret:myapp/prod/db-AbCdEf

The 6 random characters at the end of the ARN (AbCdEf) are to prevent unauthorized enumeration of secrets. In an IAM policy, use a wildcard like arn:.../myapp/prod/db-*.

Reading — boto3 #

import boto3, json

client = boto3.client("secretsmanager")
resp = client.get_secret_value(SecretId="myapp/prod/db")
creds = json.loads(resp["SecretString"])

print(creds["username"], creds["password"])

Caching — fetching it every call costs #

Since it’s charged per API call, calling get_secret_value on every request is expensive and slow.

Use the pattern of once in the global scope + caching.

import boto3, json

# once in the INIT phase of the Lambda container
_secrets = None

def get_db_creds():
    global _secrets
    if _secrets is None:
        resp = boto3.client("secretsmanager").get_secret_value(
            SecretId="myapp/prod/db")
        _secrets = json.loads(resp["SecretString"])
    return _secrets

def handler(event, context):
    creds = get_db_creds()
    ...

This is the same context as the global variable pattern of Chapter 17 Lambda basics. Powertools’s Parameters handles caching / TTL / multiple secrets in one call.

from aws_lambda_powertools.utilities import parameters

# TTL 5-minute caching
creds = parameters.get_secret("myapp/prod/db", transform="json",
                              max_age=300)

Automatic rotation #

Secrets Manager’s biggest advantage. At a set interval, it generates a new password and automatically updates the user’s password in the DB.

aws secretsmanager rotate-secret \
  --secret-id myapp/prod/db \
  --rotation-lambda-arn arn:aws:lambda:ap-northeast-2:...:SecretsManagerRDSPostgreSQLRotationSingleUser \
  --rotation-rules AutomaticallyAfterDays=30

AWS provides managed rotation Lambdas as templates (RDS / Redshift / DocumentDB / Aurora). It rotates to a new password every 30 days.

The 4 steps of rotation (for understanding) #

1. createSecret      → generate a new password, store as AWSPENDING
2. setSecret         → apply the new password to the DB
3. testSecret        → test the connection with the new password
4. finishSecret      → AWSPENDING → AWSCURRENT, the old → AWSPREVIOUS

The code always fetches AWSCURRENT (the default). Briefly during rotation, AWSPREVIOUS is also valid.

Single-user vs Multi-user rotation #

• Single user: swaps one user’s password. A brief connection drop is possible at the moment of rotation.
• Multi-user: alternates between two users. Zero-downtime rotation is possible. Recommended for operations.

Parameter Store (SSM) #

Standard vs Advanced #

Start with Standard by default.

Creating #

aws ssm put-parameter \
  --name /myapp/prod/aws-region \
  --value ap-northeast-2 \
  --type String

# hierarchy (free — structure with '/')
aws ssm put-parameter \
  --name /myapp/prod/feature/new-checkout \
  --value enabled \
  --type String

aws ssm put-parameter \
  --name /myapp/prod/external-api-key \
  --value sk_live_abc123 \
  --type SecureString \
  --key-id alias/aws/ssm

SecureString encrypts with KMS. The AWS-managed key (alias/aws/ssm) is free.

Reading #

import boto3

ssm = boto3.client("ssm")

# single
resp = ssm.get_parameter(
    Name="/myapp/prod/external-api-key",
    WithDecryption=True
)
api_key = resp["Parameter"]["Value"]

# multiple (hierarchy)
resp = ssm.get_parameters_by_path(
    Path="/myapp/prod/",
    Recursive=True,
    WithDecryption=True
)
for p in resp["Parameters"]:
    print(p["Name"], p["Value"])

from aws_lambda_powertools.utilities import parameters

# single
api_key = parameters.get_parameter(
    "/myapp/prod/external-api-key", decrypt=True, max_age=300)

# hierarchy
all_params = parameters.get_parameters(
    "/myapp/prod/", decrypt=True, max_age=300)

Versioning #

Every time you put a new value under the same name, versions accumulate as 1, 2, 3, …. You can roll back to an old value.

# view the value of version 5
aws ssm get-parameter-history --name /myapp/prod/db-host

# revert to version 3 (copy into a new version)
aws ssm put-parameter --name /myapp/prod/db-host \
  --value $(aws ssm get-parameter --name /myapp/prod/db-host:3 \
    --query 'Parameter.Value' --output text) --overwrite

Integration with ECS / Lambda #

The way you’ll meet most often. Inject secrets directly into the Task Definition / function environment variables.

ECS Task Definition #

{
  "containerDefinitions": [
    {
      "name": "web",
      "image": "...",
      "environment": [
        {"name": "ENV", "value": "production"}
      ],
      "secrets": [
        {
          "name": "DATABASE_URL",
          "valueFrom": "arn:aws:secretsmanager:ap-northeast-2:123456789012:secret:myapp/prod/db-AbCdEf"
        },
        {
          "name": "STRIPE_KEY",
          "valueFrom": "arn:aws:ssm:ap-northeast-2:123456789012:parameter/myapp/prod/stripe-key"
        }
      ]
    }
  ]
}

ECS fetches the secret right before the Task starts and injects it as the container’s environment variable. In the code you only have to look at os.environ["DATABASE_URL"].

The permissions needed (the Execution Role of Chapter 15 ECS and Fargate).

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue"],
      "Resource": "arn:aws:secretsmanager:ap-northeast-2:123456789012:secret:myapp/prod/*"
    },
    {
      "Effect": "Allow",
      "Action": ["ssm:GetParameters"],
      "Resource": "arn:aws:ssm:ap-northeast-2:123456789012:parameter/myapp/prod/*"
    },
    {
      "Effect": "Allow",
      "Action": ["kms:Decrypt"],
      "Resource": "arn:aws:kms:ap-northeast-2:123456789012:key/*"
    }
  ]
}

Only a specific field of a JSON secret #

When a Secrets Manager secret is JSON, ECS can hand only a specific field as an environment variable.

{
  "name": "DB_PASSWORD",
  "valueFrom": "arn:aws:secretsmanager:...:myapp/prod/db-AbCdEf:password::"
}

The :password:: at the end of the ARN extracts only the password field of the JSON. Between the first : and the last : is the JSON key; after the second : is the version.

Secrets in Lambda #

Lambda is the same pattern. Don’t stamp secrets directly into the function’s environment variables; fetch them with boto3 in the code. Or use the AWS Parameters and Secrets Lambda Extension as a caching sidecar — call it at localhost:2773 from inside the function.

import urllib.request, json, os

def get_secret(name):
    url = f"http://localhost:2773/secretsmanager/get?secretId={name}"
    req = urllib.request.Request(url,
        headers={"X-Aws-Parameters-Secrets-Token": os.environ["AWS_SESSION_TOKEN"]})
    with urllib.request.urlopen(req) as resp:
        data = json.loads(resp.read())
    return json.loads(data["SecretString"])

The Extension caches automatically (TTL configurable) — you don’t have to write boto3 caching code.

Connection with IaC #

Terraform #

resource "aws_secretsmanager_secret" "db" {
  name = "myapp/prod/db"
}

resource "aws_secretsmanager_secret_version" "db" {
  secret_id = aws_secretsmanager_secret.db.id
  secret_string = jsonencode({
    username = "myapp"
    password = random_password.db.result
    host     = aws_db_instance.main.address
  })
}

resource "random_password" "db" {
  length  = 32
  special = false
}

Don’t stamp the password itself directly into IaC. Generate it with random_password, or let AWS manage it with rotation. Terraform itself is covered in Chapter 25 Terraform intro.

CloudFormation’s dynamic reference #

Resources:
  MyDB:
    Type: AWS::RDS::DBInstance
    Properties:
      MasterUsername: '{{resolve:secretsmanager:myapp/prod/db:SecretString:username}}'
      MasterUserPassword: '{{resolve:secretsmanager:myapp/prod/db:SecretString:password}}'

The {{resolve:...}} pattern fetches secrets / parameters directly inside the stack. Plaintext doesn’t remain in the stack template / logs.

Cost #

Secrets Manager #

• $0.40 / secret / month
• $0.05 / 10,000 API calls

10 secrets + 100 calls daily = $4 + nearly 0 = $4 / month.

Parameter Store (Standard) #

• Free (up to 40 / sec throughput)
• When using KMS, KMS API call cost ($0.03 / 10,000)

10 SecureStrings + 100 calls daily = nearly 0.

The point of the price difference #

If you don’t need rotation / managed integration, Parameter Store SecureString is overwhelmingly cheaper. With 100 secrets, it’s Secrets Manager $40 / month vs Parameter Store ~$0.

Separating secrets from configuration #

The recommended operational pattern.

Secrets Manager
├── /myapp/prod/db          (automatic rotation)
└── /myapp/prod/jwt-signing (automatic rotation)

Parameter Store
├── /myapp/prod/db-host
├── /myapp/prod/aws-region
├── /myapp/prod/feature/new-checkout   (feature flag)
├── /myapp/prod/log-level
└── /myapp/prod/external-api-key       (SecureString — a secret that isn't rotated)

Put only the core secrets that need rotation in Secrets Manager. Cut cost by keeping the rest in Parameter Store.

Cross-Account / Cross-Region #

Resource Policy #

Allows a user in another account access to a secret.

aws secretsmanager put-resource-policy \
  --secret-id myapp/prod/db \
  --resource-policy '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::222222222222:root"},
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*"
    }]
  }'

Replication #

Automatically replicates to another region (multi-region operations).

aws secretsmanager replicate-secret-to-regions \
  --secret-id myapp/prod/db \
  --add-replica-regions Region=us-east-1

When the primary is rotated, the replica auto-syncs too. Multi-region · disaster recovery is covered in Chapter 30 Disaster recovery · backup.

Pitfalls you’ll often hit #

1) The secret isn’t injected into the ECS Task #

At Task start, “ResourceInitializationError … not authorized to perform secretsmanager:GetSecretValue” comes up. A missing Execution Role permission. Or a missing kms:Decrypt on the SecureString’s KMS key.

2) A boto3 call on every request #

If you do get_secret_value every time inside the code’s handler, API calls explode and you get cost + latency. Fetch it once in the module’s global scope + caching.

3) A typo in the shape of the JSON-field-extraction ARN #

In arn:...:db-AbCdEf:password::, if the colon count / position is off by even one character, it silently becomes an empty string. Copy it as-is from the console.

4) A plaintext secret in IaC #

Terraform secret_string = "abc123" remains in plaintext in the state file. Always use random_password or external input, and encrypt the state (S3 + KMS).

5) Call failure with the old password after rotation #

For a short time during rotation, a connection made with the old password can drop. Use multi-user rotation + connection retry.

6) Unintended disclosure of a secret’s name #

Even though the 6 random characters of the ARN (-AbCdEf) remaining in CloudTrail / logs are hard to guess, anyone with console permission can see the secret’s name itself (myapp/prod/db). Reduce permissions (a separate IAM per secret) and narrow the exposure surface.

7) Parameter Store throughput limit #

When you hit Standard’s 40 req/s limit, throttling occurs. If you call it frequently on a hot path, use Advanced or caching.

Exercises #

1. Write down all the secrets and configuration your app holds, and following the pattern in §“Separating secrets from configuration,” split which go into Secrets Manager and which into Parameter Store SecureString. Write in one sentence the criterion that separates the items needing automatic rotation from those that don’t.
2. When a “ResourceInitializationError” occurs because a secret isn’t injected into an ECS Task, write the two permissions you should check, basing it on §“Integration with ECS / Lambda” and the Execution Role of Chapter 15 ECS and Fargate.
3. Basing it on the figures in §“Cost,” compute the monthly cost difference between putting all 100 secrets in Secrets Manager versus putting only the 5 that need rotation in Secrets Manager and the other 95 in Parameter Store SecureString.

In short: Secrets don’t go in code, git, README, Dockerfile, or plaintext environment variables; they go in Secrets Manager or Parameter Store. If you need automatic rotation (RDS templates) and always-KMS, use Secrets Manager ($0.40 per secret); for configuration and secrets that don’t need rotation, use the free Parameter Store SecureString. From code, fetch with once-in-the-module-global-scope + Powertools caching, and ECS auto-injects into environment variables via the Task Definition’s secrets, but the Execution Role permission is mandatory. Don’t stamp plaintext secrets into IaC; use random_password or {{resolve:...}}.

Next chapter #

The next Chapter 21 Step Functions intro is the last of Part 3. It covers the way to bundle several Lambda / ECS / external API calls into one workflow. It puts together the AWS workflow engine: State machine / Task / Choice / Parallel / Map, Standard vs Express, Lambda orchestration, and error handling and retry.