AWS Advanced #6: Secrets Manager / Parameter Store

DB passwords, external API keys, OAuth client secrets — where should they live? Storing them in code or git is absolutely out, and plaintext env vars are dangerous. AWS has two services for this — Secrets Manager and SSM Parameter Store.

This post covers their differences, automatic rotation, the fetch-from-code patterns, ECS / Lambda integration, IaC wiring, and cost — secret and configuration management, all in one piece.

Where secrets must not go #

A continuation of Basics #6 Security fundamentals. Absolutely don’t:

• Plaintext in code (PASSWORD = "abc123")
• .env in git (deleted but forever in history)
• Pasted into README, Slack, wiki, email
• Dockerfile env vars / build args (baked into image layers)
• Plaintext environment in ECS Task Definitions (visible in console / IaC / logs)

Where they should go: Secrets Manager or SSM Parameter Store.

One-line comparison #

One-liner decision guide #

DB password / external API key + auto-rotation needed → Secrets Manager
   │
General config (DB host, region, feature flag) → Parameter Store
   │
No-rotation secret (e.g. external API key) → Parameter Store SecureString (cheap)

Secrets Manager #

Create #

aws secretsmanager create-secret \
  --name myapp/prod/db \
  --description "Postgres for myapp prod" \
  --secret-string '{
    "username": "myapp",
    "password": "very-strong-secret",
    "host": "myapp-prod.abc123.ap-northeast-2.rds.amazonaws.com",
    "port": 5432,
    "dbname": "myapp"
  }'

ARN returned:

arn:aws:secretsmanager:ap-northeast-2:123456789012:secret:myapp/prod/db-AbCdEf

The 6 random chars at the end (AbCdEf) prevent secret enumeration. Use wildcards in IAM policies like arn:.../myapp/prod/db-*.

Read — boto3 #

import boto3, json

client = boto3.client("secretsmanager")
resp = client.get_secret_value(SecretId="myapp/prod/db")
creds = json.loads(resp["SecretString"])

print(creds["username"], creds["password"])

Caching — fetching every call costs #

API calls are billed, so calling get_secret_value on every request is both expensive and slow.

Fetch once at module level + cache pattern:

import boto3, json

# In Lambda's INIT phase, runs once
_secrets = None

def get_db_creds():
    global _secrets
    if _secrets is None:
        resp = boto3.client("secretsmanager").get_secret_value(
            SecretId="myapp/prod/db")
        _secrets = json.loads(resp["SecretString"])
    return _secrets

def handler(event, context):
    creds = get_db_creds()
    ...

Powertools’ Parameters does cache / TTL / multi-secret in one line:

from aws_lambda_powertools.utilities import parameters

# 5-minute TTL cache
creds = parameters.get_secret("myapp/prod/db", transform="json",
                              max_age=300)

Auto-rotation #

Secrets Manager’s biggest feature. Periodically generates a new password and updates the user’s password in the DB.

aws secretsmanager rotate-secret \
  --secret-id myapp/prod/db \
  --rotation-lambda-arn arn:aws:lambda:ap-northeast-2:...:SecretsManagerRDSPostgreSQLRotationSingleUser \
  --rotation-rules AutomaticallyAfterDays=30

AWS provides managed rotation Lambdas as templates (RDS / Redshift / DocumentDB / Aurora). Rotates every 30 days.

The 4 rotation phases (for understanding) #

1. createSecret      → generate new password, store as AWSPENDING
2. setSecret         → apply new password to DB
3. testSecret        → connect with the new password
4. finishSecret      → AWSPENDING → AWSCURRENT, old → AWSPREVIOUS

Code always pulls AWSCURRENT (default). During rotation AWSPREVIOUS is briefly valid too.

Single-user vs Multi-user rotation #

• Single-user: swap one user’s password. Brief connection blip possible at swap moment
• Multi-user: alternate between two users. Zero-downtime rotation. Recommended for production

Parameter Store (SSM) #

Standard vs Advanced #

Start with Standard.

Create #

aws ssm put-parameter \
  --name /myapp/prod/aws-region \
  --value ap-northeast-2 \
  --type String

# Hierarchy (free — structure with '/')
aws ssm put-parameter \
  --name /myapp/prod/feature/new-checkout \
  --value enabled \
  --type String

aws ssm put-parameter \
  --name /myapp/prod/external-api-key \
  --value sk_live_abc123 \
  --type SecureString \
  --key-id alias/aws/ssm

SecureString is KMS-encrypted. The AWS-managed key (alias/aws/ssm) is free.

Read #

import boto3

ssm = boto3.client("ssm")

# single
resp = ssm.get_parameter(
    Name="/myapp/prod/external-api-key",
    WithDecryption=True
)
api_key = resp["Parameter"]["Value"]

# multi (hierarchy)
resp = ssm.get_parameters_by_path(
    Path="/myapp/prod/",
    Recursive=True,
    WithDecryption=True
)
for p in resp["Parameters"]:
    print(p["Name"], p["Value"])

from aws_lambda_powertools.utilities import parameters

# single
api_key = parameters.get_parameter(
    "/myapp/prod/external-api-key", decrypt=True, max_age=300)

# hierarchy
all_params = parameters.get_parameters(
    "/myapp/prod/", decrypt=True, max_age=300)

Versioning #

Each put to the same name creates a new version — 1, 2, 3, and so on. You can roll back to any previous value.

# see version 5
aws ssm get-parameter-history --name /myapp/prod/db-host

# revert to version 3 (copy as new version)
aws ssm put-parameter --name /myapp/prod/db-host \
  --value $(aws ssm get-parameter --name /myapp/prod/db-host:3 \
    --query 'Parameter.Value' --output text) --overwrite

ECS / Lambda integration #

The most common pattern. Inject secrets straight into Task Definition / function env vars.

ECS Task Definition #

{
  "containerDefinitions": [
    {
      "name": "web",
      "image": "...",
      "environment": [
        {"name": "ENV", "value": "production"}
      ],
      "secrets": [
        {
          "name": "DATABASE_URL",
          "valueFrom": "arn:aws:secretsmanager:ap-northeast-2:123456789012:secret:myapp/prod/db-AbCdEf"
        },
        {
          "name": "STRIPE_KEY",
          "valueFrom": "arn:aws:ssm:ap-northeast-2:123456789012:parameter/myapp/prod/stripe-key"
        }
      ]
    }
  ]
}

ECS fetches the secret right before Task start and injects it as a container env var. Inside code you just read os.environ["DATABASE_URL"].

Permissions required (#1 Execution Role):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue"],
      "Resource": "arn:aws:secretsmanager:ap-northeast-2:123456789012:secret:myapp/prod/*"
    },
    {
      "Effect": "Allow",
      "Action": ["ssm:GetParameters"],
      "Resource": "arn:aws:ssm:ap-northeast-2:123456789012:parameter/myapp/prod/*"
    },
    {
      "Effect": "Allow",
      "Action": ["kms:Decrypt"],
      "Resource": "arn:aws:kms:ap-northeast-2:123456789012:key/*"
    }
  ]
}

Specific JSON field only #

When a Secrets Manager secret is JSON, ECS can pull a specific field as an env var:

{
  "name": "DB_PASSWORD",
  "valueFrom": "arn:aws:secretsmanager:...:myapp/prod/db-AbCdEf:password::"
}

The :password:: at the end of the ARN extracts the password field. Between the first : and last : is the JSON key; after the last : is the version.

Lambda secrets #

Same pattern. Don’t put secrets in function env vars; fetch via boto3 in code. Or use the AWS Parameters and Secrets Lambda Extension as a caching sidecar — call localhost:2773 from inside the function.

import urllib.request, json, os

def get_secret(name):
    url = f"http://localhost:2773/secretsmanager/get?secretId={name}"
    req = urllib.request.Request(url,
        headers={"X-Aws-Parameters-Secrets-Token": os.environ["AWS_SESSION_TOKEN"]})
    with urllib.request.urlopen(req) as resp:
        data = json.loads(resp.read())
    return json.loads(data["SecretString"])

The Extension auto-caches (configurable TTL) — no caching code needed.

IaC wiring #

Terraform #

resource "aws_secretsmanager_secret" "db" {
  name = "myapp/prod/db"
}

resource "aws_secretsmanager_secret_version" "db" {
  secret_id = aws_secretsmanager_secret.db.id
  secret_string = jsonencode({
    username = "myapp"
    password = random_password.db.result
    host     = aws_db_instance.main.address
  })
}

resource "random_password" "db" {
  length  = 32
  special = false
}

Don’t put the password value directly in IaC. Generate via random_password, or let AWS rotation manage it.

CloudFormation dynamic reference #

Resources:
  MyDB:
    Type: AWS::RDS::DBInstance
    Properties:
      MasterUsername: '{{resolve:secretsmanager:myapp/prod/db:SecretString:username}}'
      MasterUserPassword: '{{resolve:secretsmanager:myapp/prod/db:SecretString:password}}'

The {{resolve:...}} pattern fetches secrets / parameters inside the stack. Plaintext doesn’t end up in the template / logs.

Cost #

Secrets Manager #

• $0.40 per secret / month
• $0.05 per 10,000 API calls

10 secrets + ~100 calls/day = $4 + ~0 = $4 / month

Parameter Store (Standard) #

• Free (up to 40 req/s throughput)
• KMS API call cost (~$0.03 / 10,000) when SecureString

10 SecureStrings + ~100 calls/day = nearly free

The price gap #

Without rotation / managed integration, Parameter Store SecureString is dramatically cheaper. 100 secrets: Secrets Manager ~$40/month vs Parameter Store ~$0.

Splitting secrets and config #

Recommended ops pattern:

Secrets Manager
├── /myapp/prod/db          (auto-rotation)
└── /myapp/prod/jwt-signing (auto-rotation)

Parameter Store
├── /myapp/prod/db-host
├── /myapp/prod/aws-region
├── /myapp/prod/feature/new-checkout   (feature flag)
├── /myapp/prod/log-level
└── /myapp/prod/external-api-key       (SecureString — secret without rotation)

Only the rotation-required core secrets in Secrets Manager. Everything else in Parameter Store for cost savings.

Cross-Account / Cross-Region #

Resource Policy #

Allow access from another account:

aws secretsmanager put-resource-policy \
  --secret-id myapp/prod/db \
  --resource-policy '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::222222222222:root"},
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*"
    }]
  }'

Replication #

Auto-replicate to other regions (multi-region ops):

aws secretsmanager replicate-secret-to-regions \
  --secret-id myapp/prod/db \
  --add-replica-regions Region=us-east-1

When the primary rotates, replicas sync automatically.

Common pitfalls #

1) Secret not injected to ECS Task #

Task start fails with “ResourceInitializationError … not authorized to perform secretsmanager:GetSecretValue”. Execution Role permission missing. Or for SecureString, missing kms:Decrypt on the KMS key.

2) boto3 call per request #

Calling get_secret_value inside the handler on every invocation floods the API with calls, adding cost and latency. Fetch at module level and cache the result.

3) JSON field extraction ARN typo #

arn:...:db-AbCdEf:password:: — one wrong colon and you silently get an empty string. Copy from the console.

4) Plaintext secrets in IaC #

secret_string = "abc123" in Terraform — plaintext in state. Always random_password or external input, and encrypt state (S3 + KMS).

5) Old password failures during rotation #

A connection made before rotation may break briefly. Multi-user rotation + connection retry.

6) Unintended secret name disclosure #

The 6-char random suffix (-AbCdEf) is hard to guess in logs / CloudTrail, but the secret name itself (myapp/prod/db) is visible to anyone with console access. Reduce permissions (per-secret IAM) to shrink the exposure surface.

7) Parameter Store throughput cap #

Hitting Standard’s 40 req/s → throttling. Heavy hot-path use → Advanced or caching.

Wrap-up #

Here is what this post covered:

• Where secrets must not go — code, git, README, Dockerfile, plaintext env vars
• Secrets Manager — auto-rotation (RDS templates), always KMS, $0.40 per secret
• Parameter Store — config + secrets. Standard free, SecureString free
• Decision guide — auto-rotation needed → Secrets Manager. Otherwise → Parameter Store SecureString
• boto3 + cache — module-level fetch + Powertools’ parameters
• The 4 rotation phases — createSecret → setSecret → testSecret → finishSecret. Multi-user is zero-downtime
• ECS integration — Task Definition secrets auto-injects env vars. Execution Role permission required
• Specific JSON field — :fieldName:: at the ARN end
• Lambda Extension — Parameters and Secrets Lambda Extension as caching sidecar
• IaC — random_password generation, CloudFormation {{resolve:...}}
• Secret / config split — only rotation-required core in Secrets Manager
• Cross-Account / Cross-Region — Resource Policy / Replication
• Pitfalls — Execution Role permission, call flooding, ARN typo, IaC plaintext, rotation drops, throughput cap

Up next — Step Functions #

The series finale covers tying multiple Lambda / ECS / external API calls into a single workflow.

#7 Step Functions covers state machine / Task / Choice / Parallel, Standard vs Express, Lambda orchestration, error handling and retry — the AWS workflow, all in one piece.