K8s Practice #3: DB Integration — RDS / Secrets Manager / External Secrets / Connection Pool

The third post in the K8s Practice series. In #2, myshop-api became externally accessible, but there is still nowhere for data to persist. This post fills that gap. We bring up RDS PostgreSQL with Terraform, store the master password in AWS Secrets Manager, auto-sync it into a K8s Secret with External Secrets Operator, grant permissions via IRSA without static credentials, add PgBouncer as a connection pool, and automate schema migration as a Job.

This series is K8s Practice, 6 posts.

• #1 EKS Cluster Setup — Terraform / eksctl / IRSA / Addons
• #2 App deployment skeleton — Deployment / Service / Ingress / Helm
• #3 DB integration — RDS / Secrets Manager / External Secrets / connection pool ← this post
• #4 CI/CD pipeline — GitHub Actions / ECR / ArgoCD
• #5 Monitoring/alarming — Prometheus / CloudWatch / Alertmanager
• #6 Operations checklist — upgrades / backup,recovery / cost / security

RDS — bringing up PostgreSQL with Terraform #

Running a PostgreSQL StatefulSet inside K8s is an option, but the standard for operational environments is managed RDS. Backups, Multi-AZ failover, patches, and monitoring are all offloaded to AWS, leaving us to focus solely on cluster operations.

Terraform module #

module "rds" {
  source  = "terraform-aws-modules/rds/aws"
  version = "~> 6.0"

  identifier = "myshop-${var.env}"

  engine            = "postgres"
  engine_version    = "16.3"
  family            = "postgres16"
  major_engine_version = "16"
  instance_class    = var.env == "prod" ? "db.m6g.large" : "db.t4g.medium"

  allocated_storage     = 50
  max_allocated_storage = 500
  storage_type          = "gp3"
  storage_encrypted     = true

  db_name  = "myshop"
  username = "myshop_admin"
  port     = 5432

  manage_master_user_password = true
  master_user_secret_kms_key_id = aws_kms_key.rds.arn

  multi_az               = var.env == "prod"
  db_subnet_group_name   = var.db_subnet_group_name
  vpc_security_group_ids = [aws_security_group.rds.id]

  backup_retention_period = var.env == "prod" ? 30 : 7
  backup_window           = "03:00-04:00"
  maintenance_window      = "Mon:04:00-Mon:05:00"

  performance_insights_enabled = true
  monitoring_interval          = 60
  monitoring_role_arn          = aws_iam_role.rds_monitoring.arn

  enabled_cloudwatch_logs_exports = ["postgresql"]

  deletion_protection = var.env == "prod"
  skip_final_snapshot = var.env != "prod"
}

Key options to flag:

• manage_master_user_password = true — RDS generates the master password itself and stores it in Secrets Manager, so no person ever sees it.
• multi_az — prod with Multi-AZ for failover capability, dev with single AZ for cost savings.
• storage_encrypted — KMS encryption. Operational standard.
• performance_insights_enabled — PostgreSQL query performance analysis. Almost no impact on RDS itself’s cost.
• deletion_protection — turn on for prod. Prevents terraform destroy accidents.

Security group — only EKS nodes can access #

resource "aws_security_group" "rds" {
  name_prefix = "myshop-${var.env}-rds-"
  vpc_id      = var.vpc_id
}

resource "aws_security_group_rule" "rds_from_eks" {
  type                     = "ingress"
  from_port                = 5432
  to_port                  = 5432
  protocol                 = "tcp"
  security_group_id        = aws_security_group.rds.id
  source_security_group_id = var.eks_node_security_group_id
  description              = "Allow from EKS worker nodes"
}

Port 5432 accepts connections only from the EKS node security group; RDS is not directly accessible from anywhere else — this is the operational standard. When temporary direct access is needed, use the bastion or SSM Session Manager approach covered in #6.

Master password — placed in Secrets Manager #

With manage_master_user_password = true enabled, RDS auto-creates the password and stores it in Secrets Manager in the following format.

{
  "username": "myshop_admin",
  "password": "<RDS-generated random>",
  "engine": "postgres",
  "host": "myshop-prod.abcdef.ap-northeast-2.rds.amazonaws.com",
  "port": 5432,
  "dbname": "myshop"
}

How a Pod inside K8s reads this secret is the next step.

External Secrets Operator — syncing K8s Secret with cloud secret #

This is one of the three standard models for safely managing secrets discussed in Advanced #6 GitOps. It is the controller that auto-syncs AWS Secrets Manager secrets into K8s Secrets.

Installation #

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace \
  --set installCRDs=true

After installation, two new CRDs are registered in the cluster — ClusterSecretStore and ExternalSecret.

Secrets Manager access permission via IRSA #

Attach an IAM Role granting Secrets Manager read permissions to the External Secrets Operator’s ServiceAccount via IRSA.

data "aws_iam_policy_document" "secrets_read" {
  statement {
    actions = [
      "secretsmanager:GetSecretValue",
      "secretsmanager:DescribeSecret",
    ]
    resources = [
      "arn:aws:secretsmanager:${var.region}:${var.account_id}:secret:rds!cluster-myshop-${var.env}/*",
      "arn:aws:secretsmanager:${var.region}:${var.account_id}:secret:myshop/${var.env}/*",
    ]
  }
}

resource "aws_iam_policy" "secrets_read" {
  name   = "myshop-${var.env}-external-secrets-read"
  policy = data.aws_iam_policy_document.secrets_read.json
}

module "external_secrets_irsa" {
  source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.0"

  role_name = "myshop-${var.env}-external-secrets"

  oidc_providers = {
    main = {
      provider_arn = var.oidc_provider_arn
      namespace_service_accounts = [
        "external-secrets:external-secrets"
      ]
    }
  }

  role_policy_arns = {
    main = aws_iam_policy.secrets_read.arn
  }
}

The ARN pattern in resources is the key — only granting access to myshop secrets, blocking other teams’ secrets from being read. Least privilege principle.

ClusterSecretStore — definition of secret source #

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-northeast-2
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets

This object declares cluster-wide that “secrets are pulled from one place — AWS Secrets Manager.” The auth.jwt.serviceAccountRef points to the ServiceAccount with IRSA attached. External Secrets Operator calls STS’s AssumeRoleWithWebIdentity using that ServiceAccount’s projected token to obtain Secrets Manager permissions.

ExternalSecret — fetching secrets via manifest #

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: myshop-api-db
  namespace: myshop
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: myshop-api-db
    creationPolicy: Owner
    template:
      data:
        DATABASE_URL: "postgresql://{{ .username }}:{{ .password }}@{{ .host }}:{{ .port }}/{{ .dbname }}?sslmode=require"
  data:
    - secretKey: username
      remoteRef:
        key: rds!cluster-myshop-prod
        property: username
    - secretKey: password
      remoteRef:
        key: rds!cluster-myshop-prod
        property: password
    - secretKey: host
      remoteRef:
        key: rds!cluster-myshop-prod
        property: host
    - secretKey: port
      remoteRef:
        key: rds!cluster-myshop-prod
        property: port
    - secretKey: dbname
      remoteRef:
        key: rds!cluster-myshop-prod
        property: dbname

What happens with this single manifest:

1. External Secrets Operator fetches the rds!cluster-myshop-prod secret from Secrets Manager every hour
2. Pulls the 5 fields of that secret (username, password, host, port, dbname)
3. Assembles those values into Connection String format via template.data.DATABASE_URL
4. Creates a K8s Secret named myshop-api-db storing it as a single DATABASE_URL key

The myshop-api Pod just needs to inject this Secret as environment variables via envFrom.

envFrom:
  - configMapRef:
      name: myshop-api
  - secretRef:
      name: myshop-api-db   # ← auto-synced by External Secrets

The DATABASE_URL left as a placeholder in #2 is filled with the real value at this point. And when the RDS password is rotated, the K8s Secret is also auto-updated with the new value within an hour.

The need for Pod restart #

Even when a K8s Secret is updated, environment variables inside a running Pod are not auto-updated. Values injected via envFrom are fixed at Pod start time. After a password rotation, Pods must be restarted for the new password to take effect.

kubectl rollout restart deployment/myshop-api -n myshop

External Secrets integrates with a separate component called Reloader that can automatically trigger a rollout restart when a Secret changes. It is frequently included as part of the standard setup for operational clusters.

Connection pool — why, and PgBouncer #

If 5 myshop-api Pods are running and each holds 50 PostgreSQL connections, the cluster occupies 250 RDS connections in total. Each RDS instance class has a max_connections limit — db.t4g.medium defaults to about 100, db.m6g.large to about 800. In environments where Pods scale via HPA, this limit is reached quickly.

A connection pooler enters this gap. The most standard is PgBouncer.

The role of PgBouncer #

[5 myshop-api Pods] ──→ [2 PgBouncer] ──→ [RDS PostgreSQL]
   50 conn each                            20 backend conn

myshop-api connects to PgBouncer, which multiplexes those connections into a small number of backend connections to RDS. In transaction pooling mode, a single PostgreSQL connection handles short transactions from multiple clients in sequence, making connection usage highly efficient.

Manifest #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pgbouncer
  namespace: myshop
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: pgbouncer
  template:
    metadata:
      labels:
        app.kubernetes.io/name: pgbouncer
    spec:
      containers:
        - name: pgbouncer
          image: edoburu/pgbouncer:1.22.1
          ports:
            - containerPort: 6432
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: myshop-api-db
                  key: DATABASE_URL
            - name: POOL_MODE
              value: transaction
            - name: MAX_CLIENT_CONN
              value: "1000"
            - name: DEFAULT_POOL_SIZE
              value: "20"
            - name: SERVER_RESET_QUERY
              value: "DISCARD ALL"
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 128Mi
---
apiVersion: v1
kind: Service
metadata:
  name: pgbouncer
  namespace: myshop
spec:
  selector:
    app.kubernetes.io/name: pgbouncer
  ports:
    - port: 5432
      targetPort: 6432

myshop-api’s DATABASE_URL is now overridden per environment to point not directly at RDS but at pgbouncer.myshop.svc.cluster.local:5432.

config:
  DATABASE_URL: "postgresql://myshop_admin:$(DB_PASSWORD)@pgbouncer.myshop.svc.cluster.local:5432/myshop?sslmode=disable"

One trap of transaction pooling #

In PgBouncer’s transaction pooling mode, PostgreSQL’s prepared statements, advisory locks, and session variables cannot be safely used. When a transaction ends, the backend connection is handed to another client, so session-level state is not preserved. When ORMs automatically use prepared statements (as some SQLAlchemy configurations do), that feature must be disabled or session pooling mode must be used instead.

RDS Proxy is a managed alternative that plays the same role. AWS operates it and IAM authentication integration is deep, but it adds cost and the transaction pooling limitation applies equally.

Schema migration — the Job pattern #

Moving database schema to a new version is resolved as a Job in K8s.

Job manifest #

apiVersion: batch/v1
kind: Job
metadata:
  name: myshop-api-migrate-1.4.2
  namespace: myshop
spec:
  backoffLimit: 3
  ttlSecondsAfterFinished: 86400
  template:
    spec:
      serviceAccountName: myshop-api
      restartPolicy: OnFailure
      containers:
        - name: migrate
          image: 123456789012.dkr.ecr.ap-northeast-2.amazonaws.com/myshop-api:1.4.2
          command: ["alembic", "upgrade", "head"]
          envFrom:
            - secretRef:
                name: myshop-api-db

This follows the Job pattern covered in Intermediate #1. Use the same image as myshop-api and simply swap the command for the migration tool (e.g., alembic, flyway, golang-migrate). ttlSecondsAfterFinished: 86400 auto-deletes the Job object after 24 hours.

Coupling with deployment — Helm hook #

New version Pods should only come up after migration finishes. The pattern that enforces order is Helm hook.

metadata:
  annotations:
    "helm.sh/hook": pre-upgrade,pre-install
    "helm.sh/hook-weight": "0"
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded

A Job with this annotation runs before the new Deployment during helm upgrade, and the upgrade only proceeds if the Job succeeds. A migration failure naturally cascades into a deployment failure, preventing new code from starting against a broken schema.

Comparison with initContainer #

There is also the pattern of running migration as an initContainer. But if myshop-api has 5 Pods, migration would be attempted 5 times simultaneously. Some migration tools (alembic, flyway) prevent duplicate runs via advisory locks, but from K8s’s perspective, running migration once as a Job is the cleaner approach.

IAM authentication — the path of removing the password itself #

The most advanced pattern is RDS IAM authentication. It eliminates the password entirely, accessing RDS directly with IRSA tokens.

import boto3
import psycopg2

rds_client = boto3.client('rds')
token = rds_client.generate_db_auth_token(
    DBHostname='myshop-prod.abcdef.ap-northeast-2.rds.amazonaws.com',
    Port=5432,
    DBUsername='myshop_app',
    Region='ap-northeast-2'
)

conn = psycopg2.connect(
    host='myshop-prod.abcdef.ap-northeast-2.rds.amazonaws.com',
    port=5432,
    user='myshop_app',
    password=token,  # ← not a password but an IAM token
    dbname='myshop',
    sslmode='require'
)

generate_db_auth_token creates a 15-minute token from IAM credentials. That token goes in the position of the PostgreSQL password. No need to worry about password rotation, and all access is recorded in CloudTrail.

But there are downsides too.

• New tokens must be obtained every 15 minutes, making integration with connection pools tricky.
• On the PostgreSQL side, user and grant management for IAM authentication (the rds_iam group) is required.
• Using it together with PgBouncer transaction pooling is very difficult.

The traditional password + Secrets Manager + External Secrets model offers the best balance between operational burden and security. IAM authentication is worth considering for environments that require one additional layer of strictness.

Checks after first connection #

Commands to check at the point both migration Job and myshop-api deployment are done.

kubectl get secret myshop-api-db -n myshop -o jsonpath='{.data.DATABASE_URL}' | base64 -d

kubectl exec -it deployment/myshop-api -n myshop -- \
  psql "$DATABASE_URL" -c "SELECT version();"

kubectl exec -it deployment/pgbouncer -n myshop -- \
  psql -p 6432 pgbouncer -c "SHOW POOLS;"

kubectl get externalsecret myshop-api-db -n myshop
kubectl describe externalsecret myshop-api-db -n myshop

These four commands verify that secret sync, DB connection, and connection pool are all functioning normally. When PostgreSQL’s version() returns a result, myshop-api is ready to handle real data.

Closing #

We followed one complete cycle of wiring up myshop-api’s data store. We brought up RDS PostgreSQL with Terraform, stored the master password in Secrets Manager, auto-synced that secret into a K8s Secret via External Secrets Operator, granted permissions via IRSA without static credentials, added PgBouncer as the connection pool, and automated schema migration with a Helm hook-based Job. We also noted the more advanced IAM authentication pattern. At this point myshop-api is a complete service with an external entry point, running workload, and a live DB connection, but deploying new versions still requires a manual helm upgrade. The next post automates that gap — covering a full GitOps pipeline where GitHub Actions builds containers and pushes to ECR, and ArgoCD detects manifest changes and syncs to the cluster automatically.