App Deployment Skeleton

In Chapter 21 EKS cluster setup one empty EKS cluster was prepared. The nodes are up and the system Pods (CoreDNS, kube-proxy, VPC CNI, EBS CSI) are alive, but not a single line of our workload has gone on top of it yet. This chapter fills that empty space. We organize the sample backend service myshop-api into 9 manifests, auto-provision an ALB with the AWS Load Balancer Controller to create an external entry point, and wrap all those manifests into a Helm chart so they deploy to the two environments dev / prod with different values.

By the end of this chapter myshop-api is accessible from outside over HTTPS. It’s an empty shell that only returns 200 on /health/ready because it has no data store, but we fill that gap in the next Chapter 23 with RDS · External Secrets.

The standard manifest bundle of myshop-api #

We organize the standard bundle for putting one production workload on K8s into a single table.

The models we covered split out at the object level across Parts 1 ~ 3 come together inside this one table. This chapter is the stage of assembling those models into one production manifest bundle. Rather than the standalone usage of each object, we focus on how environment variables are injected when assembled together.

Namespace and ServiceAccount #

apiVersion: v1
kind: Namespace
metadata:
  name: myshop
  labels:
    name: myshop
    team: platform
    env: prod
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: myshop-api
  namespace: myshop
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/myshop-prod-api

The Namespace labels are the standard covered in Chapter 7 Namespace and labels. If you attach labels like team / env / cost-center consistently, Prometheus in Chapter 25 Monitoring · alerts uses them when grouping workloads, and the cost tracking of Chapter 28 Cost optimization uses the same labels as keys.

The ServiceAccount’s IRSA annotation is the annotation from Chapter 16 RBAC / ServiceAccount in depth. The myshop-api Pod automatically receives the permissions of the IAM Role written there while running as this ServiceAccount. The Role’s own definition is managed with the Terraform pattern organized in Chapter 21.

module "myshop_api_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.0"

  role_name = "myshop-prod-api"

  oidc_providers = {
    main = {
      provider_arn               = var.oidc_provider_arn
      namespace_service_accounts = ["myshop:myshop-api"]
    }
  }

  role_policy_arns = {
    secrets = aws_iam_policy.myshop_secrets_read.arn
  }
}

We write the ARN of the Role created by this Terraform into the ServiceAccount annotation. After that, the AWS SDK inside the Pod can call Secrets Manager · S3 · CloudWatch with this Role’s permissions. The trust constraint of namespace_service_accounts is the security boundary that binds it so only the myshop-api ServiceAccount in the myshop namespace can take on this Role.

ConfigMap and Secret #

apiVersion: v1
kind: ConfigMap
metadata:
  name: myshop-api
  namespace: myshop
data:
  LOG_LEVEL: "info"
  FEATURE_NEW_CHECKOUT: "true"
  CACHE_TTL_SECONDS: "300"
  AWS_REGION: "ap-northeast-2"
---
apiVersion: v1
kind: Secret
metadata:
  name: myshop-api
  namespace: myshop
type: Opaque
stringData:
  DATABASE_URL: "postgresql://will-be-replaced-by-external-secrets"

We don’t keep the Secret’s actual value in the manifest. In Chapter 23 DB integration we cover how the External Secrets Operator automatically syncs secrets in AWS Secrets Manager into K8s Secrets. In this chapter we only lay out the skeleton, and we weave it back together along with the secret-management model of Chapter 20 GitOps in Chapter 24 CI / CD pipeline.

Deployment — Pod replicas #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myshop-api
  namespace: myshop
  labels:
    app.kubernetes.io/name: myshop-api
    app.kubernetes.io/component: backend
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: myshop-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: myshop-api
        app.kubernetes.io/component: backend
    spec:
      serviceAccountName: myshop-api
      containers:
        - name: api
          image: 123456789012.dkr.ecr.ap-northeast-2.amazonaws.com/myshop-api:1.4.2
          ports:
            - containerPort: 8000
              name: http
          envFrom:
            - configMapRef:
                name: myshop-api
            - secretRef:
                name: myshop-api
          resources:
            requests:
              cpu: "200m"
              memory: "256Mi"
            limits:
              cpu: "1"
              memory: "512Mi"
          readinessProbe:
            httpGet:
              path: /health/ready
              port: http
            initialDelaySeconds: 5
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health/live
              port: http
            initialDelaySeconds: 30
            periodSeconds: 10
          startupProbe:
            httpGet:
              path: /health/live
              port: http
            failureThreshold: 30
            periodSeconds: 10
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: myshop-api

The standard elements of a production manifest are gathered here. Pointing out which chapter of this book each was unpacked in gives the following.

• strategy.rollingUpdate — the rolling update of Chapter 4 Deployment / ReplicaSet. With maxUnavailable: 0, the basic zero-downtime protection that takes down the old version only once the new version has come up sufficiently.
• envFrom — the bundled injection of Chapter 6 ConfigMap · Secret. It unpacks all keys of the ConfigMap and Secret into environment variables at once.
• The three probes — the three from Chapter 12 Health checks. readiness is immediate (starts after 5 seconds), liveness is conservative (starts after 30 seconds), and startup is the grace period for cases where initialization is long.
• resources — the requests / limits of Chapter 11 Resource requests and limits. This workload belongs to the Burstable QoS class where requests < limits.
• topologySpreadConstraints — spreads Pods across multiple AZs. Even if one AZ dies, the Pods of the other AZ survive.
• The app.kubernetes.io/ standard labels — the standard from Chapter 7. selector · monitoring · logging are all grouped by these labels.

Service — a fixed entry point inside the cluster #

apiVersion: v1
kind: Service
metadata:
  name: myshop-api
  namespace: myshop
  labels:
    app.kubernetes.io/name: myshop-api
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: myshop-api
  ports:
    - name: http
      port: 80
      targetPort: http
      protocol: TCP

type: ClusterIP is the key. The Service itself is a virtual IP accessible only inside the cluster, and external exposure is handled by the Ingress we’ll create next. It is exactly the model covered in Chapter 5 Service. The app’s external exposure is decided at the Ingress, not at the Service — this separation is the standard pattern in a production cluster, and the starting point of a security · operations model that keeps internal communication and external exposure in different layers.

AWS Load Balancer Controller — mapping Ingress to an ALB #

EKS’s Ingress is a standard K8s Ingress object, but to resolve that object into an actual ALB (AWS Application Load Balancer), a component called the AWS Load Balancer Controller has to be installed in the cluster. That component, touched on in Chapter 10 Ingress §“Per-cloud Ingress Controllers,” is covered in its full setup stage here.

helm repo add eks https://aws.github.io/eks-charts
helm repo update

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=myshop-prod \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

The ServiceAccount is created in advance with IRSA (the same pattern as the EBS CSI IRSA manifest of Chapter 21). This controller watches Ingress objects inside the cluster, and when an Ingress is created it provisions a matching ALB through the AWS API. It’s useful to remember the controller’s role as a bridge between Kubernetes and AWS — the same pattern repeats in the External Secrets Operator of Chapter 23 and the CloudWatch Container Insights of Chapter 25.

Ingress manifest #

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myshop-api
  namespace: myshop
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-northeast-2:123456789012:certificate/abc-123
    alb.ingress.kubernetes.io/healthcheck-path: /health/ready
    external-dns.alpha.kubernetes.io/hostname: api.myshop.example.com
spec:
  rules:
    - host: api.myshop.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myshop-api
                port:
                  number: 80

We organize the meaning of each annotation.

• scheme: internet-facing — an internet-exposed ALB. For internal use, set it to internal.
• target-type: ip — registers Pod IPs directly into the ALB target group. The ALB watches Pod IP changes directly.
• listen-ports / ssl-redirect — accepts only HTTPS and auto-redirects HTTP.
• certificate-arn — an ACM certificate. You issue the certificate for the domain bound to Route 53 through ACM in advance.
• healthcheck-path — the health check path the ALB sends to each Pod. It’s common to match it to the same path as the readiness probe of Chapter 12.
• external-dns.alpha.kubernetes.io/hostname — the external-dns component automatically registers this ALB’s DNS as an A record in Route 53.

About 1 ~ 2 minutes after you apply this manifest, the shape forms where a new ALB is up in the AWS console and the api.myshop.example.com domain is connected to that ALB.

HPA — automatic Pod adjustment by traffic #

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myshop-api
  namespace: myshop
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myshop-api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300

When average CPU usage goes over 60 %, it increases the Pod count, and when it drops, it decreases. It’s exactly the model of Chapter 13 Autoscaling. The stabilization window in behavior is the operational key — the asymmetry of setting scale-up fast at 30 seconds and scale-down conservatively at 5 minutes is the heart of it. It responds quickly to traffic spikes while keeping Pods from jittering down on a temporary dip.

PodDisruptionBudget — the availability floor #

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myshop-api
  namespace: myshop
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: myshop-api

What does this one manifest prevent? During a voluntary disruption like a node upgrade, node drain, or Cluster Autoscaler reclaiming a node, it guarantees that myshop-api Pods do not drop below 2 at the same time. It prevents a recurring Kubernetes operational incident — “Pods all went down at once during an EKS upgrade and caused downtime.” Why a PDB is essential is explained in the cluster upgrade procedure of Chapter 26 Operations checklist.

The key to a PDB is the qualifier voluntary. Involuntary disruptions like a node’s sudden failure cannot be prevented by a PDB — that case is handled by topologySpreadConstraints and a multi-AZ node group.

Bundling with a Helm chart #

The 9 manifests we’ve written so far work on their own. But to deploy the same workload to dev / prod with different values, the burden of replicating the manifests per environment arises. This is where Helm comes in. The position of Helm touched on in Chapter 18 CRD and Operator §“Bundling with a Helm chart” is covered as a full production chart here.

Chart structure #

charts/myshop-api/
├── Chart.yaml
├── values.yaml             # defaults
├── values-dev.yaml         # dev override
├── values-prod.yaml        # prod override
└── templates/
    ├── _helpers.tpl
    ├── namespace.yaml
    ├── serviceaccount.yaml
    ├── configmap.yaml
    ├── secret.yaml
    ├── deployment.yaml
    ├── service.yaml
    ├── ingress.yaml
    ├── hpa.yaml
    └── pdb.yaml

Chart.yaml #

apiVersion: v2
name: myshop-api
description: myshop API service
type: application
version: 0.1.0
appVersion: "1.4.2"
maintainers:
  - name: platform-team
    email: platform@myshop.example.com

version is the version of the chart itself, and appVersion is the version of the application that chart deploys. The reason they’re separated is that the chart manifests often stay the same while only the image tag changes.

values.yaml — defaults #

image:
  repository: 123456789012.dkr.ecr.ap-northeast-2.amazonaws.com/myshop-api
  tag: "1.4.2"
  pullPolicy: IfNotPresent

replicaCount: 3

resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    cpu: 1
    memory: 512Mi

serviceAccount:
  create: true
  name: myshop-api
  irsaRoleArn: ""

config:
  LOG_LEVEL: info
  FEATURE_NEW_CHECKOUT: "true"
  CACHE_TTL_SECONDS: "300"

ingress:
  enabled: true
  host: api.myshop.example.com
  certificateArn: ""

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 20
  targetCPUUtilizationPercentage: 60

pdb:
  enabled: true
  minAvailable: 2

values-prod.yaml — per-environment override #

replicaCount: 5

resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 2
    memory: 1Gi

serviceAccount:
  irsaRoleArn: arn:aws:iam::123456789012:role/myshop-prod-api

config:
  LOG_LEVEL: warn

ingress:
  host: api.myshop.example.com
  certificateArn: arn:aws:acm:ap-northeast-2:123456789012:certificate/abc-123

autoscaling:
  minReplicas: 5
  maxReplicas: 50

Template — example of one Deployment #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "myshop-api.fullname" . }}
  labels:
    {{- include "myshop-api.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "myshop-api.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "myshop-api.selectorLabels" . | nindent 8 }}
    spec:
      serviceAccountName: {{ .Values.serviceAccount.name }}
      containers:
        - name: api
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          envFrom:
            - configMapRef:
                name: {{ include "myshop-api.fullname" . }}
            - secretRef:
                name: {{ include "myshop-api.fullname" . }}
          resources:
            {{- toYaml .Values.resources | nindent 12 }}

Helpers like {{ include "myshop-api.fullname" . }} are defined in _helpers.tpl. It’s a standard Helm chart pattern, and if you make the skeleton with helm create it comes in automatically.

Deploy — same chart, different environments #

helm upgrade --install myshop-api charts/myshop-api \
  -n myshop --create-namespace \
  -f charts/myshop-api/values.yaml \
  -f charts/myshop-api/values-dev.yaml

helm upgrade --install myshop-api charts/myshop-api \
  -n myshop --create-namespace \
  -f charts/myshop-api/values.yaml \
  -f charts/myshop-api/values-prod.yaml

The same chart applies differently per environment. This single command is replaced by the ArgoCD Application sync of Chapter 24 CI / CD pipeline, but it remains valid for manual deployment during the development stage. The git single-source model of Chapter 20 GitOps leads naturally into this Helm chart’s directory.

cert-manager and external-dns — auxiliary components #

If, instead of issuing · renewing ACM certificates yourself, you want to automatically obtain Let’s Encrypt certificates inside the cluster, you adopt cert-manager, and to automatically manage Route 53 DNS records via manifests, you adopt external-dns alongside it.

helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager \
  -n cert-manager --create-namespace \
  --set installCRDs=true

helm install external-dns external-dns/external-dns \
  -n external-dns --create-namespace \
  --set provider=aws \
  --set aws.region=ap-northeast-2 \
  --set "domainFilters[0]=myshop.example.com"

This is the standard setup of a production cluster. Whenever a domain is added, it’s done with one annotation line in the manifest instead of opening the Route 53 console. cert-manager isn’t needed when you use ACM certificates, but in multi-cluster · multi-cloud environments cert-manager is more consistent. The ServiceAccounts of both components also have to receive an IAM Role with the same pattern as the EBS CSI IRSA of Chapter 21 before they can change Route 53 records.

Checks after the first deploy #

These are the items to check right after deploying.

kubectl get all -n myshop
kubectl get hpa -n myshop
kubectl get pdb -n myshop
kubectl get ingress -n myshop

kubectl describe ingress myshop-api -n myshop | grep "Address"

curl -i https://api.myshop.example.com/health/ready

These four steps confirm whether everything from Pod · HPA · Ingress · external entry is working. When a 200 response comes back, it’s the moment the cluster’s first workload has entered production. If the ALB’s Address is empty or a 503 comes back, it helps to look ahead at the Ingress debugging section of Chapter 27 kubectl debugging patterns.

Exercises #

1. Lay out this chapter’s 9 manifests in one directory and apply them in order to the dev EKS cluster stood up in Chapter 21. In the output of kubectl get all -n myshop, check that the Deployment is 3 / 3, the ReplicaSet matches, all Pods are Running, and the Service has a ClusterIP assigned. It takes about 1 ~ 2 minutes for the Ingress’s ALB Address to come up, and during this time observe along in the AWS console as the ALB · TargetGroup · Route 53 A record get created in turn.
2. Convert this chapter’s 9 manifests into one Helm chart bundle. After making the skeleton with helm create myshop-api, organize how the three files values.yaml · values-dev.yaml · values-prod.yaml branch, and pre-verify with helm template whether dev’s replicaCount and prod’s replicaCount apply differently. Note in one paragraph how this pre-verification connects to the ArgoCD diff model of Chapter 24 CI / CD.
3. Switch the whenUnsatisfiable of topologySpreadConstraints between the two values ScheduleAnyway and DoNotSchedule and observe how the Pod distribution changes. When there are only two AZs but replicaCount is 3, compare in one paragraph which option creates which trade-off, against your own operational scenario. Also point out how the decision of single AZ vs multi-AZ in the VPC module of Chapter 21 shakes the result of this option.

In one line: the standard for one production workload is the 9 objects Namespace · ServiceAccount · ConfigMap · Secret · Deployment · Service · Ingress · HPA · PodDisruptionBudget, and if you abstract this bundle into a Helm chart, the same chart unfolds to dev / prod with different values. EKS’s Ingress is actually resolved to an ALB by the AWS Load Balancer Controller, and an ACM certificate + Route 53 + external-dns bind the three pieces of domain · TLS · DNS with one annotation line in the manifest.

Next chapter #

At this point myshop-api is accessible from outside over HTTPS, but it’s an empty shell that only returns 200 on /health/ready because it has no data store. In the next chapter we fill that empty part.

In Chapter 23 DB integration we cover the connection with RDS PostgreSQL, the path of safely injecting the DB password with Secrets Manager, the flow of syncing K8s Secrets with the cloud secret store using the External Secrets Operator, and the operational principles of the connection pool.