K8s Practice #2: App Deployment Skeleton — Deployment / Service / Ingress / Helm

10 min read

The second post in the K8s Practice series. We brought up an EKS cluster in #1, but the workload still hasn’t been placed on top of it. This post is one cycle of putting the imaginary service myshop-api on that empty cluster. We write the bundle from Deployment to Ingress as manifests, auto-create an ALB with AWS Load Balancer Controller to establish the external entry point, and abstract the whole bundle as a Helm chart so the same chart deploys to dev and prod with different values.

This series is K8s Practice, 6 posts.

Looking at myshop-api’s manifests as one bundle #

To put myshop-api on EKS, the following objects are needed.

ObjectRole
NamespaceIsolation unit binding workloads
ServiceAccountPod’s ID + IRSA attachment point (DB connection in #3)
ConfigMapEnvironment config (log level, feature flags, etc.)
SecretDB password, etc. (actual values filled via External Secrets in #3)
DeploymentActual workload — Pod replicas
ServiceFixed virtual IP for inside-cluster access
IngressExternal entry point (resolved as ALB)
HorizontalPodAutoscalerCPU-based Pod count auto-adjustment
PodDisruptionBudgetMinimum availability guarantee during node upgrades

These 9 objects form the standard bundle for one operational workload. The sections below walk through each one.

Namespace and ServiceAccount #

00-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: myshop
  labels:
    name: myshop
    team: platform
    env: prod
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: myshop-api
  namespace: myshop
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/myshop-prod-api

The Namespace’s labels are the labels covered in Basics #7. Consistently attaching labels like team / env / cost-center lets #5’s Prometheus reuse them as is when grouping workloads.

The ServiceAccount’s IRSA annotation is the annotation covered in Advanced #2. The myshop-api Pod automatically receives the permissions of the IAM Role written here while running with this ServiceAccount. The Role’s definition itself is managed in Terraform.

terraform/modules/myshop-api/irsa.tf
module "myshop_api_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.0"

  role_name = "myshop-prod-api"

  oidc_providers = {
    main = {
      provider_arn               = var.oidc_provider_arn
      namespace_service_accounts = ["myshop:myshop-api"]
    }
  }

  role_policy_arns = {
    secrets = aws_iam_policy.myshop_secrets_read.arn
  }
}

The ARN of the Role created by this Terraform module is written in the ServiceAccount annotation. As a result, the AWS SDK inside the Pod can call Secrets Manager, S3, and CloudWatch using this Role’s permissions.

ConfigMap and Secret #

01-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: myshop-api
  namespace: myshop
data:
  LOG_LEVEL: "info"
  FEATURE_NEW_CHECKOUT: "true"
  CACHE_TTL_SECONDS: "300"
  AWS_REGION: "ap-northeast-2"
---
apiVersion: v1
kind: Secret
metadata:
  name: myshop-api
  namespace: myshop
type: Opaque
stringData:
  DATABASE_URL: "postgresql://will-be-replaced-by-external-secrets"

The Secret’s actual value isn’t placed in the manifest. #3 covers the flow where External Secrets Operator auto-syncs AWS Secrets Manager secrets into K8s Secrets. This post locks in only the skeleton, and the secret management model covered in Advanced #6 GitOps comes back in #4.

Deployment — Pod replicas #

02-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myshop-api
  namespace: myshop
  labels:
    app.kubernetes.io/name: myshop-api
    app.kubernetes.io/component: backend
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: myshop-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: myshop-api
        app.kubernetes.io/component: backend
    spec:
      serviceAccountName: myshop-api
      containers:
        - name: api
          image: 123456789012.dkr.ecr.ap-northeast-2.amazonaws.com/myshop-api:1.4.2
          ports:
            - containerPort: 8000
              name: http
          envFrom:
            - configMapRef:
                name: myshop-api
            - secretRef:
                name: myshop-api
          resources:
            requests:
              cpu: "200m"
              memory: "256Mi"
            limits:
              cpu: "1"
              memory: "512Mi"
          readinessProbe:
            httpGet:
              path: /health/ready
              port: http
            initialDelaySeconds: 5
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health/live
              port: http
            initialDelaySeconds: 30
            periodSeconds: 10
          startupProbe:
            httpGet:
              path: /health/live
              port: http
            failureThreshold: 30
            periodSeconds: 10
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: myshop-api

Standard elements of operational manifests gathered:

  • strategy.rollingUpdate — with maxUnavailable: 0, the new version must come up sufficiently before the old is brought down. The default protection of zero downtime.
  • envFrom — inject ConfigMap and Secret as environment variables at once. No need to write each key individually.
  • Three probes — the three covered in Intermediate #5. liveness conservatively (start after 30s), readiness immediately (5s), startup as a grace period for long initialization.
  • resources — requests / limits covered in Intermediate #4. This workload is Burstable QoS (requests < limits).
  • topologySpreadConstraints — distribute Pods across multiple AZs. Even if one AZ dies, Pods in the other AZ survive.
  • app.kubernetes.io/ standard labels — the standard from Basics #7. Selector, monitoring, logging are all grouped by these labels.

Service — fixed entry point inside cluster #

03-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: myshop-api
  namespace: myshop
  labels:
    app.kubernetes.io/name: myshop-api
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: myshop-api
  ports:
    - name: http
      port: 80
      targetPort: http
      protocol: TCP

type: ClusterIP is the key. The Service itself is a virtual IP only accessible inside the cluster, and external exposure is the responsibility of the next Ingress to be made. The same model covered in Basics #5.

AWS Load Balancer Controller — Ingress to ALB mapping #

EKS uses the standard K8s Ingress object, but to actually resolve that object into an ALB (AWS Application Load Balancer), a component called AWS Load Balancer Controller must be installed in the cluster.

Install via Helm
helm repo add eks https://aws.github.io/eks-charts
helm repo update

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=myshop-prod \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

The ServiceAccount is pre-created with IRSA in Terraform. The controller watches Ingress objects in the cluster, and whenever an Ingress is created, it provisions a matching ALB via the AWS API.

Ingress manifest #

04-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myshop-api
  namespace: myshop
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-northeast-2:123456789012:certificate/abc-123
    alb.ingress.kubernetes.io/healthcheck-path: /health/ready
    external-dns.alpha.kubernetes.io/hostname: api.myshop.example.com
spec:
  rules:
    - host: api.myshop.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myshop-api
                port:
                  number: 80

Each annotation has meaning.

  • scheme: internet-facing — internet-exposed ALB. Internal-only is internal.
  • target-type: ip — registers Pod IPs directly to the ALB target group. ALB watches Pod IP changes.
  • listen-ports / ssl-redirect — accept HTTPS only and auto-redirect HTTP.
  • certificate-arn — ACM certificate. Pre-issue certificate for the domain bound with Route53 via ACM.
  • healthcheck-path — health check path ALB sends to each Pod.
  • external-dns.alpha.kubernetes.io/hostname — the external-dns component auto-registers this ALB’s DNS as an A record in Route53.

About 1–2 minutes after applying this manifest, a new ALB appears in the AWS console and the api.myshop.example.com domain is pointed at it.

HPA — Pod auto-adjustment per traffic #

05-hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myshop-api
  namespace: myshop
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myshop-api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300

When average CPU utilization exceeds 60%, scale up Pod count; when it drops, scale down. The same model covered in Intermediate #6. The behavior stabilization window is the key — scale-up at 30s for fast response, scale-down at 5min for conservatism. This setting provides fast response to traffic spikes while preventing Pods from oscillating on transient dips.

PodDisruptionBudget — the floor of availability #

06-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myshop-api
  namespace: myshop
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: myshop-api

What this single manifest prevents: during voluntary disruptions like node upgrades, node drain, or Cluster Autoscaler’s node reclaim, it guarantees that myshop-api never drops below 2 running Pods simultaneously. This blocks the classic K8s operational incident — “all Pods came down at once during an EKS upgrade and caused downtime.”

Bundling as a Helm chart #

The 7–9 manifests written so far work on their own, but deploying the same workload to dev / prod with different values means duplicating manifests per environment. Helm enters at this point.

Chart structure #

charts/myshop-api/ directory
charts/myshop-api/
├── Chart.yaml
├── values.yaml             # defaults
├── values-dev.yaml         # dev override
├── values-prod.yaml        # prod override
└── templates/
    ├── _helpers.tpl
    ├── namespace.yaml
    ├── serviceaccount.yaml
    ├── configmap.yaml
    ├── secret.yaml
    ├── deployment.yaml
    ├── service.yaml
    ├── ingress.yaml
    ├── hpa.yaml
    └── pdb.yaml

Chart.yaml #

Chart.yaml
apiVersion: v2
name: myshop-api
description: myshop API service
type: application
version: 0.1.0
appVersion: "1.4.2"
maintainers:
  - name: platform-team
    email: platform@myshop.example.com

values.yaml — defaults #

values.yaml
image:
  repository: 123456789012.dkr.ecr.ap-northeast-2.amazonaws.com/myshop-api
  tag: "1.4.2"
  pullPolicy: IfNotPresent

replicaCount: 3

resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    cpu: 1
    memory: 512Mi

serviceAccount:
  create: true
  name: myshop-api
  irsaRoleArn: ""

config:
  LOG_LEVEL: info
  FEATURE_NEW_CHECKOUT: "true"
  CACHE_TTL_SECONDS: "300"

ingress:
  enabled: true
  host: api.myshop.example.com
  certificateArn: ""

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 20
  targetCPUUtilizationPercentage: 60

pdb:
  enabled: true
  minAvailable: 2

values-prod.yaml — environment override #

values-prod.yaml
replicaCount: 5

resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 2
    memory: 1Gi

serviceAccount:
  irsaRoleArn: arn:aws:iam::123456789012:role/myshop-prod-api

config:
  LOG_LEVEL: warn

ingress:
  host: api.myshop.example.com
  certificateArn: arn:aws:acm:ap-northeast-2:123456789012:certificate/abc-123

autoscaling:
  minReplicas: 5
  maxReplicas: 50

Template — Deployment example #

templates/deployment.yaml — partial
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "myshop-api.fullname" . }}
  labels:
    {{- include "myshop-api.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "myshop-api.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "myshop-api.selectorLabels" . | nindent 8 }}
    spec:
      serviceAccountName: {{ .Values.serviceAccount.name }}
      containers:
        - name: api
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          envFrom:
            - configMapRef:
                name: {{ include "myshop-api.fullname" . }}
            - secretRef:
                name: {{ include "myshop-api.fullname" . }}
          resources:
            {{- toYaml .Values.resources | nindent 12 }}

Helpers like {{ include "myshop-api.fullname" . }} are defined in _helpers.tpl. This is a standard Helm chart pattern; running helm create to generate the skeleton includes them automatically.

Deployment #

dev deploy
helm upgrade --install myshop-api charts/myshop-api \
  -n myshop --create-namespace \
  -f charts/myshop-api/values.yaml \
  -f charts/myshop-api/values-dev.yaml
prod deploy
helm upgrade --install myshop-api charts/myshop-api \
  -n myshop --create-namespace \
  -f charts/myshop-api/values.yaml \
  -f charts/myshop-api/values-prod.yaml

The same chart renders differently per environment. This command is replaced by ArgoCD Application sync at the #4 CI/CD stage, but it remains valid for manual deployments during development.

cert-manager and external-dns — auxiliary components #

To automatically obtain and renew Let’s Encrypt certificates inside the cluster instead of managing ACM certificates directly, adopt cert-manager. To auto-manage Route53 DNS records from manifests, adopt external-dns alongside it.

Install cert-manager
helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager \
  -n cert-manager --create-namespace \
  --set installCRDs=true
Install external-dns
helm install external-dns external-dns/external-dns \
  -n external-dns --create-namespace \
  --set provider=aws \
  --set aws.region=ap-northeast-2 \
  --set "domainFilters[0]=myshop.example.com"

This is the standard setup for operational clusters. Whenever a domain is added, a single annotation in the manifest is all that’s needed — no opening the Route53 console. cert-manager is unnecessary when using ACM certificates, but it is more consistent across multi-cluster / multi-cloud environments.

Checks after first deploy #

Items to check right after deploy.

Basic health check
kubectl get all -n myshop
kubectl get hpa -n myshop
kubectl get pdb -n myshop
kubectl get ingress -n myshop
ALB provisioning check (after 1–2 min)
kubectl describe ingress myshop-api -n myshop | grep "Address"
Access from outside
curl -i https://api.myshop.example.com/health/ready

These four commands verify that Pods, HPA, Ingress, and the external entry point are all operational. When 200 responses start coming back, the cluster’s first workload is live.

Closing #

We put myshop-api on the empty EKS cluster as a bundle of manifests, auto-provisioned an ALB via AWS Load Balancer Controller to make the external entry point, and abstracted all manifests as a Helm chart so the same chart unfolds with different values across dev / prod. At this point, myshop-api is accessible from outside but has no data store — it’s an empty shell that only returns 200 on /health/ready. The next post fills that void — connection with RDS PostgreSQL, the path of safely injecting passwords via Secrets Manager, the flow of syncing K8s Secret with cloud secret store via External Secrets Operator, and connection pool operational principles — all in one cycle.

X