EKS Cluster Setup

This is the first chapter of Part 4 (EKS in Production). If the 20 chapters of Parts 1 ~ 3 were a path that followed K8s at the object level, from the model of a single manifest to policy engines and observability, the 6 chapters of Part 4 are the flow of putting one real service on top of that and operating it. We put a fictional backend service, myshop-api, on EKS, connect it to RDS, deploy it through CI/CD, and follow through to the monitoring · operations stages together. This chapter is the starting point of that flow — the stage of standing up an EKS cluster from scratch. We organize the flow of declaring a VPC and cluster with Terraform, setting up the node group and IRSA, and layering on the essential add-ons.

By the end of this chapter you have one empty EKS cluster + IRSA / OIDC provider / essential add-ons / kubeconfig, all prepared. The remaining 5 chapters of Part 4 are a flow of placing one service onto this empty cluster step by step.

myshop-api — the fictional service that runs through Part 4 #

Let’s set the scenario that ties Part 4 together in one line. Assume a fictional company, myshop, is putting its own backend API service, myshop-api, onto EKS. The spec is simple.

• A small REST API written in Python (FastAPI) or Go
• Uses RDS PostgreSQL as its data store
• Exposed externally over HTTPS
• Two environments, prod / dev
• Automatically adjusts the Pod count according to traffic variation

We follow this scenario consistently from the cluster of Chapter 21 to the operations cycle of Chapter 26. Each chapter is structured so it becomes the input for the next, and by the point you’ve followed through to the final Chapter 26 you have the shape of a real production cluster.

Choosing the setup tool — Terraform vs eksctl #

There are several options for creating an EKS cluster.

The de facto standard for production clusters is Terraform. You can declare the VPC · IAM · EKS control plane · node group · add-ons · RDS · Route 53 in one codebase and keep it in git, so the cluster itself becomes reproducible as code. The same manifest applies identically to the dev / prod environments, and changes go in through PR review. The patterns of Chapter 20 GitOps extend straight from K8s manifests to the cluster’s own infrastructure.

eksctl’s abstraction is too tightly bound to EKS for it to take that position, but for quick setup and learning it’s the most intuitive. This chapter covers Terraform as the main path and touches on eksctl briefly as a comparison option.

Terraform project structure #

We start by laying out the structure of myshop-api’s infrastructure Terraform code.

terraform/
├── modules/
│   ├── network/         # VPC, subnets, NAT, routing
│   ├── eks/             # EKS cluster + node group
│   └── addons/          # VPC CNI, EBS CSI, IRSA roles
├── envs/
│   ├── dev/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── terraform.tfvars
│   └── prod/
│       ├── main.tf
│       └── ...
└── versions.tf

This is a structure that puts reusable units in modules/ and instantiates dev / prod differently in envs/. The difference between dev / prod is about instance type, node count, and multi-AZ, while the modules themselves are shared. If the per-environment manifest branching of Chapter 7 Namespace and labels was the pattern of K8s manifests, this chapter’s envs/ structure is the infrastructure pattern.

Provider and backend #

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }

  backend "s3" {
    bucket         = "myshop-tfstate"
    key            = "eks/prod/terraform.tfstate"
    region         = "ap-northeast-2"
    dynamodb_table = "myshop-tfstate-lock"
    encrypt        = true
  }
}

provider "aws" {
  region = "ap-northeast-2"

  default_tags {
    tags = {
      Project     = "myshop"
      Environment = "prod"
      ManagedBy   = "terraform"
    }
  }
}

This is the standard pattern of keeping the state file in S3 and the lock in DynamoDB. default_tags automatically tags every resource created with this provider, so the Cost Allocation Tags-based cost tracking of Chapter 28 Cost optimization rolls along naturally.

VPC — the foundation of EKS #

EKS does not create its own VPC. It’s a structure that plants the control plane’s ENIs on top of a VPC you create. So before making the cluster you have to decide on the VPC and subnets first.

VPC module — terraform-aws-modules/vpc/aws #

It’s standard to use a community module rather than defining the VPC by hand from scratch.

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.project}-${var.env}"
  cidr = "10.10.0.0/16"

  azs              = ["ap-northeast-2a", "ap-northeast-2c"]
  private_subnets  = ["10.10.1.0/24",  "10.10.2.0/24"]
  public_subnets   = ["10.10.101.0/24", "10.10.102.0/24"]
  database_subnets = ["10.10.201.0/24", "10.10.202.0/24"]

  enable_nat_gateway     = true
  single_nat_gateway     = var.env == "dev"
  one_nat_gateway_per_az = var.env == "prod"

  enable_dns_hostnames = true
  enable_dns_support   = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = "1"
  }
}

When bound to EKS, the key things are the two tags at the end.

• kubernetes.io/role/elb = 1 — attached to the public subnets, so the AWS Load Balancer Controller creates external-facing LBs in these subnets.
• kubernetes.io/role/internal-elb = 1 — attached to the private subnets, so internal LBs (inside the cluster + inside the VPC) are created here.

Without these tags there’s no way to know which subnet an LB should go into, and the Ingress we’ll create in Chapter 22 App deployment skeleton won’t work.

single NAT vs NAT per AZ #

single_nat_gateway is turned on only in dev. The cost of a single NAT Gateway is not small (hourly + per data transfer), so in the dev environment we bundle to one, while prod has one per AZ so that if one AZ dies, the workloads of the other AZ don’t lose their NAT.

EKS module — control plane + node group #

Once the VPC is ready, we define the EKS cluster itself.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = "${var.project}-${var.env}"
  cluster_version = "1.32"

  vpc_id     = var.vpc_id
  subnet_ids = var.private_subnet_ids

  cluster_endpoint_public_access  = true
  cluster_endpoint_private_access = true

  enable_irsa = true

  cluster_addons = {
    coredns = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
    vpc-cni = {
      most_recent = true
    }
    aws-ebs-csi-driver = {
      most_recent              = true
      service_account_role_arn = module.ebs_csi_irsa.iam_role_arn
    }
  }

  eks_managed_node_groups = {
    general = {
      desired_size = var.env == "prod" ? 3 : 2
      min_size     = 2
      max_size     = 10

      instance_types = ["t3.medium"]
      capacity_type  = var.env == "prod" ? "ON_DEMAND" : "SPOT"

      labels = {
        role = "general"
      }
    }
  }
}

This single module produces all of the following.

• The EKS control plane (managed K8s 1.32)
• The cluster’s IAM Role
• The OIDC provider (the foundation of IRSA — enable_irsa = true)
• The Managed Node Group (worker nodes created as EC2 instances)
• Four standard add-ons (VPC CNI, CoreDNS, kube-proxy, EBS CSI Driver)

The OIDC provider covered in the IRSA section of Chapter 16 RBAC / ServiceAccount in depth is activated automatically here, and later, in Chapter 23 DB integration, that OIDC becomes the foundation of trust when we bind a ServiceAccount to an IAM Role.

The Managed Node Group as a standard #

EKS worker nodes can be composed in three ways.

The Managed Node Group is the standard for first adoption. Define the instance type · size · capacity_type (ON_DEMAND / SPOT) in a manifest and EKS automatically manages the joining · removal · upgrade of nodes. The node’s own OS is Amazon Linux 2 or Bottlerocket, and it pulls an AMI with kubelet · containerd · the CNI agent preinstalled.

Two modes of the cluster endpoint #

The combination of the two flags cluster_endpoint_public_access and cluster_endpoint_private_access controls access to the cluster API server.

The security guidance for prod environments is usually the last one (private only), but to call kubectl directly from GitHub Actions, public has to be on. The common compromise is to enable public + IP restriction (cluster_endpoint_public_access_cidrs) together. In Chapter 24 CI / CD pipeline we cover the flow of choosing this endpoint mode by which side — GitHub Actions or ArgoCD — will be calling it.

IRSA — granting IAM Roles to add-ons #

Add-ons like the EBS CSI Driver, AWS Load Balancer Controller, and External Secrets call AWS APIs from inside the cluster. We cover the pattern of granting permissions to these calls with IRSA. This is the stage where the model seen in Chapter 16 leads into a full Terraform setup.

module "ebs_csi_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.0"

  role_name = "${var.project}-${var.env}-ebs-csi"

  attach_ebs_csi_policy = true

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["kube-system:ebs-csi-controller-sa"]
    }
  }
}

This module automatically creates the following.

• An IAM Role (named myshop-prod-ebs-csi)
• A policy (EBS disk create · delete · attach permissions)
• A trust policy (the same trust policy from Chapter 16 — restricting it so that only the ebs-csi-controller-sa ServiceAccount in the kube-system namespace can take on this Role)

This IAM Role’s ARN is passed to cluster_addons.aws-ebs-csi-driver.service_account_role_arn above, so EKS automatically attaches an annotation to the EBS CSI add-on’s ServiceAccount. As a result, the flow where creating a PVC covered in Chapter 9 PV / PVC / StorageClass automatically provisions an EBS volume works.

The same pattern applies to the External Secrets of Chapter 23 and to the CloudWatch side of Chapter 25 Monitoring · alerts. A 1:1 mapping of ServiceAccount + IAM Role per workload is the standard security structure of K8s in production.

kubeconfig — accessing the cluster #

After you create the cluster with Terraform, you need to fetch the kubeconfig so that kubectl can call that cluster locally. The kubeconfig model seen in Chapter 2 Local environment applies to EKS directly.

aws eks update-kubeconfig \
  --region ap-northeast-2 \
  --name myshop-prod

kubectl get nodes
kubectl get pods -A

NAME                                            STATUS   ROLES    AGE   VERSION
ip-10-10-1-145.ap-northeast-2.compute.internal  Ready    <none>   3m    v1.32.0
ip-10-10-2-201.ap-northeast-2.compute.internal  Ready    <none>   3m    v1.32.0
ip-10-10-2-83.ap-northeast-2.compute.internal   Ready    <none>   3m    v1.32.0

If three nodes are up and the system Pods in the kube-system namespace (coredns, kube-proxy, aws-node, ebs-csi-controller) are all in Running state, the cluster is healthy.

Access permissions — the role of the aws-auth ConfigMap #

The IAM user / Role that created the EKS cluster automatically gets system:masters permissions. To give access to another IAM user, you modify the aws-auth ConfigMap or use the new model from 1.23+, EKS Access Entries. Access Entries are more standard and are recommended in the Terraform module too. The standard RBAC ClusterRoles (view / edit / admin) of Chapter 14 RBAC / NetworkPolicy / ResourceQuota bind naturally with EKS Access Entries.

access_entries = {
  developers = {
    principal_arn = "arn:aws:iam::123456789012:role/Developer"

    policy_associations = {
      view = {
        policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
        access_scope = {
          type = "cluster"
        }
      }
    }
  }
}

This is a manifest binding one IAM Role to cluster-wide view permission. The K8s RBAC view ClusterRole is mapped automatically, so a user with this Role can read every object in the cluster.

eksctl — the path of quick setup #

When you need a fast cluster for learning or a PoC, eksctl finishes in one line.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: myshop-dev
  region: ap-northeast-2
  version: "1.32"

vpc:
  nat:
    gateway: Single

managedNodeGroups:
  - name: general
    instanceType: t3.medium
    desiredCapacity: 2
    minSize: 2
    maxSize: 5
    spot: true

addons:
  - name: vpc-cni
  - name: coredns
  - name: kube-proxy
  - name: aws-ebs-csi-driver

iam:
  withOIDC: true

eksctl create cluster -f cluster.yaml

This single command automatically creates everything from the VPC · EKS · node group · OIDC provider to the default add-ons. It takes about 15 ~ 20 minutes, and when it’s done the kubeconfig is refreshed automatically too.

The shape of eksctl can be summarized as follows.

• Strengths — the lowest learning curve; one manifest creates an entire cluster at once.
• Weaknesses — it’s hard to manage together with surrounding resources like multi-environment / RDS / Route 53 as a single set. It uses CloudFormation internally, so its state is separate from Terraform.

The goal for a production cluster is almost always Terraform, but for first learning or a one-off PoC, eksctl is the fastest.

Karpenter — the new path of node autoscaling #

A Managed Node Group can do its own autoscaling (Cluster Autoscaler), but its instance types are limited to a few predefined ones. In environments where traffic varies widely and workloads have diverse resource demands, Karpenter is establishing itself as the new standard.

The model touched on in Chapter 13 Autoscaling §“Karpenter — EKS’s faster alternative” leads here into a full manifest. It looks at Pods in Pending state, picks an instance type matching that Pod’s resource demand in real time, and brings up a new node. Instead of a fixed instance pool, it picks the most suitable one from all of AWS’s EC2 types.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["t3.medium", "t3.large", "m5.large", "m5.xlarge"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  limits:
    cpu: 100
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized

Adopting Karpenter is a burden at the starting point of Chapter 21, so it’s natural to start with a Managed Node Group and switch to Karpenter once the traffic pattern has settled. We revisit this in Chapter 26 Operations checklist and Chapter 28 Cost optimization.

First checks after cluster setup #

These are the check commands that are good to run once right after the cluster comes up.

kubectl version --short
kubectl get nodes -o wide

kubectl get pods -n kube-system

aws eks describe-cluster \
  --name myshop-prod \
  --region ap-northeast-2 \
  --query "cluster.identity.oidc.issuer" \
  --output text

aws eks list-addons --cluster-name myshop-prod --region ap-northeast-2
aws eks describe-addon --cluster-name myshop-prod \
  --addon-name vpc-cni --region ap-northeast-2

These four commands confirm whether the cluster · nodes · system Pods · OIDC · add-ons are all healthy. If anything looks off, it’s safer to fix it before moving on to the next chapter.

A first impression of cost #

The cost of an EKS cluster has roughly three parts.

The starting cost of the smallest prod cluster is about $200 ~ $300 per month. Using SPOT instances + Single NAT in the dev environment drops it to less than half of that. Cost is covered in Chapter 28 Cost optimization, but it’s good to be aware of it from the moment you stand up the cluster.

Exercises #

1. Apply this chapter’s Terraform code to an AWS account you can use for learning and stand up one dev EKS cluster. After refreshing the kubeconfig with aws eks update-kubeconfig, record the output of kubectl get nodes -o wide and kubectl get pods -n kube-system. Run the four commands of §“First checks after cluster setup” in order and check that the OIDC provider URL · add-on status · 9 or more system Pods are all healthy.
2. Compare the trade-offs of the three combinations of cluster_endpoint_public_access / cluster_endpoint_private_access (public-only / public + private / private-only) against your own operational scenario in one paragraph. Note how, in Chapter 24 CI / CD pipeline, whether GitHub Actions will call kubectl directly or whether you’ll go with a pull model where ArgoCD only watches git binds with this endpoint decision.
3. Referring to this chapter’s EBS CSI IRSA module, write a skeleton of the IRSA Terraform manifest for the AWS Load Balancer Controller and the External Secrets Operator. Organize each ServiceAccount’s namespace · name · required IAM policy with the trust policy model of Chapter 16, and explain in one paragraph why the principle of “specify both namespace + SA name” matters for isolation.

In one line: setting up a production EKS cluster is the flow of declaring the VPC · EKS control plane · node group · IRSA · standard add-ons (VPC CNI · CoreDNS · kube-proxy · EBS CSI) in one codebase with Terraform. eksctl is the path of quick learning · PoC, and Karpenter is the new standard of node autoscaling that replaces the Managed Node Group after the traffic pattern has settled. The cluster’s own cost is anchored by three parts — control plane + nodes + NAT — and the starting cost of prod is around $200 ~ $300 per month.

Next chapter #

At this point the cluster is empty — the nodes are up and the system Pods are alive, but the myshop-api we want to put on has not gone in as a single line of manifest yet. In the next chapter we fill that empty space.

In Chapter 22 App deployment skeleton we organize the five objects of Chapter 4 Deployment / Chapter 5 Service / Chapter 10 Ingress / Chapter 6 ConfigMap · Secret together, and cover the flow of wrapping it into a Helm chart and connecting it all the way to per-environment deployment. The full setup of the AWS Load Balancer Controller and cert-manager continues in this chapter too.