Ingress and the Ingress Controller

If what we covered up through Chapter 9, PV / PVC / StorageClass was the model of how data inside the cluster survives, this chapter turns to how external traffic reaches a Service inside the cluster. The LoadBalancer of Chapter 5, Service is the standard for external entry, but if one cluster has dozens of Services that must be exposed externally, running that many LoadBalancers becomes a burden on both cost and management. This chapter follows the two layers of the object Ingress, and the Ingress Controller that resolves its manifest into actual routing.

By the end of this chapter you’ll have the common operational entry shape of one Ingress Controller behind one cloud LoadBalancer, with several ClusterIP Services behind it. It is the standard pattern where TLS termination · domain routing · and path routing come together.

What a single LoadBalancer can’t solve — the starting point #

If we shrink the LoadBalancer Service from Chapter 5 to one line, it’s this — a single type: LoadBalancer line and the cloud provider automatically creates an external LB, with traffic flowing through that LB’s external IP to one Service’s backend Pods. It’s a simple and powerful model.

The problem begins once the Services exceed one. If we draw the common shape of a production cluster, it’s this.

• the web Service — the user web pages
• the api Service — the REST API backend
• the admin Service — the operator management pages
• the static Service — dedicated to static assets

If you attach type: LoadBalancer to each to expose all four externally, four cloud LBs are created. Whether AWS’s NLB · ALB, GCP’s LB, or Azure’s LB, an LB itself has an hourly cost, and this cost accumulates in proportion to the number of Services. With dozens of Services, the LB cost reaches a level you can’t ignore.

Cost isn’t the only burden. The operational burden grows along with it.

• Managing domains as many as the number of LBs — since the domain shown externally is set per LB, you have to manage that many DNS records too.
• A TLS certificate per LB separately — you have to hold certificate issuance · renewal per LB.
• Host · path-based routing isn’t possible — even if you want to send example.com/api to the api Service and example.com/static to the static Service, one LoadBalancer Service looks only at one Service’s backend. To solve this routing, someone has to stack one more tier.

The abstraction that solves these three concerns is Ingress. One Ingress Controller stands behind one cloud LB, and that Controller looks at the domain · path and splits traffic out to several Services inside the cluster. It handles TLS termination centrally too. From the outside it looks like one LB, but inside it’s a branching route table driven by the Ingress manifest.

The two layers of Ingress — the object and the controller #

Here’s a part that often becomes a trap in K8s. Traffic doesn’t flow just because you write an Ingress manifest. Ingress works only when two layers are present together.

Objects like Deployment · Service are handled directly by the K8s core (the control plane) with its own controllers, but Ingress is different. The Ingress controller is not included in the K8s core; it must be installed separately into the cluster as an external component. If you apply an Ingress manifest to a cluster with no controller, the object is created but the traffic flows nowhere — the accident takes the form of apply succeeding while no response comes from outside.

It’s easiest to keep the mental picture as follows.

[external client]
       │
       ▼
[ cloud LB ]  ←── a single Service (LoadBalancer) that the Ingress Controller exposes
       │
       ▼
[ Ingress Controller Pod ]  ←── nginx / Traefik / ALB, etc.
       │
       │  reads the routing rules of the Ingress object and branches
       ├──────────┬──────────┐
       ▼          ▼          ▼
   [Service A] [Service B] [Service C]
       │          │          │
     Pods       Pods       Pods

The key is — there is one external cloud LB, and the Ingress Controller standing behind it combines the rules of all Ingress objects to perform the actual routing. Even if you write 100 Ingress objects in the cluster, the cloud LB is usually still one.

Kinds of Ingress Controller #

The manifest format of the Ingress object is set as a standard by K8s, but how it is interpreted and resolved into routing differs per controller implementation. If we organize the controllers you meet often in operations into one table, it’s as follows.

It’s easy to understand by dividing them broadly into two branches.

• A Pod inside the cluster routes — controllers like ingress-nginx, Traefik, and HAProxy run their own nginx · Traefik Pod in the cluster, and that Pod receives and handles the traffic. The cloud LB stands in front of that Pod as a simple L4 distributor.
• A cloud resource routes — the AWS ALB Controller, GKE Ingress, and AKS AGIC have the cloud’s managed LB (ALB · CLB · Application Gateway) itself do the host · path routing. Inside the cluster there’s only a small controller Pod that translates the Ingress object into the cloud LB’s rules.

Even for the same Ingress manifest, the annotation keys and behavior change depending on which controller handles it. So when you write controller-specific annotations in a manifest, you have to start by confirming which controller you use. The part that covers the AWS Load Balancer Controller on EKS is organized once more in Chapter 22, The app deployment skeleton.

Installing the Ingress Controller #

In a production cluster you usually install one kind of controller once and have all Ingresses pass through it. Let’s note the per-environment installation shape briefly.

minikube #

One line of the addons command installs ingress-nginx.

minikube addons enable ingress

kubectl get pods -n ingress-nginx
NAME                                       READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-6d4b7f6c8-abc12   1/1     Running   0          30s

kind #

kind applies ingress-nginx with a separate manifest.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml

Additionally, when creating the kind cluster you need a setting that maps ports 80 / 443 to the host. Just follow the official ingress-nginx guide.

EKS / GKE / AKS #

On managed clusters, installing the controller with Helm is the standard pattern.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace

When using the AWS ALB Controller on EKS, separate IAM · service-account setup follows.

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  --namespace kube-system \
  --set clusterName=my-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

On GKE, the Google-managed GKE Ingress is enabled by default, so without a separate install you can have the experience of a GCLB being created automatically when you apply an Ingress manifest. AKS also has a managed option in a similar manner.

An installed controller usually runs as a set of Pod · Service · Deployment · ConfigMap in its own namespace (ingress-nginx, kube-system, etc.). Having just one kind of controller installed per cluster is usually enough.

The first Ingress manifest — simple domain routing #

Let’s start from the simplest shape. It’s an Ingress that sends all traffic coming in on example.com to port 80 of the web Service.

First, assume the backend Service is up.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  type: ClusterIP
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 80

It matters that the Service type is ClusterIP. The Service behind an Ingress being cluster-internal-only is enough. External exposure is performed instead by the Ingress Controller with its single LoadBalancer Service. There’s no reason to write type: LoadBalancer on this Service.

Now the Ingress manifest.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
spec:
  ingressClassName: nginx
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80

Let’s note the fields one line each.

• apiVersion: networking.k8s.io/v1 — the stable version of Ingress. It became stable in 1.19, and the older extensions/v1beta1 and networking.k8s.io/v1beta1 are no longer used.
• spec.ingressClassName: nginx — points to which Ingress Controller handles this object. If one cluster has only one kind of controller and it’s the default IngressClass, you may omit it, but it’s safer to specify it. We look at the details again in the IngressClass section later.
• spec.rules[].host: example.com — expresses which host’s traffic this is. It matches by looking at HTTP’s Host header and HTTPS’s SNI.
• spec.rules[].http.paths[].path: / — the path to match. / means all paths of this host.
• spec.rules[].http.paths[].pathType: Prefix — the path-matching method. The most common value.
• spec.rules[].http.paths[].backend.service — the Service name and port to send the matched traffic to.

After applying this manifest, the status looks like the following.

kubectl apply -f ingress-web.yaml
kubectl get ingress

NAME   CLASS   HOSTS         ADDRESS          PORTS   AGE
web    nginx   example.com   34.120.10.20     80      30s

The ADDRESS column fills with the IP or hostname of the external entry point. This IP is the same as the EXTERNAL-IP of the LoadBalancer Service that exposes the Ingress Controller. Even if there are 100 Ingress objects in the cluster, this ADDRESS is usually the same single IP.

DNS mapping #

To make example.com actually point at the ADDRESS above in operations, you have to set an A record (or CNAME) to that IP in your external DNS. It does not happen automatically inside K8s. There’s also an operational pattern where, using a tool like ExternalDNS, it looks at the host field of the Ingress object and automatically creates records in cloud DNS. The pattern of tying ExternalDNS and Route 53 together in an EKS environment is covered in Chapter 22, The app deployment skeleton.

path-based routing #

This is the case where you want to send each path of the same host to a different Service. The shape where example.com/api goes to the backend-api Service and example.com/static goes to the cdn Service.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
spec:
  ingressClassName: nginx
  rules:
    - host: example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: backend-api
                port:
                  number: 8080
          - path: /static
            pathType: Prefix
            backend:
              service:
                name: cdn
                port:
                  number: 80
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80

This manifest behaves as follows.

• example.com/api/users → backend-api:8080
• example.com/static/logo.png → cdn:80
• example.com/ or example.com/about → web:80

Paths are not evaluated top to bottom; rather, the longer matching path takes priority. Since /api matches longer than /, an /api/users request goes to backend-api. This priority rule can differ subtly per controller implementation, so in operations it’s safer to write the manifest so that paths don’t overlap as much as possible.

The three pathType values #

pathType sets the path-matching method. There are three.

The value used most often in operations is Prefix. You choose Exact when you want to expose just a single endpoint, and ImplementationSpecific when you want to use a controller-specific feature (e.g., ingress-nginx’s regex). If you’re thinking about manifest portability, it’s safe to unify on Prefix.

Since Prefix’s slash-boundary matching is a subtle part, to note it briefly — when you write /api, /api/users matches but /api2 does not. It’s because K8s interprets Prefix with the meaning of a “directory prefix.” This difference is sometimes related to security — it prevents the accident of an unintended path flowing to the same backend.

host + path combination #

This is the shape where different domains are handled together in one Ingress. The common configuration that splits api.example.com and app.example.com.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: multi
spec:
  ingressClassName: nginx
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: backend-api
                port:
                  number: 8080
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80

You just put two items with different hosts in the rules array. One Ingress may hold several hosts, or you may create a separate Ingress object per host. In operations the pattern of splitting Ingresses by domain group is common, like one per namespace or one per team when responsibility is separated.

The host supports wildcards too. If you write *.example.com, one-level subdomains like foo.example.com and bar.example.com all match. However, *.example.com does not match two levels like foo.bar.example.com — it follows that same rule of DNS wildcards.

TLS termination — Ingress + Secret #

In operations, external entry is almost always HTTPS. Pairing the Ingress’s tls field with a K8s Secret lets you handle certificate termination in one place. The mental model of the Secret object itself was already established in Chapter 6, ConfigMap and Secret.

First, you need a Secret holding the certificate · key.

kubectl create secret tls example-com-tls \
  --cert=fullchain.pem \
  --key=privkey.pem

The Secret’s type is kubernetes.io/tls, and inside it holds two keys, tls.crt (the certificate chain) and tls.key (the private key). You can make the same shape with a manifest too, but since you’d put the base64-encoded value of the key file into the manifest, it’s usually cleaner to create it with a command as above.

Connect this Secret to the Ingress.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - example.com
      secretName: example-com-tls
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80

For traffic of the domain written in spec.tls[].hosts, the Ingress Controller terminates TLS when it receives it, and forwards plain HTTP to the inner Service. Thanks to this shape, the Service- and Pod-side manifests don’t have to worry about HTTPS, and certificate renewal is just a matter of swapping out the Secret.

cert-manager — automatic issuance · renewal #

The standard tool for automatically issuing · renewing certificates from a free CA like Let’s Encrypt is cert-manager. Install cert-manager into the cluster and create a ClusterIssuer object once, and just writing one annotation line in the Ingress manifest proceeds with certificate issuance automatically.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - example.com
      secretName: example-com-tls
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80

Looking at the cert-manager.io/cluster-issuer annotation, cert-manager proceeds automatically through the following cycle.

1. Verify domain ownership through an ACME challenge (HTTP-01 or DNS-01).
2. Issue the certificate from Let’s Encrypt.
3. Create the certificate · key as the Secret written in secretName.
4. Automatically renew about 30 days before expiry.

cert-manager’s own installation and ClusterIssuer setup aren’t small in volume, so this chapter only notes the shape, and the in-earnest usage together with ACM integration on EKS is organized in Chapter 22, The app deployment skeleton. Since cert-manager has effectively become the standard in operations, just remembering its name and place is enough.

IngressClass — when there’s more than one controller #

Most clusters keep only one kind of controller, but there are also cases where two kinds must be present together in the same cluster. For example, the shape where you want to send public traffic via ingress-nginx and internal management traffic via the AWS ALB Controller. The object that expresses which Ingress object should be handled by which controller in this case is IngressClass.

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
  annotations:
    ingressclass.kubernetes.io/is-default-class: "true"
spec:
  controller: k8s.io/ingress-nginx

There are two key fields.

• spec.controller — the identifier that points to which controller implementation handles this IngressClass. It’s fixed per implementation, like k8s.io/ingress-nginx for ingress-nginx and ingress.k8s.aws/alb for the AWS ALB Controller.
• The ingressclass.kubernetes.io/is-default-class: "true" annotation — marks this IngressClass as the cluster’s default. If you don’t write ingressClassName in an Ingress manifest, the controller of the default IngressClass handles it.

In the Ingress manifest you write which class it belongs to via spec.ingressClassName.

spec:
  ingressClassName: alb   # handled by the AWS ALB Controller
  rules:
    - host: admin.example.com
      ...

If another Ingress in the same cluster is ingressClassName: nginx, it’s handled by ingress-nginx. The two controllers don’t clash over the same object.

kubectl get ingressclass

NAME            CONTROLLER             PARAMETERS   AGE
nginx (default) k8s.io/ingress-nginx   <none>       30d
alb             ingress.k8s.aws/alb    <none>       10d

The class marked (default) is the default. The standard is to keep only one default IngressClass per cluster. If two are marked default, it becomes ambiguous which side a new Ingress object is handled by.

Gateway API — Ingress’s successor standard #

Finally, let’s note one thing in a line. The Ingress object is enough for simple host · path routing, but to handle more expressive routing (header-based, query-parameter-based, weighted traffic distribution, etc.) it comes to depend on annotations, which has the limit of differing per controller. The successor standard that SIG-Network made to solve that limit is the Gateway API.

The Gateway API splits the objects into three.

Many controllers like ingress-nginx, Traefik, Cilium, and Istio support the Gateway API, and an operator starting a new cluster is at the stage of evaluating the Gateway API. However, Ingress is still the most widely used standard, and the objects an operator meets in an existing cluster are mostly Ingress. We’ll defer the deep model of the Gateway API to a later K8s deep-dive book, and in this chapter leave it at noting just the name and place.

The operational pattern — one LoadBalancer + many Ingresses #

Let’s pin down this chapter’s mental picture before moving on. The external entry of a production cluster usually comes down to one cloud LoadBalancer. That LoadBalancer is attached to one Service (type: LoadBalancer) that the Ingress Controller runs for its own exposure, and all external traffic passes through that one LB.

[ cloud LoadBalancer ]   ←── one. the cost is in one place too
            │
            ▼
[ Ingress Controller (the nginx Pod bundle) ]   ←── the ingress-nginx namespace
            │
            │  the routing rules of the Ingress objects
            ├──────────┬──────────┬──────────┐
            ▼          ▼          ▼          ▼
       [web]      [api]       [admin]    [static]   ←── each its own ClusterIP Service

The effect is straightforward:

• The cloud LB cost is centralized — even as the number of Services grows, the LB stays one.
• TLS termination is centralized — the certificate · cert-manager setup stays in one place too.
• Domain · path routing is expressed in manifests — exposing a new Service is a matter of adding one line of an Ingress object.
• The DNS records are simple — set *.example.com to the LB’s IP / name once and all hosts come in to that LB.

Regardless of the number of Services the cloud LB is usually one, but in cases where traffic isolation is needed (public vs internal, etc.) the pattern of running two bundles of the controller to have two LBs is common too. The shape in the IngressClass section above is an example of that operational pattern.

The availability of the Ingress Controller itself #

One risk of this structure is — the Ingress Controller can become a single point of failure. Since all external traffic passes through that Controller, if the Controller Pods fail at the same time, all external traffic is cut off. In operations it’s standard to have the following in place.

• Multiplexing the Controller Pods — set the Deployment’s replicas to 2 or more. Spread them with a PodDisruptionBudget · anti-affinity so that even on a node failure one Pod stays alive.
• Resource headroom — set the Controller Pod’s CPU · memory limits generously. This is the subject covered in the next chapter, Chapter 11, resources.requests / limits.
• Monitoring — tie the Prometheus metrics the Controller exposes (request count, 5xx ratio, response time) into alerts. Covered in Chapter 19, Observability.

Exercises #

1. After installing ingress-nginx on a local cluster (minikube or kind), try applying a simple Ingress manifest that sends traffic coming in on example.com to the web Service. Record in time order what fills the ADDRESS column of kubectl get ingress, and what response curl http://example.com gives after adding 127.0.0.1 example.com (kind) or the minikube ip value to /etc/hosts.
2. After applying ingress-paths.yaml as in the body, alternate pathType between Prefix and Exact and organize in a table where the four requests curl http://example.com/api, curl http://example.com/api/, curl http://example.com/api/users, and curl http://example.com/api2 flow to. Summarize in one paragraph, in your own words, the slash-boundary matching model of §“The three pathType values,” that an unintended path (/api2) does not flow to the same backend.
3. Consider applying two Ingress manifests with different ingressClassName values in the same cluster — for example, example.com with the nginx class and admin.example.com with the alb class. In one paragraph, using the model of §“IngressClass,” explain why a cluster with two IngressClasses must still have only one default, and how a new Ingress would behave if two classes were mistakenly marked default.

In one line: Ingress is a two-layer abstraction for host · path routing and TLS termination behind a single cloud LoadBalancer. The Ingress object is the manifest, and the Ingress Controller (ingress-nginx · ALB · GKE Ingress, etc.) reads that object and routes the traffic. The operational standard is one cloud LB + one controller deployment + many ClusterIP Services, on top of which TLS auto-issuance is handled with cert-manager and multiple controllers with IngressClass. The successor standard, the Gateway API, handles more expressive routing and is at the evaluation stage.

Next chapter #

One thing we noted briefly at the end of this chapter becomes the starting point of the next chapter — the resource headroom of the Ingress Controller Pod. Since the Controller handles all external traffic, if that Pod is short on CPU · memory, the traffic itself gets blocked. But how a Pod in K8s expresses the resources it needs, and how that expression affects the scheduler’s decisions and the node’s OOM behavior, we haven’t covered yet.

Chapter 11, resources.requests / limits organizes the model of the two Pod-manifest fields resources.requests and resources.limits, the difference in behavior of the two resources CPU and memory (CPU is throttling, memory is OOMKill), the QoS classes (Guaranteed / Burstable / BestEffort), and the operational pattern of enforcing namespace defaults with LimitRange.