PV / PVC / StorageClass

Up through Chapter 6, ConfigMap and Secret, what we’d separated out of the manifest was configuration and secrets. Values like image tags · DB hosts · API keys moved into external objects so the workload definition is not tied to its environment. But one more dimension remains — the data itself. The filesystem inside a container is a temporary space that disappears when the container dies, while things like DB data · user-uploaded files · Prometheus’s metric time series must survive beyond a Pod’s lifecycle. This chapter follows how Kubernetes expresses that persistent-disk model through the triangle of PersistentVolume, PersistentVolumeClaim, and StorageClass.

By the end of this chapter you’ll be at a state where you can read in a single line what the volumeClaimTemplates we noted briefly in Chapter 8, StatefulSet / DaemonSet / Job / CronJob actually creates on top of all this.

The temporariness of the container filesystem — the starting point #

In Chapter 8, one of the things we noted that StatefulSet solves was “a 1:1 persistent volume per Pod,” and we deferred the detailed model of that PVC to the next chapter. This chapter is the one that covers all of that deferred content. Let’s organize it from the starting point — why we need an object called a persistent volume.

A container’s default filesystem is a temporary space confined inside that container. When the container terminates, the files accumulated inside it disappear too. When a container restarts inside a Pod, even within the same Pod the filesystem starts clean as a fresh one. And if the Pod itself moves to another node, that temporary space is even less likely to remain. The Deployment model we saw in Chapter 4, Deployment and ReplicaSet was “a Pod can die and come back up at any time,” and while that temporariness is natural for stateless workloads, it’s a problem for workloads that must hold state.

To keep data alive, you have to write not inside the Pod’s filesystem but to a disk outside it. That disk must stay alive even when the Pod dies, and must be remountable unchanged when a new Pod comes up. The way K8s expresses this requirement is the separation of the three objects PV / PVC / StorageClass.

There are non-persistent volumes like emptyDir too #

Not all K8s volumes are persistent. There are also volumes that last only while the Pod is alive, like emptyDir — used for things like two containers in one Pod exchanging files, or scratch space for working with large temporary files. emptyDir disappears when the Pod disappears. The subject of this chapter is the opposite — the model of a disk that survives separated from the Pod’s lifecycle.

The triangle — PV / PVC / StorageClass #

If we summarize the role of the three objects in one line each, it’s as follows.

This separation is the core of K8s’s persistent-data model. The app developer writes only the PVC — writing “I need a 5Gi RWO disk.” Which disk type from which cloud satisfies that request is held by the SC, and the representation of the actual disk is mapped to a PV. Thanks to this, the same manifest can flow to EBS on an AWS cluster, to PD on GCP, and to NFS or Ceph on an on-prem cluster.

As a mental picture you can draw it as follows.

app manifest
   │
   ▼
PVC (5Gi, RWO, storageClassName=fast-ssd)
   │
   ├── (static) find a matching one among the PVs the admin pre-created → Bound
   │
   └── (dynamic) the provisioner of StorageClass(fast-ssd)
                 creates a new disk and registers it as a PV → Bound to that PV

The state Bound means a PVC and a PV are paired 1:1. One PVC binds to only one PV, and one PV binds to only one PVC too. A Pod only looks at the PVC and mounts it; it doesn’t look directly at the PV. Thanks to this one layer of indirection, you can leave the workload manifest unchanged even when the disk backend changes.

Static provisioning — the simplest model #

Before getting into dynamic provisioning, let’s follow the simplest model first — the flow where an admin creates a PV directly and a PVC binds to it. This model isn’t used often in operations, but it’s the best starting point for understanding the matching rules between a PV and a PVC.

On a local cluster like minikube or kind, hostPath, which borrows one path of the node’s host filesystem as the backing for a PV, is common. It isn’t suitable for production, but it’s enough for learning.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-local-1g
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  hostPath:
    path: /mnt/data/pv-local-1g

spec.capacity.storage is the size of the disk, spec.accessModes is how many can mount it at once and in what way (covered shortly), and persistentVolumeReclaimPolicy is how to handle this PV when the PVC disappears. storageClassName: manual is a mark that says “a hand-made PV that belongs to no SC” — this label is used in matching with a PVC.

Here’s the PVC to use with it.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: manual

The rule by which a PVC chooses a PV is clear. All three of the following must match for it to bind.

• Is the storageClassName the same — in the example above, both sides are manual. Writing storageClassName: "" (empty string) in a PVC matches only PVs with no SC, while omitting the field itself follows the cluster’s default SC.
• Does the PV satisfy the PVC’s required accessModes — if the PVC requested RWO and the PV supports both RWO / RWX, it’s OK; the reverse (the PV can only do RWO but the PVC requires RWX) does not match.
• Is the capacity at or above the PVC’s request — if the PVC requested 1Gi and the PV is 1Gi or more, it can match. However, if a large PV binds to a small PVC, the difference is wasted.

kubectl apply -f pv-static.yaml
kubectl apply -f pvc-static.yaml
kubectl get pv,pvc

NAME                          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM           STORAGECLASS   AGE
persistentvolume/pv-local-1g  1Gi        RWO            Retain           Bound    default/data    manual         10s

NAME                         STATUS   VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/data   Bound    pv-local-1g   1Gi        RWO            manual         5s

Both the PV and the PVC being STATUS: Bound, holding each other’s names in the CLAIM / VOLUME columns, is the normal shape. When some Pod mounts this PVC, inside that Pod’s container it just looks like an ordinary directory, but that directory is in fact mapped to the node’s /mnt/data/pv-local-1g.

1:1 direct binding — claimRef / volumeName #

The matching above is the way K8s finds it on its own via storageClassName + accessModes + capacity, but if you really want to bind a specific PV to a specific PVC only, you can also write it explicitly.

• Write the PVC’s namespace + name in the PV’s spec.claimRef
• Write the PV name in the PVC’s spec.volumeName

When these two go in together, K8s ignores the other matching candidates and binds them in that 1:1. Using this pattern in operations is rare, but it shows up now and then in migration or disk-recovery scenarios.

accessModes — who can mount it and how #

accessModes is the field that expresses the disk’s concurrent-mount possibility. There are four.

The two you see most often in operations are RWO and RWX. And which mode is possible is decided by the kind of backend disk.

In one line — block storage can do RWO, file storage can go up to RWX. A workload that needs RWX (a pattern where several Pods use the same directory together, e.g., WordPress’s upload directory, a shared cache) must pick an NFS-family backend. A PVC requiring RWX on a block disk will never bind.

ReadWriteOncePod — preventing DB split-brain #

ReadWriteOncePod, stabilized from 1.22, is a tighter constraint than RWO. RWO is “several Pods can mount it together within one node,” but RWOP is “only a single Pod can mount it across the whole cluster.” It’s used as a safeguard for workloads like a database that break if two processes touch the same data files at once — it also prevents the accident of a Pod in a different namespace on the same node mistakenly pulling in the same PVC.

StorageClass and dynamic provisioning #

Static provisioning carries a self-evident burden — the cluster admin has to create the PV in advance, by human hand. Every time a new disk is needed, the cycle of creating a disk in the cloud console, writing the PV manifest, and applying it becomes a human task. In a production cluster this shape quickly becomes a bottleneck.

StorageClass fills that gap. If you create an SC just once in advance, then when a PVC pointing to that SC comes in, K8s’s provisioner automatically creates a disk, registers it as a PV, and binds it to the PVC. The human hand drops out of the PV step.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
  iops: "3000"
  throughput: "125"
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

A PVC pointing to this SC becomes as light as the following.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: fast-ssd

When this PVC is applied, the following happens automatically.

1. K8s looks at the PVC’s storageClassName and finds the fast-ssd SC.
2. The SC’s provisioner (ebs.csi.aws.com) is called and creates a new 5Gi gp3 volume on AWS EBS.
3. That EBS volume is automatically registered as a PV object.
4. The new PV becomes Bound with the PVC.

If we briefly organize examples of per-cluster provisioners in a table, it’s as follows.

A production cluster usually has one default SC (default StorageClass) designated, and if you omit storageClassName in a PVC, that default SC is applied automatically. Which SC is the default is marked by the SC’s annotation storageclass.kubernetes.io/is-default-class: "true".

kubectl get sc

NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
fast-ssd (default)   ebs.csi.aws.com         Retain          WaitForFirstConsumer   true                   10d
slow-hdd             ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   10d

The SC marked (default) is the default. If both SCs in one cluster are marked default, the behavior of a newly created PVC becomes ambiguous, so the operational standard is to keep only one as the default. The actual cost trade-offs of EBS CSI · gp3 on EKS are covered once more in Chapter 28, Cost optimization.

The three key fields of a StorageClass #

Let’s note the four fields you’ll touch most often in an SC manifest.

provisioner #

The field that points to which CSI driver creates the disk. As per the table above, the value is fixed per cluster environment. CSI (Container Storage Interface) is the standard interface through which K8s talks to external storage drivers; it became stable in 1.13, and after that all the in-tree drivers (the old drivers that were included in the K8s core) were moved to CSI external drivers. The provisioners you touch in a new cluster are almost all in the *.csi.* form.

reclaimPolicy #

The policy for what to do with the PV (and the real disk behind it) that was bound to a PVC when that PVC disappears. Two values are the practical choices.

You may see a value Recycle in old documents, but it’s deprecated, so don’t use it in new manifests.

Delete is convenient but dangerous. If someone in a production cluster mistakenly deletes a PVC, the disk disappears with it and ends up unrecoverable. So in operations the pattern of pinning the SC’s reclaimPolicy to Retain is common. Even if a PVC is deleted, the PV remains in the Released state, and the data inside it is preserved too. If you decide cleanup is needed, a human can delete the PV explicitly, or rebind it to another PVC to recover the data.

NAME           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM             AGE
pv-...         5Gi        RWO            Retain           Released   default/data      30m

Released means the bond with the PVC is broken but the disk is alive. If you’ll have no use for this PV again, clean it up explicitly with kubectl delete pv ... — but note that at this point the cloud disk does not disappear automatically, so you have to clean it up separately in the cloud console to stop the cost.

volumeBindingMode #

The policy for when to create the PV (and disk) when a PVC is created.

Immediate is simple but invites accidents in a multi-availability-zone (AZ) environment. For example, AWS EBS is a disk tied to one AZ. If with Immediate the disk was created in ap-northeast-2a, but the Pod that mounts that PVC is placed by the scheduler on an ap-northeast-2c node — that Pod can never mount that disk on that node. It looks fine in the manifest, but it becomes an accident where the Pod is stuck in Pending.

WaitForFirstConsumer cuts off that accident at the source. Even when the PVC is created, it doesn’t create the disk and instead waits — when the Pod that mounts that PVC appears, the scheduler decides which node (which AZ) to put that Pod on, and then it creates the disk in that AZ. It’s the safe default for operations, and the SC manifest above is set to this value too. On a single-AZ cluster Immediate is no big problem, but in a multi-AZ environment you should write WaitForFirstConsumer almost as an obligation.

allowVolumeExpansion #

Set it to true and you can later grow the PVC’s spec.resources.requests.storage to grow the disk along with it. The default is false, and you don’t operate by flipping an SC that was once set to true back to false — setting it to true when first creating the SC is almost always the better choice. Shrinking a disk during operations is tricky, but growing one happens often. We’ll look at the detailed behavior again in a later section.

Mounting a PVC in a Pod #

Once a PVC becomes Bound, it’s the Pod’s turn to mount and use it. The shape of the Pod manifest is the same in any workload (Pod / Deployment / StatefulSet) — reference the PVC in spec.volumes + attach it to a path inside the container with spec.containers[].volumeMounts. It’s the same shape as the ConfigMap volume mount in Chapter 6, but the difference is that that one is temporary (refreshed on a minute scale when the ConfigMap changes) while this one is persistent.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          volumeMounts:
            - name: html
              mountPath: /usr/share/nginx/html
      volumes:
        - name: html
          persistentVolumeClaim:
            claimName: data

spec.volumes[].persistentVolumeClaim.claimName points to the PVC data created above, and that volume is mounted at the container’s /usr/share/nginx/html. From the container code’s point of view that path is just an ordinary directory, and the files written inside it survive even when the Pod dies and comes back up.

When several Pods mount one PVC #

What happens if you grow the Deployment above’s replicas to 2? If the PVC data is RWO and a block backend like EBS, mounting is possible simultaneously only when the two Pods are placed on the same node. A second Pod placed on a different node fails to mount because that disk is already attached to another node, and it’s stuck in Pending or ContainerCreating.

There are two ways to avoid this accident.

• Move the workload to an RWX backend (NFS / EFS, etc.) — create a PVC with accessModes: ReadWriteMany so several Pods use it together.
• Move the workload to a StatefulSet so each Pod uses its own PVC — each Pod has its own disk. That’s the volumeClaimTemplates of the next section.

A workload that doesn’t put data on a disk, like a stateless web server, has no PVC in the first place, so this accident doesn’t happen. The moment you need a PVC mount is exactly the moment you have to pick one of these two ways.

Revisiting a StatefulSet’s volumeClaimTemplates #

Let’s spread open again the StatefulSet manifest we saw briefly in Chapter 8.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: web
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          volumeMounts:
            - name: data
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-ssd
        resources:
          requests:
            storage: 1Gi

Expressed precisely, volumeClaimTemplates is a template that automatically creates a PVC per Pod. When the manifest above is applied with replicas: 3, K8s does the following.

1. When creating Pod web-0, it automatically creates PVC data-web-0.
2. When creating Pod web-1, it automatically creates PVC data-web-1.
3. When creating Pod web-2, it automatically creates PVC data-web-2.

The PVC naming rule is <volumeClaimTemplates.metadata.name>-<statefulset.metadata.name>-<ordinal>. In the example above the template name is data and the StatefulSet name is web, so they become data-web-0, data-web-1, data-web-2.

Each PVC is mapped to a PV through the dynamic provisioning of the SC specified by storageClassName: fast-ssd. In an AWS environment, three EBS volumes are newly created, and each is bound 1:1 to one Pod. Even when a Pod dies and comes back up, the same index (e.g., web-0) remounts the same PVC (data-web-0), so the data is preserved.

This volumeClaimTemplates + WaitForFirstConsumer SC combination rolls well even in the multi-AZ environment seen in Chapter 8 — because the EBS volume is created in whichever AZ each Pod is scheduled to, the AZ-mismatch accident is automatically cut off.

PVCs remaining on scale-down is separate from reclaimPolicy #

The behavior we noted in Chapter 8, “scaling down a StatefulSet leaves the PVCs,” is a part that’s easy to confuse with the PV’s reclaimPolicy. The two behaviors are policies at different layers.

• PVC preservation on StatefulSet scale-down — this is the StatefulSet controller’s behavior. Even if you reduce replicas, the PVC objects remain unchanged. You can change this behavior with spec.persistentVolumeClaimRetentionPolicy from 1.27.
• Handling the PV (and disk) on PVC deletion — this is the SC’s reclaimPolicy. The policy for what to do with the PV that PVC was binding and the disk behind it, when the PVC actually disappears.

Because scaling down doesn’t make the PVC disappear automatically, reclaimPolicy has no occasion to fire, and only when a human explicitly deletes the PVC does the SC’s reclaimPolicy act. These two layers together become a safety net and create the shape in which data survives an operational accident.

PVC expansion — allowVolumeExpansion #

It often happens that the disk of a PVC in operation runs short. With an SC made with allowVolumeExpansion: true, you can grow the PVC’s spec.resources.requests.storage to grow the disk along with it.

kubectl edit pvc data

spec:
  resources:
    requests:
      storage: 10Gi   # 5Gi -> 10Gi

K8s proceeds through the following steps automatically.

1. It requests a cloud disk expansion (e.g., EBS volume modification) from the CSI driver.
2. The cloud disk grows to the new size.
3. It expands the filesystem inside the container mounting that disk (xfs_growfs / resize2fs).

Most CSI drivers proceed through the above three without a Pod restart (online expansion). However, in some environment · filesystem combinations a Pod restart is required, and in this case a condition like FileSystemResizePending shows up in the PVC status. Restarting the Pod (e.g., a Deployment rolling update, deleting one Pod of a StatefulSet) finishes that step.

K8s does not support disk shrink. Writing a smaller storage in the PVC is rejected. If you want to shrink a disk, the only way is to create a new smaller PVC, move the data over, and clean up the old PVC.

Where backup and snapshots sit #

One more layer that stacks on top of the PVC and PV model is the VolumeSnapshot and VolumeSnapshotClass objects. It’s the way of expressing a cloud disk’s snapshot feature as K8s manifests, and the CSI driver must support snapshots (the CSI drivers of AWS EBS · GCP PD · Azure Disk all support them).

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: data-snap-2026-05-09
spec:
  volumeSnapshotClassName: ebs-snap
  source:
    persistentVolumeClaimName: data

When this object is created, the CSI driver takes a snapshot of the cloud disk and registers its handle as a VolumeSnapshotContent inside K8s. Later, restoring a new PVC from that snapshot is also done with a manifest (pointing the PVC’s spec.dataSource at the snapshot). We defer going deep to Chapter 26, Operations checklist and Chapter 30, Upgrade strategy, but just noting how backup · recovery stack on top of the persistent-data model is enough.

In operations, data backup usually goes one of two ways.

• Tools on top of K8s VolumeSnapshot — K8s-native backup tools like Velero or Kasten K10 manage VolumeSnapshots in bundles.
• App-level dumps — for a DB, the way of sending app-level dumps like pg_dump / mysqldump / a Redis RDB to separate storage (S3, etc.). It guarantees consistency better than a disk snapshot.

Exercises #

1. Following the body above, apply pv-static.yaml (1Gi, manual SC) and pvc-static.yaml in order, then try changing the PVC’s accessModes to ReadWriteMany or growing the storage to 2Gi and applying again. Record in time order how K8s rejects it with which error (or whether it stalls in Pending), and in one line note which of the three conditions in §“The rule by which a PVC chooses a PV” was broken.
2. When creating a dynamic-provisioning SC, try volumeBindingMode as each of Immediate and WaitForFirstConsumer. In a multi-AZ environment (EKS / GKE), you can directly reproduce the scenario where a PVC made with Immediate lands in a different AZ from the Pod and the Pod is stuck in Pending. Compare the kubectl describe pod Events of the two cases and, in your own words, summarize in one paragraph why WaitForFirstConsumer is the safe default for multi-AZ.
3. Bring up Chapter 8’s web-statefulset.yaml with replicas: 3, then force-delete a Pod with kubectl delete pod web-0 and check whether the freshly recreated web-0 remounts the same PVC (data-web-0). Next, record whether the PVCs data-web-1 and data-web-2 remain when you scale down to replicas: 1, and whether the new Pods remount the old PVCs and see the previous data when you grow back to replicas: 3. Organize how this behavior is a separate safety net from the SC’s reclaimPolicy with the model in §“PVCs remaining on scale-down is separate from reclaimPolicy.”

In one line: The persistent-data model stands on the separation of three objects — PV (the disk itself, cluster scope), PVC (the request, namespace scope), StorageClass (the blueprint for how to create a PV). The app developer writes only the PVC, and the SC’s provisioner creates a PV dynamically. The safe operational defaults are the three lines reclaimPolicy: Retain, volumeBindingMode: WaitForFirstConsumer, allowVolumeExpansion: true, and a StatefulSet’s volumeClaimTemplates automatically creates a PVC per Pod on top of this model.

Next chapter #

What we’ve covered up through this chapter was the model of how data inside a Pod survives inside the cluster. The subject of the next chapter turns the viewpoint outside the cluster — how external traffic comes into a Service inside the cluster.

In Chapter 5 we noted that among Service’s three types (ClusterIP / NodePort / LoadBalancer), LoadBalancer is the standard for external entry, but if one cluster has dozens of Services that need external exposure, running that many LoadBalancers is a burden on both cost and management. The requirement to split routing by domain or path can’t be solved with a single LoadBalancer tier either.

Chapter 10, Ingress and the Ingress Controller follows the model of the object Ingress, which centralizes that burden, and the Ingress Controller (nginx / Traefik / GKE Ingress, etc.) that resolves its manifest into actual traffic routing. HTTP / HTTPS routing, TLS termination, virtual hosts, and path-based routing are all organized into one manifest.