K8s Intermediate #7: RBAC / NetworkPolicy / ResourceQuota — Security and Resource Policy

The final post in the K8s Intermediate series. From #1 through #6 we built up the model of operating workloads layer by layer. Four kinds of controllers, persistent data, external entry points, resource requests/limits, health checks, and autoscaling — one complete cycle of stably bringing up a single Pod and scaling it with load is now in hand. This post covers the topic stacked one more layer on top — security and resource control for environments where multiple teams and namespaces share one cluster. There are three keywords: RBAC (who can do what), NetworkPolicy (what traffic flows), and ResourceQuota (how much can be created). All three objects share namespace-level scope, and they fill the gap noted in Basics #7 — that “Namespace itself is not a security boundary.” Because this is the last post in the series, it also includes a full 7-post retrospective and a preview of the next track (K8s Advanced).

This series is K8s Intermediate, 7 posts.

• #1 StatefulSet / DaemonSet / Job / CronJob — Controllers beyond Deployment
• #2 PV / PVC / StorageClass — the persistent data model
• #3 Ingress and Ingress Controller — the external entry point
• #4 resources.requests / limits — Pod resource requests and limits
• #5 Health checks — liveness / readiness / startup probes
• #6 Autoscaling — HPA / VPA / Cluster Autoscaler
• #7 RBAC / NetworkPolicy / ResourceQuota — security and resource policy ← this post

Common coordinates of the three policies — the namespace level #

In Basics #7 when covering Namespace, one line was left — Namespace itself is not a security boundary. It’s just a logical compartment that splits object names; the real isolation is made by policy objects layered on top. The body of those layered policies is the three topics of this post.

All three objects are either applied at the namespace level (NetworkPolicy, ResourceQuota, LimitRange, Role/RoleBinding) or are cluster-scoped but bound to a namespace (the ClusterRole + RoleBinding combination). The isolation of multi-tenant operation — where dev / staging / prod or team A / team B share one cluster — is only complete when all three dimensions are in place.

The flow of this post is simple. Permission first — RBAC covered in the most detail, then traffic (NetworkPolicy), then resource (ResourceQuota / LimitRange).

RBAC — who can do what #

RBAC (Role-Based Access Control) is a model expressing permissions at the K8s API call level. Every action like kubectl get pods, kubectl create deployment, kubectl delete secret is in the end an HTTP request to the K8s API server. RBAC checks for each one of those requests “can this subject use this verb on this resource.”

The model is expressed by four objects. Permission bundles (Role / ClusterRole) and objects connecting permissions to subjects (RoleBinding / ClusterRoleBinding).

There is one subtle point here. RoleBinding can also reference a ClusterRole. A ClusterRole is a permission bundle, and a RoleBinding is the object that decides in which namespace and to whom that bundle is granted. The most common operational pattern is to “create a standard ClusterRole (e.g., view, edit) once and grant it to different people in each namespace via RoleBinding.”

Subject — the receiving side of permission #

Subjects receiving permission are three kinds:

• User — a human user. K8s itself has no user database; external authentication (OIDC, x509 client certificates, cloud IAM mapping, etc.) tells K8s the username.
• Group — a group of users. Like User, group info is delivered to K8s by external authentication.
• ServiceAccount — an ID used by Pods when they access the K8s API. Not a person but the identity of a workload.

Among these three, what is directly created and managed via K8s manifests is almost exclusively ServiceAccount. Users and Groups are produced by external authentication and do not exist as objects inside K8s — they appear only in the subjects field of a RoleBinding.

ServiceAccount — the Pod’s ID #

Every Pod lives in the cluster carrying the ID of some ServiceAccount. If the manifest doesn’t write spec.serviceAccountName, the namespace’s default ServiceAccount is automatically attached. When a tool like kubectl inside the Pod accesses the K8s API, that call is performed with that ServiceAccount’s permissions.

kubectl get serviceaccounts -n default

NAME      SECRETS   AGE
default   0         3d

The default default ServiceAccount has no permissions granted. In an operational cluster with RBAC applied, attempting kubectl get pods from inside a Pod naturally results in:

Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" in the namespace "default"

Seeing this message and binding permissions via RoleBinding makes the same command work. The flow follows in the next example.

A bundle of RBAC manifests #

Cleaning up the simplest scenario in one manifest bundle. Create a ServiceAccount called pod-reader in the dev namespace, grant that SA “the permission to read Pods,” and verify that kubectl get pods works inside a Pod using that SA.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: pod-reader
  namespace: dev
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: dev
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-reader
  namespace: dev
subjects:
  - kind: ServiceAccount
    name: pod-reader
    namespace: dev
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

The responsibility of each object in one line:

• ServiceAccount pod-reader — a new ID in the dev namespace. Yet to receive any permission.
• Role pod-reader — a permission bundle allowing get, list, watch verbs on the pods resource. apiGroups: [""] points to the core API group (Pod, Service, ConfigMap, etc.).
• RoleBinding pod-reader — the object connecting the Role and the ServiceAccount. subjects is the receiving side, roleRef the giving side.

After applying these three at once, spin up a Pod using that ServiceAccount.

apiVersion: v1
kind: Pod
metadata:
  name: reader
  namespace: dev
spec:
  serviceAccountName: pod-reader
  containers:
    - name: kubectl
      image: bitnami/kubectl:1.30
      command: ["sleep", "3600"]

kubectl apply -f rbac-pod-reader.yaml
kubectl apply -f reader-pod.yaml
kubectl exec -n dev reader -- kubectl get pods -n dev

NAME     READY   STATUS    RESTARTS   AGE
reader   1/1     Running   0          30s

Inside the same Pod, attempting an unauthorized action confirms it’s blocked.

kubectl exec -n dev reader -- kubectl create deployment nginx --image=nginx -n dev

error: failed to create deployment: deployments.apps is forbidden: User "system:serviceaccount:dev:pod-reader" cannot create resource "deployments" in API group "apps" in the namespace "dev"

Because the pod-reader Role only granted get/list/watch on Pods and didn’t grant create on Deployments, denial happens precisely there.

The core of verbs and resources #

Putting frequently used verbs in Role / ClusterRole’s rules in a table:

To allow only reading, give get, list, watch together. Because commands like kubectl get internally use list, and watch is used to refresh the informer cache. To allow writing too, add create, update, patch, delete.

resources writes plural names like pods, services, configmaps. apiGroups is the API group that resource belongs to. The core group (Pod / Service / ConfigMap / Secret) is [""] (empty string), Deployment / StatefulSet / DaemonSet are ["apps"], Job / CronJob are ["batch"], Ingress is ["networking.k8s.io"]. Which group a resource belongs to can be checked at once with kubectl api-resources.

kubectl api-resources

kubectl auth can-i — verifying permissions #

After touching RBAC, the next question soon follows — “so, can this user really do this action?” Often, looking at manifests for a long time doesn’t make the result clear at a glance. The command K8s answers this question with directly is kubectl auth can-i.

kubectl auth can-i create pods -n dev
kubectl auth can-i delete deployments -n prod

Outputs yes or no. To ask from another user or ServiceAccount’s perspective, use the --as option for impersonation.

kubectl auth can-i create pods --as=alice -n dev
kubectl auth can-i list secrets --as=system:serviceaccount:dev:pod-reader -n dev

The --as option itself needs permission (impersonation permission). It’s the most frequently used pattern when an operational cluster admin verifies that newly created RBAC policy works as intended. To see all permissions at once, append --list.

kubectl auth can-i --list --as=alice -n dev

Common trap — too-broad ClusterRole #

The most common mistake when first applying RBAC is the pattern of giving in to laziness and binding the ClusterRole cluster-admin to a ServiceAccount.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: app-admin
subjects:
  - kind: ServiceAccount
    name: app
    namespace: dev
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

cluster-admin grants unrestricted access to the entire K8s API. If the token of a ServiceAccount with this ClusterRoleBinding leaks from even a single Pod, that token can read every Secret in every namespace of the cluster and delete any object. A single vulnerability in a library inside the container image turns that one exposure into a cluster-wide incident.

The standard operational principle is least privilege. Pick exactly the verbs and resources the workload truly needs, bundle them as a Role, and grant via RoleBinding. K8s pre-creates a set of standard ClusterRoles, and the general starting point is to use them via RoleBinding scoped to a namespace.

By scoping view / edit / admin to specific namespaces via RoleBinding, you can quickly grant appropriate permission sets to people, teams, and services; the standard practice is to grant cluster-admin only to a handful of cluster operators via ClusterRoleBinding.

automountServiceAccountToken #

A ServiceAccount’s token is by default auto-mounted into the Pod at /var/run/secrets/kubernetes.io/serviceaccount/token. For Pods that have no reason to call the K8s API, it’s safer not to mount even that token. You can turn it off via manifest’s automountServiceAccountToken: false.

apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  automountServiceAccountToken: false
  containers:
    - name: nginx
      image: nginx:1.27

With this one line, even if the Pod is somehow compromised, there’s no token for direct K8s API access. A regular recommendation in security guides.

NetworkPolicy — controlling traffic between Pods #

If RBAC handles permission to the K8s API, NetworkPolicy handles IP traffic between Pods. When two Pods in the same cluster know each other’s IP and try to communicate, the policy that checks whether that traffic is allowed.

The default is all-pass #

The default of K8s’s network model is simple — without NetworkPolicy, all Pods can communicate with each other. Same namespace or different, knowing the Pod’s IP is enough to freely send packets. In a multi-tenant cluster, leaving this default in place results in Pods of one namespace freely accessing DB Pods of another namespace.

The behavior rule of NetworkPolicy can be summarized in two lines:

• For any Pod matched by at least one NetworkPolicy, default-deny is applied for the directions listed in that policy’s policyTypes.
• Only traffic explicitly allowed by that policy’s ingress / egress rules is permitted.

A Pod with no matching policy has the rule above not triggered, so all traffic passes. With even one matching policy, that direction switches to a whitelist model.

CNI must support NetworkPolicy #

NetworkPolicy is standard at the K8s manifest level, but the actual blocking of traffic is done by the CNI (Container Network Interface) plugin. So if the CNI doesn’t support NetworkPolicy, applying the manifest does nothing. Traffic still flows.

If you plan to use NetworkPolicy in an operational cluster, you must choose the CNI at cluster creation time. Applying a NetworkPolicy manifest to a default EKS cluster and then asking “why isn’t traffic being blocked?” is a common trap — without CNI support, the manifest is just an object sitting in etcd with no effect.

default-deny → allow pattern #

The standard operational pattern is to lay down a default-deny policy in the namespace and explicitly allow only the necessary communication.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: prod
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

podSelector: {} is an empty selector pointing to “all Pods in this namespace.” With both Ingress and Egress in policyTypes and not a single ingress / egress rule line, all Pods in this namespace are blocked from incoming and outgoing traffic.

Leaving it like this prevents Pods from even doing DNS lookups, so at minimum DNS traffic must be allowed.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: prod
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Then add per-workload necessary communication one manifest at a time — frontend going to backend on 80/TCP, backend going to DB on 5432/TCP, and so on.

NetworkPolicy manifest — frontend → backend #

Writing the most common shot in one manifest. An ingress policy where backend Pods only receive 8080/TCP traffic incoming from frontend Pods.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-allow-frontend
  namespace: prod
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

How to read it:

• spec.podSelector — Pods this policy applies to. Only Pods labeled app=backend.
• policyTypes: [Ingress] — clarifies this is a policy on the incoming direction. egress isn’t controlled by this policy.
• ingress[0].from — where to allow traffic from. From Pods labeled app=frontend.
• ingress[0].ports — on which ports. Only 8080/TCP.

The selector for from can be picked from three — and these three can be used separately or together.

Placing namespaceSelector and podSelector together in the same from item means “Pods with a specific label in a specific namespace.” Writing them as separate items means “all Pods in that namespace OR Pods with that label in the same namespace.” You need to know exactly which behavior is intended and write accordingly.

ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            env: prod
        podSelector:
          matchLabels:
            app: frontend

ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            env: prod
      - podSelector:
          matchLabels:
            app: frontend

That these two manifests differ in meaning is a regular trap of NetworkPolicy. The one above is “only frontend Pods in env=prod namespace,” and the one below is “all Pods in env=prod namespace OR frontend Pods in the same namespace.”

egress rules — outgoing direction #

The mirror of ingress is egress. The policy of allowing only backend Pods to go to DB on port 5432 looks like this.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-egress-to-db
  namespace: prod
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432

With this single policy applied, backend Pods can’t go anywhere except postgres Pod’s 5432/TCP. Combined with the default-deny + allow-dns above, backend Pods’ outgoing communication is limited to “DNS + DB.” Narrowing exactly via whitelist what each workload’s outbound communication is, is the standard operational security shape.

Limits of NetworkPolicy #

NetworkPolicy is L3/L4 policy. It only sees IP, port, and protocol. L7 policies like HTTP method or path are outside NetworkPolicy’s scope — Cilium’s L7 policy or service meshes like Istio / Linkerd handle that dimension. NetworkPolicy also applies only to traffic inside the cluster — external inbound is controlled at the Ingress / LoadBalancer layer, and external outbound is controlled separately at the NAT gateway and security group level. A cluster’s overall security posture is built from multiple layers of policy working together.

ResourceQuota — namespace resource total cap #

In #4 we covered the per-container resource model — requests and limits. Objects defining a single container’s guarantee and cap. What stacks one more layer on top is per-namespace total caps in the form of ResourceQuota.

The operational scenario is clear. When dev / staging / prod or multiple teams share a single cluster, you must prevent the dev namespace from consuming all available CPU and starving prod workloads. ResourceQuota fills that gap.

ResourceQuota manifest #

apiVersion: v1
kind: ResourceQuota
metadata:
  name: dev-quota
  namespace: dev
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    pods: "50"
    services: "20"
    configmaps: "30"
    secrets: "30"
    persistentvolumeclaims: "10"
    requests.storage: 100Gi

With this ResourceQuota applied to the dev namespace, the following totals can’t exceed the limits:

• Sum of requests.cpu across all Pods in that namespace ≤ 4 cores
• Sum of requests.memory ≤ 8Gi
• Sum of limits.cpu ≤ 8 cores
• Sum of limits.memory ≤ 16Gi
• Object counts — 50 Pods, 20 Services, 30 ConfigMaps and Secrets each, 10 PVCs
• Sum of PVC requests.storage ≤ 100Gi

Object creation exceeding the cap is rejected at the K8s API server. With 4 cores already allocated in the dev namespace, attempting to create an additional Pod with requests.cpu: 1 has the Pod creation request rejected at the admission stage with the following message:

Error from server (Forbidden): error when creating "...": pods "..." is forbidden: exceeded quota: dev-quota, requested: requests.cpu=1, used: requests.cpu=4, limited: requests.cpu=4

Verifying ResourceQuota behavior #

kubectl get resourcequota -n dev
kubectl describe resourcequota dev-quota -n dev

Name:            dev-quota
Namespace:       dev
Resource         Used    Hard
--------         ----    ----
configmaps       12      30
limits.cpu       6       8
limits.memory    12Gi    16Gi
pods             18      50
persistentvolumeclaims  3   10
requests.cpu     3       4
requests.memory  6Gi     8Gi
requests.storage 30Gi    100Gi
secrets          15      30
services         8       20

The shape where the Used / Hard table appears on one page. In an operational cluster, when checking how full a namespace is, describe is the fastest tool.

Pairing with LimitRange — per-container defaults and caps #

One subtle trap of ResourceQuota: when creating a Pod in a namespace where ResourceQuota is active, omitting requests/limits on a container causes the Pod to be rejected. For ResourceQuota to compute totals, every container must have resource values, and the admission check fails for containers that omit them.

The operational safety net for this part is LimitRange. LimitRange is an object setting per-container defaults and max/min.

apiVersion: v1
kind: LimitRange
metadata:
  name: dev-limits
  namespace: dev
spec:
  limits:
    - type: Container
      default:
        cpu: 200m
        memory: 256Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      max:
        cpu: "2"
        memory: 2Gi
      min:
        cpu: 50m
        memory: 64Mi

When a Pod manifest comes into a namespace with this LimitRange applied:

• If the container has no requests, defaultRequest (100m / 128Mi) is auto-filled
• If the container has no limits, default (200m / 256Mi) is auto-filled
• If the container’s requests or limits exceed max (2 / 2Gi), rejected
• If below min (50m / 64Mi), rejected

Splitting ResourceQuota and LimitRange’s responsibilities, one line each:

The operational approach is to have both. LimitRange fills in gaps from manifests that omit resource values, and ResourceQuota prevents the accumulated totals from exceeding the caps. Both must coexist for multi-tenant resource policy to run stably in production.

scopes / scopeSelector — applied to only some Pods #

ResourceQuota by default applies to all Pods in a namespace, but scope can narrow it. A frequently used pattern is splitting by PriorityClass — for example, separately limiting only the resource total for high-priority workloads, or separately limiting only BestEffort QoS Pods.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: high-priority-quota
  namespace: dev
spec:
  hard:
    requests.cpu: "2"
    requests.memory: 4Gi
  scopeSelector:
    matchExpressions:
      - operator: In
        scopeName: PriorityClass
        values: ["high"]

In the basic operational shape, starting with one whole-namespace policy without scope and adding scoped policies when needed is the safer way.

Three policies’ collaboration — isolation in a multi-tenant cluster #

Cleaning up the shape of isolation made by the three objects in one picture:

[ RBAC ]                 Who can create objects
   │                     (verbs × resources × namespace)
   │
[ NetworkPolicy ]        Who the created Pods communicate with
   │                     (podSelector × from/to × ports)
   │
[ ResourceQuota ]        How much that namespace can create
   │                     (cpu/memory totals + object counts)
   │
[ LimitRange ]           Per-container defaults and caps
                         (default / max / min)

The three objects look like they run separately, but in reality they make one namespace’s isolation together. RBAC applied to the dev namespace lets only the dev team touch objects inside, NetworkPolicy blocks dev Pods from going to prod’s DB traffic, and ResourceQuota caps how much CPU/memory/object count dev can consume. With all three in place, isolation where an incident in dev doesn’t leak to staging and prod is established.

For this isolation to work cleanly, one more precondition is needed — a well-designed namespace structure. Deciding how to divide namespaces by environment (dev / staging / prod), team (team-a / team-b), or service unit is a decision to make at initial cluster setup. Leaving that structure vague while trying to apply RBAC / NetworkPolicy / ResourceQuota means the policy boundaries shift with every change, and there is never a stable baseline to align against.

Series retrospective — what entered hands through 7 K8s Intermediate posts #

As the last post in the series, here’s a look back at all seven. If the Basics series brought us to the stage of reading and writing one manifest, the Intermediate series added one operational layer at a time on top.

• #1 — StatefulSet / DaemonSet / Job / CronJob. Four kinds of controllers beyond Deployment, covering the four patterns: workloads needing identity and disk, workloads needing one per node, one-shot tasks, and periodic execution tasks.
• #2 — PV / PVC / StorageClass. The persistent data model — static and dynamic provisioning, accessModes, reclaimPolicy, volumeBindingMode, allowVolumeExpansion, and what StatefulSet’s volumeClaimTemplates produces.
• #3 — Ingress and Ingress Controller. The object that gathers external entry points in one place and the controller that resolves those manifests into actual traffic routing, covering HTTP/HTTPS, TLS termination, virtual hosts, and path-based routing.
• #4 — resources.requests / limits. Per-container resource requests and caps, with requests as the scheduling baseline and limits as the OOM/CPU throttling baseline, plus the flow of the three QoS tiers determining eviction priority.
• #5 — Health checks. liveness / readiness / startup probes, looking at how K8s judges whether a container is alive, service-ready, or in the initialization phase.
• #6 — HPA / VPA / Cluster Autoscaler. The three dimensions of Pod count, Pod resources, and node count automatically changing with load, also flagging the metrics-server, custom metrics, and HPA + VPA conflict trap.
• #7 — RBAC / NetworkPolicy / ResourceQuota. Through security and resource policy, the three dimensions of multi-tenant cluster isolation — who is allowed, what traffic is allowed, how much is allowed.

Having followed all seven, you are at the stage where you can read the intent and operational traps of any object you encounter in a company’s cluster manifest directory. Seeing kind: StatefulSet you naturally think of volumeClaimTemplates and headless Service; seeing kind: Ingress you ask what Ingress Controller backs it; seeing an empty limits under resources you suspect both OOM risk and the absence of a LimitRange. The model of how a single manifest runs inside the cluster has settled into a clear mental picture.

Next track — K8s Advanced #

The deep topics intentionally deferred in the K8s Intermediate series are the storyline of the K8s Advanced track. Planned as 6 posts, summarized in advance:

Once these six topics are all covered, you take one more step toward seeing K8s with the eyes of a person setting up a cluster. The track moves from the stage of writing manifests well to the stage of deciding which policy engine to install, which observability stack to pick, and how to organize the GitOps pipeline.

After that — K8s Practice #

After the advanced track, K8s Practice is being prepared as 6 posts. If the advanced track covers the depth of K8s’s object model and policy, the practice track is one cycle of putting a real service on top of it and operating it.

Going through both tracks (advanced + practice) brings you close to the full perspective of a person who adopts and operates K8s. The end of the Intermediate series is the midpoint of that bigger picture — the object model is in hand, and the depth of policy, extension, and operations layered on top is what comes next.

Closing #

The K8s Intermediate series of 7 posts is complete. This post covered the three policy objects — RBAC, NetworkPolicy, ResourceQuota — that together make multi-tenant cluster isolation possible. RBAC controls K8s API permissions, NetworkPolicy controls traffic between Pods, and ResourceQuota (together with its partner LimitRange) controls namespace resource totals. The model in which multiple environments and teams can safely share one cluster is only established when all three dimensions are applied together. Looking at the series as a whole: Basics #7 was the stage of reading and writing a single manifest, and Intermediate #7 was the stage of adding one operational layer at a time on top of that foundation. In the next track, K8s Advanced, topics ranging from CNI depth and RBAC internals to policy engines, CRD/Operator, observability, and GitOps will be covered from the perspective of setting up and operating clusters. Then in K8s Practice, one end-to-end cycle of putting a real service on EKS will be followed through.