RBAC / ServiceAccount in Depth

Chapter 14 RBAC / NetworkPolicy / ResourceQuota covered RBAC’s four objects (Role, ClusterRole, RoleBinding, ClusterRoleBinding) and the three subjects that receive permissions (User, Group, ServiceAccount). The principle of “least privilege” and the pattern of using a standard ClusterRole scoped to a namespace with a RoleBinding was where that chapter wrapped up. This chapter adds one more layer of depth that you meet in a production cluster — Aggregated ClusterRole (extending a permission bundle by label), Impersonation (temporarily calling with another subject’s permissions), the lifecycle change of the ServiceAccount token (the projected token becoming the default in K8s 1.22 and the halt of automatic legacy-secret creation in 1.24), and the connection to external IAM (EKS’s IRSA, GKE’s Workload Identity).

By the end of this chapter you’ll have the standard operational shape where a Kubernetes ServiceAccount simultaneously holds RBAC permissions inside the cluster and cloud IAM permissions outside the cluster. The core foundation of Chapter 29 Secret Operations’s “zero passwords” pattern is this chapter’s IRSA / Workload Identity.

Aggregated ClusterRole — a permission bundle merged by label #

In a production cluster, there are times you want to extend permissions without touching a standard ClusterRole directly. For example, the view ClusterRole K8s pre-creates is an object that bundles read permissions on all standard resources. When your team adds a new resource kind defined by a CRD (CustomResourceDefinition), and you decide “people with view permission should automatically be able to read this new resource too,” you need to be able to express this behavior without directly modifying the view ClusterRole.

The object that fills this gap is the Aggregated ClusterRole. The model is simple: instead of writing its own permission list directly, a ClusterRole writes a label selector with aggregationRule, and Kubernetes gathers and merges the rules of other ClusterRoles matching that selector.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: view
aggregationRule:
  clusterRoleSelectors:
    - matchLabels:
        rbac.authorization.k8s.io/aggregate-to-view: "true"
rules: []  # empty — the controller fills it automatically

The reason rules is empty is the key. K8s’s RBAC controller scans all ClusterRoles in the cluster, gathers the rules of those with the label rbac.authorization.k8s.io/aggregate-to-view: "true", and fills them into the rules field of the ClusterRole above. When you want to add a new permission, you just create one new ClusterRole carrying that label.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: my-crd-view
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
rules:
  - apiGroups: ["myteam.example.com"]
    resources: ["widgets"]
    verbs: ["get", "list", "watch"]

Apply this single manifest and, from that moment, the read permission for the widgets resource is automatically merged into the standard view ClusterRole. Every user who was receiving view via a RoleBinding gets read permission on the new resource without changing a single line of code.

There are three standard labels — aggregate-to-view, aggregate-to-edit, aggregate-to-admin. The view / edit / admin ClusterRoles K8s pre-creates gather permissions by their respective labels. For an ops team that has adopted the CRD covered in Chapter 18 the CRD and Operator pattern, this model is the cleanest path to extending custom RBAC. The source of truth for permissions becomes scattered, but the upside of naturally extending the meaning of a standard ClusterRole is large.

Impersonation — calling with another subject’s permissions #

During operations, there are times you want to check in advance “what actions can this user perform.” Cases like verifying what actions are possible with a permission before granting it to a human user, reproducing a permission issue that came in from outside, or checking a ServiceAccount’s permission scope.

K8s’s Impersonation feature fills this need. When the caller calls the API with their own credentials and sends a header alongside saying “please act as user X for this call,” the API server performs the permission check with X’s permissions. Because X’s permissions, not the caller’s own, are applied, the caller must first have impersonate permission.

kubectl --as=alice@example.com get pods -n payments
kubectl --as=system:serviceaccount:default:my-sa get secrets

The --as option is the caller-side expression of impersonation. The first line above is a call to “view Pods in the payments namespace with the permissions of user alice@example.com,” and the second is a call to “view Secrets with the permissions of the my-sa ServiceAccount in the default namespace.” This chapter unfolds the depth of the option we touched on once in §“kubectl auth can-i” of Chapter 14.

For this call to go through, the caller’s RBAC must have the following permission.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: user-impersonator
rules:
  - apiGroups: [""]
    resources: ["users", "groups", "serviceaccounts"]
    verbs: ["impersonate"]

The impersonate verb is the key. A user with this permission can send calls with another user’s permissions beyond their own, so it’s effectively a permission that can bypass the cluster’s entire permission model. Because the impersonate permission itself is very sensitive, in a production environment it’s granted only to a small number of SRE or security-team members.

The standard tool for permission checking — kubectl auth can-i #

The day-to-day use of impersonation is usually tied to the kubectl auth can-i command.

kubectl auth can-i create secrets --as=alice@example.com -n payments

yes

This call doesn’t actually create a Secret. It only runs the API server’s permission-decision logic and returns yes / no. It’s the first tool you call to check that an intent is correct before applying a new RoleBinding, or to pin down where things are blocked when a permission issue comes in. The completed flow of the debugging tree is organized in Chapter 27 kubectl debugging patterns.

ServiceAccount token — from legacy secret to projected token #

In Chapter 14 we noted that a ServiceAccount’s token is automatically mounted at /var/run/secrets/kubernetes.io/serviceaccount/token inside the Pod. This flow changed in two stages. In K8s 1.22 the projected token became the default, and in K8s 1.24 the automatic creation of the legacy secret was halted. Since it’s a change you often run into in a production cluster, let’s pin it down in order.

The old model — a permanent token in an auto-created Secret object #

The default behavior up to K8s 1.21 was as follows. When you created a ServiceAccount, K8s automatically created a same-named Secret object and placed the ServiceAccount’s JWT token inside that Secret. This token was a permanent token with no expiry. When a Pod used that ServiceAccount, the kubelet mounted that Secret to place the token inside the container. This is the model we briefly touched on as kubernetes.io/service-account-token in the Secret type table of Chapter 6 ConfigMap and Secret.

This model had two problems.

• No expiry — once a token leaks externally, it stays valid permanently until you delete that ServiceAccount or rotate the token.
• The Pod and the token are decoupled — the token inside the Secret is independent of the Pod’s lifecycle. There’s also no audience (whom the token is for) information.

The new model — Projected Token (Bound ServiceAccount Token) #

From K8s 1.22, the projected token became the default behavior. Creating a ServiceAccount no longer auto-creates a legacy Secret object. Instead, when a Pod comes up, the kubelet issues a short-lived JWT token dedicated to that Pod and mounts it directly into the Pod’s filesystem. This token has three characteristics.

• It has an expiry — 1 hour by default, optionally adjustable shorter / longer. The kubelet automatically issues a new token before expiry and re-mounts it (rotate).
• The audience is specified — which API server’s calls this token is valid for is written inside the token.
• It’s bound to the Pod — the token is bound to the Pod’s UID, so when the Pod disappears, that token is no longer valid.

spec:
  containers:
    - name: app
      volumeMounts:
        - name: kube-api-access
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          readOnly: true
  volumes:
    - name: kube-api-access
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 3607
              audience: ""
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    fieldPath: metadata.namespace

The kubelet automatically adds this projected volume when creating a Pod. It’s a default behavior that goes into every Pod even if you don’t write it in the manifest directly. Because the path inside the container (/var/run/secrets/...) is the same as the old model, the code inside the container doesn’t need to care about the token-model change.

When you need the old way — an explicit Secret #

There remain cases where a token with no expiry is needed, like when an external tool in a CI pipeline calls the K8s API, or when an IDE’s K8s plugin needs a token to grab once and keep using. Such cases now require explicitly creating a Secret.

apiVersion: v1
kind: Secret
metadata:
  name: my-sa-token
  annotations:
    kubernetes.io/service-account.name: my-sa
type: kubernetes.io/service-account-token

The type and the annotation are the key. Create this Secret and K8s fills my-sa’s permanent token inside it. Because it’s a token with no expiry, the impact of an external leak is large — in operations it’s safer to use a projected token whenever possible, create an explicit Secret only when truly needed, and set a token rotation policy alongside it. The full operational pattern of rotation policy is covered in Chapter 29 Secret Operations.

Connecting to external IAM — IRSA and Workload Identity #

Everything so far has been about permissions on the K8s API itself. But a production cluster’s workloads call cloud services beyond the K8s API — S3 buckets, RDS databases, KMS keys, Secrets Manager secrets. These calls have their permissions evaluated outside K8s’s RBAC, that is, in the cloud’s IAM.

The traditional method was to put cloud credentials (access key + secret key) into a K8s Secret and mount it to the Pod. The problem with this method is obvious — once credentials come in as a Secret, they have no expiry, they’re hard to rotate, and once leaked their impact is large.

EKS’s IRSA (IAM Roles for Service Accounts) and GKE’s Workload Identity solve this problem the same way — they connect a K8s ServiceAccount to a cloud IAM Role and exchange the projected token for IAM’s temporary credentials. It’s a model that doesn’t store credentials statically, but issues and uses a short-lived token at call time.

The IRSA flow — EKS’s ServiceAccount + IAM Role #

IRSA’s setup is three steps.

1. Enable an OIDC provider on the EKS cluster
   → Configure AWS IAM to trust EKS's ServiceAccount JWT tokens
2. Create an AWS IAM Role and write the OIDC provider above + a specific ServiceAccount in the trust policy
   → "only tokens issued by ns/my-app's SA = app-sa can take this Role"
3. Attach a Role ARN annotation to the K8s ServiceAccount
   → eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT:role/my-app-role

On a cluster where this setup is done, when a Pod calls the AWS API the following flow happens automatically.

1. Read the projected token mounted in the Pod
2. Send that token to AWS STS's AssumeRoleWithWebIdentity API
3. STS verifies the token with the EKS OIDC provider
4. Check whether it matches the ServiceAccount written in the trust policy
5. If it matches, return the IAM Role's temporary credentials (15 minutes ~ 12 hours)
6. The AWS SDK calls S3 with those credentials

Since all these steps happen automatically inside the AWS SDK, the application code doesn’t handle credentials directly. The code just calls boto3.client('s3') and that’s it; the credential chain inside the SDK follows the flow above.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
  namespace: my-app
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-app-s3-role
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  namespace: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
    spec:
      serviceAccountName: app-sa  # ← use the SA created above
      containers:
        - name: app
          image: my-app:latest

One ServiceAccount and one IAM Role are bound 1:1, and every Pod using that ServiceAccount gets that IAM Role’s permissions. The need to store static credentials in a K8s Secret disappears. The hands-on setup of IRSA on EKS (creating the OIDC provider · IAM Role with Terraform, etc.) is covered in Chapter 21 EKS Cluster Setup, and the “zero DB passwords” pattern combined with RDS IAM auth is organized in Chapter 23 DB Integration — RDS · External Secrets.

GKE Workload Identity — the same model, a different name #

GKE’s Workload Identity is essentially the same model. It binds a K8s ServiceAccount to a GCP IAM Service Account and exchanges the projected token through GCP STS to receive temporary credentials.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
  namespace: my-app
  annotations:
    iam.gke.io/gcp-service-account: my-app-sa@PROJECT.iam.gserviceaccount.com

Only the annotation’s key and value format differ. AKS also adopted the same model with Azure Workload Identity, and the shape is a per-cloud-provider adapter layered on top of the identical structure of OIDC + STS-based token exchange.

Operational principles to pin down at adoption #

IRSA and Workload Identity are nearly standard on the security front, but let’s pin down a few operational principles for the setup stage.

• 1 ServiceAccount = 1:1 mapping to one cloud IAM Role — don’t pile too many permissions onto one SA; split ServiceAccounts per workload and attach a least-privilege IAM Role to each SA.
• Specify both namespace and SA name in the trust policy — if you specify only the namespace, another SA in that namespace can take the same Role, breaking isolation.
• Token audience verification — when STS verifies the token, it checks the audience (sts.amazonaws.com, etc.). You must match the audience of the ServiceAccount’s projected token to the value STS expects.
• Credential lifetime — the IAM Role’s temporary credentials are issued at STS-call time and are usually valid for 1 hour. The AWS SDK refreshes them automatically, so no application-side handling is needed.

Exercises #

1. Check how the aggregationRule of the standard view ClusterRole is written in your cluster (kubectl get clusterrole view -o yaml). Organize the shape where rules is empty and only aggregationRule exists into one paragraph against the model of §“Aggregated ClusterRole,” and write one manifest yourself that merges a new CRD’s read permission into view.
2. Check whether the default ServiceAccount can create Pods with kubectl auth can-i create pods --as=system:serviceaccount:default:default -n default. Then recall the pod-reader RoleBinding from Chapter 14, and reproduce, with the model of §“Impersonation,” the flow where kubectl auth can-i list pods --as=system:serviceaccount:dev:pod-reader -n dev responds yes from that SA’s standpoint. Also record what error occurs when a user without impersonate permission attempts the same command.
3. In an EKS environment, attach an IRSA annotation to one ServiceAccount and simulate, through the six steps of §“The IRSA flow,” how the AWS SDK inside a Pod using that SA obtains credentials. Organize into one paragraph what isolation problem arises if you write only the namespace and not the SA name in the trust policy, and how Chapter 29 Secret Operations’s “zero passwords” pattern layers on top of this chapter’s IRSA.

In one line: Aggregated ClusterRole extends a standard permission bundle by label, and Impersonation verifies another subject’s permission intent with kubectl auth can-i. Since K8s 1.22, the ServiceAccount token has used a projected token with expiry · audience · rotation by default, and since 1.24 the automatic creation of the legacy Secret has been halted. IRSA / Workload Identity is a model that maps a K8s ServiceAccount 1:1 to a cloud IAM Role and pulls static credentials out of the cluster, with a per-cloud-provider adapter layered on top of the common structure of OIDC + STS token exchange.

Next chapter #

Up through this chapter we’ve organized the depth of the permission model of RBAC and ServiceAccount. The next chapter’s subject shifts the vantage point one notch again — the stage that inspects · transforms a manifest just before it’s stored in etcd. If RBAC asks “can this user perform this action,” the next chapter’s admission asks “does this manifest conform to the cluster’s policy.”

Chapter 17 Admission Controller follows the K8s standard model of ValidatingAdmissionWebhook · MutatingAdmissionWebhook, a comparison of the two tools that layer a policy engine on top — OPA Gatekeeper (based on the Rego language) and Kyverno (K8s-native YAML policy) — and the manifest patterns of operational policies like “reject containers without limits” and “enforce specific labels.” The path to enforcing, at the admission stage, policies like the “no cluster-admin bundle” we touched on in §“A common pitfall — too-broad ClusterRole” of Chapter 14 opens up.