K8s Advanced #6: GitOps — ArgoCD / Flux

The last post in the K8s Advanced series. From #1 CNI to #5 Observability, we built up the cluster’s data plane, permissions, policy, extension, and observation layer by layer. This post covers the way itself that all those manifests enter the cluster — GitOps. Instead of a person manually running kubectl apply, the operational model where the source of truth for manifests is placed in git and a controller inside the cluster watches git to sync automatically. ArgoCD and Flux are the two standard implementations of this model, and as the last post in the series, this organizes both tools’ models, operational patterns, series retrospective, and the next track in one cycle.

This series is K8s Advanced, 6 posts.

• #1 CNI in depth — Calico / Cilium / eBPF
• #2 RBAC / ServiceAccount in depth — Aggregated ClusterRole / Impersonation / IRSA / Workload Identity
• #3 Admission Controller — OPA Gatekeeper / Kyverno
• #4 CRD and the Operator pattern — controller-runtime
• #5 Observability — Prometheus / Grafana / Loki / OpenTelemetry
• #6 GitOps — ArgoCD / Flux ← this post

Push model and Pull model #

First, let’s compare with the model that was the standard before GitOps. The way CI/CD pipelines apply manifests to clusters is largely two paths.

Traditional CD pipelines were the push model. GitHub Actions or Jenkins ran kubectl apply -f manifests/ after building and that was it. The problems with this model are three:

• CI system holds strong cluster credentials — when the CI system is compromised, the cluster is compromised.
• Drift isn’t visible — when someone runs kubectl edit directly on the cluster, the git manifests diverge from the actual cluster, but there’s no standard mechanism to detect that divergence.
• Hard to scale to multiple clusters — to apply the same manifests to N clusters, you have to run kubectl apply N times.

The GitOps model is a path that solves these three at once. Since a controller inside the cluster watches git, there’s no need to hold cluster credentials externally; that controller continuously watches the sync state, so drift is automatically detected; and the model where each cluster watches its own git makes scaling to N natural.

The four principles of GitOps #

The four principles of GitOps organized by the OpenGitOps project are as follows.

K8s manifests are declarative, and git is versioned + immutable. ArgoCD and Flux complete GitOps by adding pull + reconciliation on top.

ArgoCD — model centered on Application CRD #

ArgoCD is a GitOps tool created by Intuit and donated to CNCF. The biggest characteristic is the rich web UI. The sync state, drift, and manifest change history of every Application in the cluster can be seen on one screen, lowering the entry barrier for operations teams.

Application CRD — ArgoCD’s unit #

The unit by which ArgoCD pulls manifests from git and syncs to the cluster is the Application CRD.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/manifests.git
    targetRevision: main
    path: apps/my-app/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: my-app
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

When this manifest is applied to ArgoCD, the following happens automatically.

1. ArgoCD controller pulls the path directory of repoURL from git
2. Auto-detects Kustomize / Helm / plain YAML and renders manifests
3. Syncs to the cluster’s destination (auto-recovers drift since automated.selfHeal: true)
4. Auto-syncs again when manifests in git change (automated)
5. Objects removed from git are also deleted from cluster (prune: true)

App of Apps — managing Applications as a bundle #

The pattern for managing multiple Applications in one place is App of Apps. A structure where one Application’s source points to a directory containing other Application manifests.

manifests/
  apps/
    root.yaml              ← root Application (manually applied to ArgoCD)
    children/
      app-a.yaml           ← Application: app-a
      app-b.yaml           ← Application: app-b
      app-c.yaml           ← Application: app-c
  ...

Just registering one root Application to ArgoCD initially, child Applications are created in turn from inside it, and each child Application syncs its own manifests. To add a new app to the cluster, just add one new child Application to git.

Sync Wave — ordered application #

There are cases where manifest application needs ordering — create Namespace first, then ConfigMap inside, then Deployment. ArgoCD expresses this order via annotation.

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "0"   # Namespace
---
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"   # ConfigMap, Secret
---
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "2"   # Deployment, StatefulSet

Lower waves are applied first, and after all objects in that wave reach healthy state, the next wave proceeds. The pattern of creating CRDs first and then applying instances of those CRDs is the most frequently encountered use case.

Flux — bundle of small components #

Flux is a GitOps tool made by Weaveworks, in the same category as ArgoCD but with a different approach. Flux v2 is designed not as one big component but as a bundle of multiple small controllers.

Each controller has its own CRD, and all actions are expressed via that CRD’s manifest.

GitRepository + Kustomization — Flux’s basic bundle #

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: manifests
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/myorg/manifests.git
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: my-app
  namespace: flux-system
spec:
  interval: 5m
  path: ./apps/my-app/overlays/prod
  prune: true
  sourceRef:
    kind: GitRepository
    name: manifests
  targetNamespace: my-app

GitRepository watches git, and Kustomization applies one directory of that git to the cluster. Thanks to the separation of the two objects, the same git can be pointed to by multiple Kustomizations at different paths.

HelmRelease — making Helm charts GitOps-native #

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: prometheus
  namespace: monitoring
spec:
  interval: 10m
  chart:
    spec:
      chart: kube-prometheus-stack
      version: "55.x"
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
  values:
    prometheus:
      prometheusSpec:
        retention: 30d

Instead of running helm install by hand, expressing as a HelmRelease manifest brings Helm chart installation/upgrade into the GitOps flow.

ArgoCD vs Flux — the grain of selection #

The grain of selection usually follows:

• If the operations team prefers GUI and wants to see all clusters on one screen — ArgoCD is natural.
• If you want to express all operations as manifests and prefer a bundle of small components — Flux fits well.

Both tools are CNCF graduated projects and have been hardened at operational scale. Picking the wrong one isn’t the kind of decision that causes a major incident.

Directory structure patterns #

The directory structure of the GitOps repo varies by operational team’s style, but two patterns are commonly seen.

1. env-per-folder — branch by environment #

manifests/
  base/
    my-app/
      deployment.yaml
      service.yaml
  envs/
    dev/
      kustomization.yaml      ← base + dev patch
    staging/
      kustomization.yaml
    prod/
      kustomization.yaml

A structure that uses Kustomize’s base + overlay pattern as is. Per-environment differences (replicas, image tag, resources) go into overlays as patches.

2. app-per-folder + branch-per-env #

manifests/        (main branch = prod)
  apps/
    my-app/
      deployment.yaml
      service.yaml
    other-app/
      ...

manifests-dev/    (dev branch)
manifests-staging/ (staging branch)

A model that uses branches as environment branches. Environment differences are expressed as commits, making audit natural, but the sync burden between branches is large.

In operations, env-per-folder is more frequently used. Since changes flow within one branch (main), PR review is simple.

How to put Secrets in git #

One of GitOps’s big homework is the path to put secrets in git. K8s Secrets can’t be put in git as plaintext. Three standard tools exist.

External Secrets Operator is the most frequently used path. The source of truth for secrets is in the external store, and only the reference enters K8s, so secret rotation finishes at once in the external store. Combined with #2 IRSA, even the credentials for accessing the secret store don’t need to be statically held inside the cluster.

Operational principles to lock in #

1. auto-sync vs manual sync — branch by environment #

Turning on syncPolicy.automated reflects git changes immediately to the cluster. Setting dev / staging to auto and prod to manual (or auto after PR merge) is common. When applying auto-sync to prod, locking in safety devices like syncOptions: PruneLast=true to handle deletions last is recommended.

2. The meaning of drift detection #

GitOps controllers continuously compare git with the actual cluster. When someone modifies via kubectl edit directly, that change is immediately detected as drift, and with selfHeal: true, it’s overwritten with the git manifest. This is GitOps’s strength but also a trap — fields auto-generated by controllers (status, auto-labels) shouldn’t appear as drift. Both ArgoCD / Flux can write fields to ignore via ignoreDifferences setting.

3. Impact of Helm value changes #

Changing HelmRelease values redeploys all objects that chart created. To prevent unintended redeploys, checking the impact scope of value changes via dry-run at the PR stage in advance is recommended.

4. Hub-spoke model for multi-cluster #

When managing N clusters via GitOps, two standard models exist.

• Each cluster has its own GitOps controller — cluster self-contained, low external dependency
• One hub cluster’s GitOps controller manages spoke clusters — operational simplification, hub availability is critical

ArgoCD fits both models well, and Flux fits the first model naturally.

Series retrospective — what entered hands through 6 K8s Advanced posts #

Being the last post, here’s a look back at all six.

• #1 — CNI in depth. The four conditions of K8s network model, CNI interface, the data plane of iptables / IPVS / eBPF, comparison of Calico and Cilium.
• #2 — RBAC / ServiceAccount in depth. Aggregated ClusterRole, Impersonation, projected token, connecting K8s ServiceAccount to cloud IAM via IRSA / Workload Identity.
• #3 — Admission Controller. The 5-stage flow of API server, mutating / validating webhooks, comparison of OPA Gatekeeper and Kyverno policy engines.
• #4 — CRD and Operator pattern. The path to extend K8s API itself, the skeleton of controller-runtime-based Operator, ownerReference / finalizer / status subresource.
• #5 — Observability. The three axes of metrics / logs / traces, Prometheus + kube-state-metrics, Loki, OpenTelemetry, Grafana, Alertmanager.
• #6 — GitOps. The operational model placing manifest source of truth in git, ArgoCD and Flux, directory structure and secret management.

The Basics series added the model of one manifest, the Intermediate series added the depth of how that manifest runs in operational clusters, and the Advanced series added the depth of policy engines, extension, observation, and synchronization layered on top. At the point of having followed all 20 posts, the view of a person who adopts and operates K8s — the view at the stage of deciding “which CNI to adopt, which policy engine to adopt, which observability stack to pick, how to organize the GitOps pipeline” — is in hand.

Next track — K8s Practice #

While the Advanced series covered the depth of K8s’s object model and policy, K8s Practice — 6 posts — is one full cycle of putting a real service on top of that foundation and operating it. Bringing back the table from Intermediate #7:

If the Basics, Intermediate, and Advanced tracks were paths to understanding K8s at the manifest level, the Practice track follows one real service from scratch on EKS — concrete adoption cases rather than abstraction.

Closing #

The K8s Advanced series of 6 posts is wrapped up. This post organized in one cycle the GitOps model placing the manifest source of truth in git — push vs pull, ArgoCD’s Application CRD and sync wave, Flux’s bundle of small controllers, directory structure, and the path to put secrets in git via Sealed Secrets / External Secrets. Looking at the series as a whole, if Basics 7 and Intermediate 7 built depth around manifests and their operations, Advanced 6 layered on policy, extension, observation, and synchronization one piece at a time. In the next track, K8s Practice — 6 posts — the full cycle from EKS cluster setup, app deployment skeleton, DB integration, CI/CD pipeline, and monitoring/alarming through to the operations checklist will be followed end to end.