1. GitOps Principles

GitOps is an operational model where the desired state of a system is declared in Git, and an automated agent continuously reconciles the live state to match the declared state. The four core principles (OpenGitOps spec):

GitOps flow:

Developer                  Git Repository            Kubernetes Cluster
    │                            │                          │
    │── git commit + push ──────▶│                          │
    │                            │◀── poll (every 3min) ────│
    │                            │       (Argo CD / Flux)   │
    │                            │──── diff detected ──────▶│
    │                            │                          │── apply/sync
    │                            │                          │── reconcile
    │                            │◀────── Health: Healthy ──│
    │                            │                          │
    │                            │  [manual kubectl apply]  │
    │                            │                          │── drift detected
    │                            │◀────── selfHeal ─────────│── revert to Git state

Push-based vs Pull-based CD

Dimension	Push-based (traditional CD)	Pull-based (GitOps)
Who initiates deployment	CI pipeline pushes to cluster	Agent in cluster pulls from Git
Credentials location	CI system holds kubeconfig/service account	Only Git credentials needed outside cluster
Attack surface	CI compromise = cluster compromise	Cluster credentials never leave cluster
Drift detection	None (pipeline only runs on trigger)	Continuous — every reconciliation loop
Auditability	CI logs (may be transient)	Git history (permanent, signed)
Multi-cluster	Complex (credential management per cluster)	Single agent per cluster, single Git repo

2. Argo CD Architecture

┌──────────────────────────────────────────────────────┐
│                    Argo CD                            │
│                                                      │
│  ┌──────────────┐  ┌───────────────┐                 │
│  │  API Server  │  │   Repo Server │                 │
│  │  (argocd-    │  │  (git clone,  │                 │
│  │   server)    │  │   template,   │                 │
│  │  REST + gRPC │  │   diff cache) │                 │
│  └──────┬───────┘  └───────┬───────┘                 │
│         │                  │                         │
│  ┌──────▼──────────────────▼───────┐                 │
│  │        Application Controller   │                 │
│  │  (reconcile loop, sync,         │                 │
│  │   health assessment,            │                 │
│  │   hook execution)               │                 │
│  └──────────────────────────────── ┘                 │
│                                                      │
│  ┌──────────────┐  ┌───────────────┐                 │
│  │  Redis       │  │  Dex (SSO)    │                 │
│  │  (cache,     │  │  (OIDC auth)  │                 │
│  │   sessions)  │  │               │                 │
│  └──────────────┘  └───────────────┘                 │
└──────────────────────────────────────────────────────┘

Storage: Application CRDs in Kubernetes etcd (not external DB)

Argo CD Install (Production)

kubectl create namespace argocd

# HA install (recommended for production — 3 app controller replicas)
kubectl apply -n argocd \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/ha/install.yaml

# Or via Helm (recommended for managing Argo CD config-as-code)
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

helm upgrade --install argocd argo/argo-cd \
  --namespace argocd \
  --create-namespace \
  --values argocd-values.yaml

# argocd-values.yaml (production)
global:
  domain: argocd.internal.example.com

configs:
  cm:
    # Resource tracking: label (default) or annotation
    application.resourceTrackingMethod: annotation

    # Timeout for sync operations
    timeout.reconciliation: 180s

    # OIDC SSO via Dex
    url: https://argocd.internal.example.com
    oidc.config: |
      name: Okta
      issuer: https://myorg.okta.com/oauth2/default
      clientId: $oidc.okta.clientId
      clientSecret: $oidc.okta.clientSecret
      requestedScopes: [openid, profile, email, groups]
      requestedIDTokenClaims:
        groups:
          essential: true

  rbac:
    policy.csv: |
      g, platform-admins, role:admin
      g, developers, role:readonly
      p, role:team-deployer, applications, sync, */*, allow
      p, role:team-deployer, applications, get, */*, allow
    policy.default: role:readonly
    scopes: '[groups]'

  params:
    server.insecure: false
    controller.diff.server.side: "true"    # use server-side diff

repoServer:
  replicas: 2
  resources:
    requests: {cpu: 250m, memory: 256Mi}
    limits: {memory: 1Gi}
  # Enable Helm plugins / custom tools
  volumes:
    - name: custom-tools
      emptyDir: {}
  initContainers:
    - name: download-tools
      image: alpine:3.18
      command: [sh, -c]
      args:
        - |
          # Install helm-secrets plugin for SOPS decryption
          helm plugin install https://github.com/jkroepke/helm-secrets

applicationSet:
  replicas: 2

server:
  replicas: 2
  service:
    type: ClusterIP
  ingress:
    enabled: true
    ingressClassName: nginx
    hostname: argocd.internal.example.com
    tls: true
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod

controller:
  replicas: 2   # HA: sharded app controller
  env:
    - name: ARGOCD_CONTROLLER_REPLICAS
      value: "2"

3. Argo CD Applications

Application CRD — Full Reference

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: order-service
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io  # cascade delete managed resources on app deletion
  annotations:
    argocd.argoproj.io/sync-wave: "5"          # sync ordering
spec:
  project: team-a                              # Argo CD AppProject for RBAC

  source:
    repoURL: https://github.com/myorg/services
    targetRevision: HEAD                       # or specific tag/branch/SHA
    path: order-service/helm

    # Helm source
    helm:
      releaseName: order-service
      valueFiles:
        - values.yaml
        - values-production.yaml
      values: |
        image:
          tag: "1.4.2"
      ignoreMissingValueFiles: false
      skipCrds: false

  # Alternative: Kustomize source
  # source:
  #   path: order-service/kustomize/overlays/production
  #   kustomize:
  #     namePrefix: prod-
  #     images:
  #       - myregistry/order-service:1.4.2

  destination:
    server: https://kubernetes.default.svc    # in-cluster
    # or: server: https://prod-us.example.com (registered external cluster)
    namespace: order-service

  syncPolicy:
    automated:
      prune: true        # delete resources removed from Git
      selfHeal: true     # revert manual kubectl changes
      allowEmpty: false  # never sync to an empty state
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true       # use SSA instead of kubectl apply
      - PrunePropagationPolicy=foreground
      - PruneLast=true             # delete resources after all others synced
      - RespectIgnoreDifferences=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

  # Ignore specific fields that change outside Git (e.g., HPA replicas)
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas                    # HPA manages this
    - group: ""
      kind: Secret
      managedFieldsManagers:
        - secrets-store-sync               # ESO manages secret data

  revisionHistoryLimit: 10

AppProject — Team Isolation

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-a
  namespace: argocd
spec:
  description: "Team A microservices"

  # Only allow syncing to specific clusters/namespaces
  destinations:
    - server: https://kubernetes.default.svc
      namespace: team-a-*    # wildcard namespace match
    - server: https://prod-eu.example.com
      namespace: team-a-*

  # Only allow sources from team's repo
  sourceRepos:
    - https://github.com/myorg/team-a-services
    - https://charts.helm.sh/stable          # approved Helm repos

  # Restrict which K8s resources can be managed
  clusterResourceWhitelist:
    - group: ""
      kind: Namespace
  namespaceResourceBlacklist:
    - group: ""
      kind: ResourceQuota              # only platform team can set quotas
    - group: networking.k8s.io
      kind: NetworkPolicy              # only platform team can create network policies

  # Orphaned resource monitoring (alert on resources not in any app)
  orphanedResources:
    warn: true

  roles:
    - name: team-a-deployer
      policies:
        - p, proj:team-a:team-a-deployer, applications, sync, team-a/*, allow
        - p, proj:team-a:team-a-deployer, applications, get, team-a/*, allow
      groups:
        - team-a-developers

4. App-of-Apps Pattern

The app-of-apps pattern uses a single root Application that manages other Applications. This allows the entire cluster state to be bootstrapped from a single kubectl apply and maintained declaratively in Git.

platform-gitops/
└── clusters/production/apps/         ← root app points here
    ├── cert-manager.yaml              ← each file is an Application CR
    ├── ingress-nginx.yaml
    ├── kube-prometheus-stack.yaml
    ├── external-secrets.yaml
    ├── gatekeeper.yaml
    ├── karpenter.yaml
    └── team-apps/
        ├── team-a-apps.yaml           ← nested app-of-apps for each team
        └── team-b-apps.yaml

# clusters/production/apps/cert-manager.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cert-manager
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-1"   # install before things that need certs
spec:
  project: platform
  source:
    repoURL: https://charts.jetstack.io
    chart: cert-manager
    targetRevision: v1.15.0
    helm:
      values: |
        installCRDs: true
        replicaCount: 2
        resources:
          requests: {cpu: 100m, memory: 64Mi}
          limits: {memory: 256Mi}
  destination:
    server: https://kubernetes.default.svc
    namespace: cert-manager
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

5. ApplicationSets

ApplicationSet is a controller that generates multiple Argo CD Application resources from a single template using generators. It enables fleet-wide deployments and per-team application management without manual Application YAML per environment.

List Generator — Explicit Multi-Cluster

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster: prod-us-east
            url: https://prod-us-east.example.com
            env: production
          - cluster: prod-eu-west
            url: https://prod-eu-west.example.com
            env: production
          - cluster: staging
            url: https://staging.example.com
            env: staging
  template:
    metadata:
      name: "guestbook-{{cluster}}"
    spec:
      project: platform
      source:
        repoURL: https://github.com/myorg/platform-gitops
        targetRevision: HEAD
        path: "apps/guestbook/overlays/{{env}}"
      destination:
        server: "{{url}}"
        namespace: guestbook
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Git Generator — Directory-based Multi-Env

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-addons
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/myorg/platform-gitops
        revision: HEAD
        directories:
          - path: "add-ons/*"           # one Application per directory
          - path: "add-ons/kustomize/*"
            exclude: true               # exclude subdirectories
  template:
    metadata:
      name: "addon-{{path.basename}}"
    spec:
      project: platform
      source:
        repoURL: https://github.com/myorg/platform-gitops
        targetRevision: HEAD
        path: "{{path}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{path.basename}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Matrix Generator — Env × App Cartesian Product

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: services-matrix
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          # Generator 1: list of clusters
          - clusters:
              selector:
                matchLabels:
                  platform.example.com/tier: workload
          # Generator 2: list of services from git
          - git:
              repoURL: https://github.com/myorg/services
              revision: HEAD
              files:
                - path: "services/*/config.json"  # reads JSON per service
  template:
    metadata:
      name: "{{name}}-{{values.service}}"
    spec:
      project: "{{values.team}}"
      source:
        repoURL: https://github.com/myorg/services
        targetRevision: HEAD
        path: "services/{{values.service}}/helm"
        helm:
          valueFiles:
            - values.yaml
            - "values-{{metadata.labels.environment}}.yaml"
      destination:
        server: "{{server}}"
        namespace: "{{values.service}}"

Pull Request Generator — Preview Environments

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: preview-environments
  namespace: argocd
spec:
  generators:
    - pullRequest:
        github:
          owner: myorg
          repo: order-service
          tokenRef:
            secretName: github-token
            key: token
          labels:
            - preview             # only PRs with this label
  template:
    metadata:
      name: "pr-{{number}}-order-service"
    spec:
      project: preview
      source:
        repoURL: https://github.com/myorg/order-service
        targetRevision: "{{head_sha}}"
        path: helm
        helm:
          values: |
            image:
              tag: "pr-{{number}}"
            ingress:
              host: "pr-{{number}}.preview.example.com"
      destination:
        server: https://kubernetes.default.svc
        namespace: "preview-pr-{{number}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

6. Sync Waves & Hooks

Sync Waves

Sync waves control the order in which resources are applied within a single sync operation. Resources in lower waves are applied first and must be healthy before higher waves begin.

# Resource annotation — wave number (default 0, range: any integer)
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "5"

# Common wave ordering pattern:
# Wave -3: Namespaces
# Wave -2: CRDs (cert-manager CRDs, Gatekeeper CRDs, etc.)
# Wave -1: Core infrastructure (cert-manager, external-dns, kube-state-metrics)
# Wave  0: Default (most resources)
# Wave  1: Applications that depend on infrastructure
# Wave  5: Ingress resources (after cert-manager and ingress controller)
# Wave 10: Smoke test jobs / verification

Resource Hooks

# PreSync hook — run before sync (e.g., database migrations)
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: myregistry/order-service:1.4.2
          command: ["/app/migrate", "up"]
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: url
      restartPolicy: Never
  backoffLimit: 3

# PostSync hook — run after all resources are Healthy (e.g., smoke test)
apiVersion: batch/v1
kind: Job
metadata:
  name: smoke-test
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded  # auto-cleanup on success
spec:
  template:
    spec:
      containers:
        - name: smoke-test
          image: curlimages/curl:8.4.0
          command:
            - sh
            - -c
            - |
              until curl -sf http://order-service.order-service/health; do
                echo "waiting..."; sleep 3
              done
              echo "smoke test passed"
      restartPolicy: Never

# Available hook types:
# PreSync     — before sync begins
# Sync        — during sync (parallel with resource apply)
# PostSync    — after all resources are Healthy
# SyncFail    — if sync fails (rollback notification, cleanup)
# PostDelete  — when application is deleted

# Hook delete policies:
# HookSucceeded      — delete after successful run (default for most)
# HookFailed         — delete after failed run
# BeforeHookCreation — delete old hook before creating new one (idempotent)

7. Argo CD RBAC & SSO

RBAC Policy Syntax

# argocd-rbac-cm ConfigMap policy.csv
# Format: p, , , ,