CI/CD Pipelines

Overview

A Kubernetes CI/CD pipeline has two distinct jobs:

CI (Continuous Integration): build the image, run tests, scan for vulnerabilities, sign the artifact
CD (Continuous Delivery): update the deployment manifest in Git; let Argo CD / Flux sync the rest

The split is deliberate. CI runs in response to code changes (push to branch). CD runs in response to config changes (merge to main). Keeping them separate means the cluster is always in sync with Git, not with whatever the last CI run decided to push.

Code push
  │
  ▼
CI Pipeline (GitHub Actions / Tekton)
  ├─ lint + unit tests
  ├─ build image (Buildkit / Kaniko)
  ├─ scan image (Trivy)
  ├─ sign image (cosign)
  ├─ push image:sha-<git-sha> to registry
  └─ open PR: update image tag in config repo
                │
                ▼ (merged by human or auto-merge if tests pass)
           GitOps Repo
                │
                ▼
           Argo CD / Flux detects diff → applies to cluster

GitHub Actions — Standard Pipeline

Build, Scan, Sign, Push

# .github/workflows/ci.yaml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  ci:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write        # required for keyless cosign signing

    steps:
    - uses: actions/checkout@v4

    - name: Set up Go
      uses: actions/setup-go@v5
      with:
        go-version: '1.23'
        cache: true

    - name: Run tests
      run: go test ./... -race -coverprofile=coverage.out

    - name: Upload coverage
      uses: codecov/codecov-action@v4
      with:
        file: coverage.out

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v3

    - name: Log in to GHCR
      if: github.event_name != 'pull_request'
      uses: docker/login-action@v3
      with:
        registry: ${{ env.REGISTRY }}
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}

    - name: Extract metadata
      id: meta
      uses: docker/metadata-action@v5
      with:
        images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
        tags: |
          type=sha,format=long,prefix=sha-
          type=ref,event=branch
          type=semver,pattern={{version}}

    - name: Build and push
      id: build
      uses: docker/build-push-action@v6
      with:
        context: .
        push: ${{ github.event_name != 'pull_request' }}
        tags: ${{ steps.meta.outputs.tags }}
        labels: ${{ steps.meta.outputs.labels }}
        cache-from: type=gha
        cache-to: type=gha,mode=max
        provenance: true
        sbom: true

    - name: Scan image with Trivy
      uses: aquasecurity/trivy-action@master
      with:
        image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:sha-${{ github.sha }}
        format: sarif
        output: trivy-results.sarif
        severity: CRITICAL,HIGH
        exit-code: '1'         # fail CI on CRITICAL or HIGH CVEs

    - name: Upload Trivy results to GitHub Security
      uses: github/codeql-action/upload-sarif@v3
      if: always()
      with:
        sarif_file: trivy-results.sarif

    - name: Install cosign
      if: github.event_name != 'pull_request'
      uses: sigstore/cosign-installer@v3

    - name: Sign image (keyless via Sigstore)
      if: github.event_name != 'pull_request'
      run: |
        cosign sign --yes \
          ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }}

    - name: Attest SBOM
      if: github.event_name != 'pull_request'
      run: |
        cosign attest --yes \
          --predicate <(cosign download sbom \
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }}) \
          --type spdxjson \
          ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }}

Config Repo Update Job (GitOps PR)

  update-config:
    needs: ci
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
    - name: Checkout config repo
      uses: actions/checkout@v4
      with:
        repository: acme/k8s-config
        token: ${{ secrets.CONFIG_REPO_PAT }}
        path: config

    - name: Update image tag
      run: |
        cd config
        # Update the image tag in the staging values file
        sed -i "s|tag: sha-.*|tag: sha-${{ github.sha }}|" \
          services/payments-api/staging/values.yaml

    - name: Create Pull Request
      uses: peter-evans/create-pull-request@v6
      with:
        path: config
        token: ${{ secrets.CONFIG_REPO_PAT }}
        commit-message: "chore(payments-api): bump image to sha-${{ github.sha }}"
        title: "chore(payments-api): bump staging image to sha-${{ github.sha }}"
        body: |
          Automated image bump from [CI run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}).

          **Image:** `${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:sha-${{ github.sha }}`
          **Commit:** ${{ github.sha }}
          **Author:** ${{ github.actor }}
        branch: "bump/payments-api-${{ github.sha }}"
        base: main
        delete-branch: true

Reusable Workflow — Shared CI Template

For multiple services sharing the same pipeline pattern, extract into a reusable workflow:

# .github/workflows/reusable-build.yaml (in platform repo)
name: Reusable Build

on:
  workflow_call:
    inputs:
      service-name:
        required: true
        type: string
      go-version:
        required: false
        type: string
        default: '1.23'
      dockerfile:
        required: false
        type: string
        default: Dockerfile
    outputs:
      image-digest:
        description: "Pushed image digest"
        value: ${{ jobs.build.outputs.digest }}
    secrets:
      CONFIG_REPO_PAT:
        required: true

jobs:
  build:
    uses: acme/platform/.github/workflows/reusable-build.yaml@main
    # ... full build steps referencing inputs.*

# In each service repo: .github/workflows/ci.yaml
name: CI
on:
  push:
    branches: [main]

jobs:
  build:
    uses: acme/platform/.github/workflows/reusable-build.yaml@main
    with:
      service-name: payments-api
      go-version: '1.23'
    secrets:
      CONFIG_REPO_PAT: ${{ secrets.CONFIG_REPO_PAT }}

Tekton — Kubernetes-Native CI

Tekton runs pipelines as Kubernetes resources (Tasks, Pipelines, PipelineRuns). It is more complex than GitHub Actions but runs entirely in-cluster, supports custom resource types, and integrates natively with Argo CD.

Core Tekton Resources

Task          — a series of Steps (container commands), reusable
Pipeline      — ordered sequence of Tasks with params and workspaces
PipelineRun   — an instance of a Pipeline with concrete params
Trigger       — listens for webhook events, creates PipelineRuns

Task: Build and Push with Buildah

apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: buildah-build-push
  namespace: tekton-pipelines
spec:
  params:
  - name: IMAGE
    description: Destination image reference
  - name: DOCKERFILE
    default: ./Dockerfile
  - name: CONTEXT
    default: .
  workspaces:
  - name: source
  - name: dockerconfig
    description: Registry credentials secret
  steps:
  - name: build
    image: quay.io/buildah/stable:latest
    securityContext:
      privileged: true
    script: |
      buildah bud \
        --format=oci \
        --tls-verify=true \
        -f $(params.DOCKERFILE) \
        -t $(params.IMAGE) \
        $(params.CONTEXT)
  - name: push
    image: quay.io/buildah/stable:latest
    securityContext:
      privileged: true
    script: |
      buildah push \
        --tls-verify=true \
        --digestfile=/workspace/source/image-digest \
        $(params.IMAGE)
  results:
  - name: IMAGE_DIGEST
    description: Digest of the built image
    value: $(steps.push.results.IMAGE_DIGEST)

Task: Trivy Scan

apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: trivy-scan
  namespace: tekton-pipelines
spec:
  params:
  - name: IMAGE
  - name: SEVERITY
    default: CRITICAL,HIGH
  steps:
  - name: scan
    image: aquasec/trivy:latest
    script: |
      trivy image \
        --exit-code 1 \
        --severity $(params.SEVERITY) \
        --format table \
        $(params.IMAGE)

Pipeline: Full Build Pipeline

apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: build-pipeline
  namespace: tekton-pipelines
spec:
  params:
  - name: repo-url
  - name: revision
  - name: image
  workspaces:
  - name: source
  - name: dockerconfig

  tasks:
  - name: clone
    taskRef:
      resolver: hub
      params:
      - name: name
        value: git-clone
    workspaces:
    - name: output
      workspace: source
    params:
    - name: url
      value: $(params.repo-url)
    - name: revision
      value: $(params.revision)

  - name: test
    runAfter: [clone]
    taskRef:
      name: golang-test
    workspaces:
    - name: source
      workspace: source

  - name: build-push
    runAfter: [test]
    taskRef:
      name: buildah-build-push
    workspaces:
    - name: source
      workspace: source
    - name: dockerconfig
      workspace: dockerconfig
    params:
    - name: IMAGE
      value: $(params.image):$(tasks.clone.results.commit)

  - name: scan
    runAfter: [build-push]
    taskRef:
      name: trivy-scan
    params:
    - name: IMAGE
      value: $(params.image):$(tasks.clone.results.commit)

  - name: sign
    runAfter: [scan]
    taskRef:
      name: cosign-sign
    params:
    - name: IMAGE
      value: $(params.image)@$(tasks.build-push.results.IMAGE_DIGEST)

Kaniko — In-Cluster Image Builds

Kaniko builds container images inside a Kubernetes pod without requiring a Docker daemon. Useful when your CI must run inside the cluster (no privileged daemonsets).

apiVersion: batch/v1
kind: Job
metadata:
  name: kaniko-build
  namespace: ci
spec:
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: kaniko-sa   # needs push permissions via IRSA / Workload Identity
      containers:
      - name: kaniko
        image: gcr.io/kaniko-project/executor:latest
        args:
        - "--context=git://github.com/acme/payments-api#refs/heads/main"
        - "--dockerfile=Dockerfile"
        - "--destination=ghcr.io/acme/payments-api:$(GIT_SHA)"
        - "--cache=true"
        - "--cache-repo=ghcr.io/acme/payments-api/cache"
        - "--snapshot-mode=redo"
        env:
        - name: GIT_SHA
          value: "abc123"
        volumeMounts:
        - name: docker-config
          mountPath: /kaniko/.docker
      volumes:
      - name: docker-config
        secret:
          secretName: registry-credentials
          items:
          - key: .dockerconfigjson
            path: config.json

Image Tag Strategy

Strategy	Format	Pros	Cons
Git SHA (recommended)	`sha-a1b2c3d`	Immutable, traceable to commit	Not human-friendly
Semantic version	`v1.4.2`	Human-readable, sortable	Requires manual or automated version bump
Branch + SHA	`main-a1b2c3d`	Know which branch it came from	Longer
`latest` (anti-pattern)	`latest`	Convenient	Non-deterministic, breaks rollbacks

# Always pin by digest in production manifests — even more immutable than SHA tag
# A tag can be reassigned; a digest cannot
image: ghcr.io/acme/payments-api@sha256:abc123...

# Extract digest after push
DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' ghcr.io/acme/payments-api:sha-abc123)

Pipeline Security Best Practices

# GitHub Actions: pin actions by SHA, not tag
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683  # v4.2.2

# Use least-privilege GITHUB_TOKEN permissions
permissions:
  contents: read
  packages: write
  id-token: write      # ONLY if using OIDC (cosign, cloud auth)
  security-events: write  # ONLY if uploading SARIF

# Never use pull_request_target with secrets unless you understand the risk
# Use environment protection rules for production deployments

# Verify pipeline identity (GitHub Actions OIDC → cosign)
# The signature embeds the workflow, repo, and SHA — verifiable offline
cosign verify \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
  --certificate-identity-regexp=https://github.com/acme/payments-api/.github/workflows/ \
  ghcr.io/acme/payments-api@sha256:abc123

Pipeline Observability

# GitHub Actions: view run summary
gh run list --repo acme/payments-api --limit 10
gh run view <run-id>
gh run watch <run-id>

# Tekton: view PipelineRuns
kubectl get pipelineruns -n tekton-pipelines
tkn pipelinerun logs <name> -f -n tekton-pipelines
tkn pipelinerun describe <name> -n tekton-pipelines

# Pipeline duration tracking (PromQL — if Tekton metrics enabled)
histogram_quantile(0.95,
  rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket[1h])
)

Local Development — inner loop before CI runs
Helm — packaging for CD deployment
Progressive Delivery — canary deployments in the CD pipeline
04-security-hardening — supply chain security, image signing