CI/CD

• Deploys run automatically from configuration. Configuration chooses the environment, not branching logic.
• Create a release branch only after QA, product acceptance, and formal change approval.
• Build once and promote that build. The build tested in staging must be the build that reaches production.
• Roll out in phases when the surface supports it instead of switching everyone at once. When it does not, such as with an installer or a store release, the flag provides staged exposure.
• Values added at build time must be in CI before the build. You cannot set them later at runtime.
• Know how to roll back before you deploy. A forward-only migration is not a rollback, and neither is a shipped binary.

Pipeline stages

The pipeline builds once, then runs the gates in parallel. This makes pipeline time equal to the slowest gate, not the total time of all gates. Every gate before deployment is defined once in quality gates and enforced here:

1. Build and publish the artifact once, and record its immutable digest.
2. Run these checks in parallel against that one digest:
   • Unit and integration tests.
   • Static analysis, lint, format.
   • Security scan: SAST, dependencies, secrets, container image.
3. Attest only after every gate passes: sign the digest and record build provenance. One failure blocks attestation, and nothing deploys.
4. Deploy the attested digest to dev, staging, and production.
5. Run smoke tests and verify.

flowchart LR
B[Build once] --> T[Tests]
B --> A[Static analysis]
B --> S[Security scan]
T --> G{All gates green?}
A --> G
S --> G
G -->|no| X[Fail: nothing deploys]
G -->|yes| At[Attest: sign + SLSA provenance]
At --> Dev[Deploy dev] --> Stg[Deploy staging] --> Prod[Deploy prod]
B -. same sha256 digest .-> Prod

Build once, promote by digest

“Build once and promote” works only when the artifact cannot change and is identified by its content, not by a mutable tag such as latest. Build the image once, record its sha256 digest, and promote that exact digest through dev, staging, and production. Staging and production then run the same bytes.

# build job: publish once, capture the digest
- id: build
  run: |
    set -euo pipefail
    docker buildx build --push -t "$REPO:$GITHUB_SHA" \
      --metadata-file /tmp/meta.json .
    digest="$(jq -re '."containerimage.digest"' /tmp/meta.json)"
    echo "digest=$digest" >> "$GITHUB_OUTPUT"

# deploy job: reference the digest, so staging == prod byte-for-byte
- run: kubectl set image deploy/api "api=$REPO@${{ needs.build.outputs.digest }}"

In GKE manifests, pin container images by digest. Never use a floating tag.

Deployment strategies

FluxFluxCDAdoptGitOps delivery for Kubernetes. Flux reconciles the cluster to the desired state declared in git, so a merge is the deploy.Used inNT-NinjaTrader/cloudNT-NinjaTrader/ninja-web-trader delivers the software. It matches the cluster to the desired state in git through Kustomize overlays. A merge to the environment path is the deployment, not an imperative kubectl apply. Name the strategy for each surface:

• Rolling update, the default for stateless GKE Deployments.
• Canary / progressive delivery, for services with a high blast radius; FlaggerFlaggerTrialFlux-native progressive delivery. Flagger shifts traffic in steps and reads SLO metrics to promote or abort a canary. shifts traffic in steps, watches the signals, then increases traffic or aborts.
• Flag-gated exposure, when the surface cannot be switched server-side, such as installers and store releases; the feature flag controls the rollout.

Database migrations

Schema changes use expand-contract (parallel change). This keeps every migration backward compatible and separately reversible. A deployment that can roll back only through a forward-only migration is not allowed.

-- Release N (expand): add nullable column, backfill, dual-write in the app
ALTER TABLE orders ADD COLUMN status_v2 VARCHAR(32) NULL;

-- Release N+1 (contract): drop the old column, only after N is fully
-- deployed and verified. DROP is destructive and not itself reversible,
-- so keep a backup and never contract in the same release that expands.
ALTER TABLE orders DROP COLUMN status_old;

See ParallelChange for the full pattern. Expand, migrate, and contract are separate releases.

Secrets and auth in CI

Do not put plaintext secrets in workflow YAML or logs. Use OIDC / Workload Identity Federation to authenticate to GCP, not long-lived service-account keys. Pin every GitHub Action to a full commit SHA, and use only a release built on node24.

permissions:
  contents: read
  id-token: write   # required for OIDC
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - uses: google-github-actions/auth@7c6bc770dae815cd3e89ee6cdf493a5fab2cc093 # v3.0.0
        with:
          workload_identity_provider: ${{ vars.WIF_PROVIDER }}
          service_account: ${{ vars.DEPLOY_SA }}

See security hardening for GitHub Actions and the supply-chain requirements in quality gates.

Infrastructure changes

Terraform changes run through AtlantisAtlantisAdoptTerraform changes plan on a PR. A human reviews the plan, and apply runs only after explicit approval. This is the infrastructure form of four-eyes.Used inNT-NinjaTrader/cloud. The plan review is the infrastructure version of four-eyes. A re-plan on a clean PR also shows drift because it displays a diff when live state no longer matches git.

Rollback and verification

A deployment is not complete until it is verified. Run smoke tests and health and readiness checks after every deployment. Roll back when verification fails. Do this automatically when the change is safe to revert automatically, such as a stateless service. Otherwise, use a documented and practiced manual process for stateful changes, migrations, and infrastructure. Document the rollback for each surface: redeploy the previous digest, or re-apply the previous state with atlantis plan and apply for infrastructure. Rollback and post-deployment verification connect to release and incidents.

Track delivery health with the DORA metrics: deployment frequency, lead time for change, change-failure rate, and mean time to recovery.

Keep build, release, and run separate. See The Twelve-Factor App.