Release and incidents

Important user-facing changes go out in stages. Define success and rollback criteria before launch. Compliance and Legal join readiness checks. A surface that needs sign-off stays behind a flag until it is approved.

Deploy is not release

Deploy the code without exposing it, then increase exposure separately. Deploy puts the artifact in production. The feature flag controls who sees it. Document a kill switch for every user-facing change.

Progressive exposure

Set numeric promotion and abort criteria before launch. Tie them to SLOs, error rate, p99 latency, and order-reject rate. When automated analysis is available, connect it to rollback instead of having a person watch a dashboard. Choose a strategy for each surface:

• Rolling update, the default for stateless GKE Deployments.
• Blue-green, for schema-sensitive or session-heavy services.
• Canary with automated analysis, for high-blast-radius trading services, where a bad deploy can affect customer money.

FlaggerFlaggerTrialFlux-native progressive delivery. Flagger shifts traffic in steps and reads SLO metrics to promote or abort a canary. runs the canary in the Flux ecosystem. It shifts traffic in steps and uses the same SLO metrics to promote or abort. The Argo RolloutsArgo RolloutsAssessAn illustrative canary pattern for high-blast-radius services, not a mandated tool. Flagger is the Flux-native choice. manifest below shows an example alternative. It is not a required tool today:

# platform-enforced analysis, not a human eyeballing a graph
strategy:
  canary:
    steps:
      - setWeight: 5
      - pause: { duration: 5m }
      - analysis:
          templates:
            - templateName: error-rate-slo   # a failed analysis aborts the
                                              # rollout; configure abort and
                                              # scale-down behavior explicitly
      - setWeight: 25
      - pause: { duration: 10m }
      - setWeight: 50
      - pause: { duration: 10m }

flowchart LR
D[Deploy dark] --> C[Canary 5%]
C --> Ck{SLOs healthy?}
Ck -- yes --> R[Ramp 25, 50, 100]
Ck -- no --> Ab[Abort and roll back]
R --> Full[Full exposure]

Rollback

Every surface has a clear, practiced rollback procedure. Do not discover it during an incident. Rollback uses GitOps first: revert the change in git, and Flux returns the cluster to its previous state. Immutable image digests make the revert predictable (see build once, promote). Database migrations must remain backward compatible, so a rollback does not leave the schema unusable. A backward-compatible migration makes rollback safe. A forward-only migration does not.

If reconciliation is too slow during a live incident, use an imperative undo as the break-glass path. It is not the normal path. Reconcile the git state afterward because Flux will undo an unrecorded manual change on the next sync.

# break-glass only; the durable rollback is the git revert above
kubectl rollout undo deployment/order-gateway -n trading
kubectl rollout status deployment/order-gateway -n trading --timeout=120s

Readiness review

Before each significant release, a production-readiness review checks that observability, alerts, and the runbook exist, the load test passed, and the rollback was practiced. This extends the merge-time quality gates to release time.

Incidents

Follow the department process for incidents: page through the escalation policy, set severity early, communicate in the open channel, and write the retro. The Incident Management page is the authority for severity levels, response SLAs, and escalation. This section summarizes its process.

• Declare severity early. User impact determines severity. For example, lost order entry or fills is the top severity. Degraded market data or an auth outage is the next level. The canonical matrix and response times are in Incident Management.
• Name roles. Incident Commander, Communications Lead, Scribe, and Subject Matter Expert. The IC coordinates and does not fix the problem.
• One source of truth. The incident channel and status page provide updates at a set cadence. The Communications Lead owns Compliance, Legal, and customer messages.

Postmortems

P0, P1, and P2 incidents get a blameless postmortem using a fixed template. Track every action item to completion with an owner and due date.

## Impact            (users, $, duration, SLO burn)
## Timeline          (detect -> mitigate -> resolve, in UTC)
## Root cause        (5 Whys)
## Contributing factors
## What went well / what didn't
## Action items

| Owner | Due | Ticket |
|---|---|---|
|  |  |  |

Use incidents to improve the standard. When an action item finds a gap, update a runbook, an alert, or this page. SREs handle domain-specific PagerDuty alerts with agentic runbook skills that perform the investigation and remediation steps.

References

• Postmortem culture, blameless retros.
• Service level objectives, error budgets that gate releases.
• Argo Rollouts, canary and Kubernetes rollback, rollout and undo mechanics.
• Incident severity levels, a reference severity model.