Observability

The table names each tool. This page explains how to use observability well.

Golden signals

Every service measures the four golden signals: latency, traffic, errors, and saturation. Use RED (Rate, Errors, Duration) for request-driven services. Use USE (Utilization, Saturation, Errors) for resources and queues. This gives dashboards a shared baseline. JVM services also export runtime metrics to help diagnose incidents on the trading backend: GC pauses, heap and off-heap memory, and thread count.

SLOs and error budgets

Each service defines SLIs, such as availability and p99 latency, a target, and the error budget that target allows. Page on the budget burn rate, not on raw CPU or a simple threshold. This reduces alert fatigue. Fast burn rates trigger pages. Slow burn rates create tickets.

# Alert on the SLO's burn rate, referencing the SLO as its own resource.
resource "google_monitoring_alert_policy" "orders_slo_fast_burn" {
  display_name = "orders availability - fast burn"
  combiner     = "OR"
  # fast-burn window; pair with a slower window (e.g. 6h) for the full
  # multi-window multi-burn-rate policy
  conditions {
    display_name = "burn rate > 14.4x over 1h"
    condition_threshold {
      filter          = "select_slo_burn_rate(\"${google_monitoring_slo.orders_availability.name}\", \"3600s\")"
      comparison      = "COMPARISON_GT"
      threshold_value = 14.4   # ~2% of a 30-day budget spent in 1h
      duration        = "0s"
    }
  }
  notification_channels = [var.pagerduty_channel]
  documentation { content = "Runbook: https://.../orders-availability" }
}

Tracing

Pass W3C Trace Context (traceparent) through the full request path. This connects one request across the Scala backend, Node, Python, and the gateways. Use a parent-based ratio sampler for head sampling. At the collector, keep error and slow traces with tail sampling. Head sampling alone cannot keep traces that it did not sample. Auto-instrument the JVM with the OpenTelemetry Java agent. Add manual spans for domain operations, using only low-cardinality attributes.

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-javaagent:/otel/opentelemetry-javaagent.jar"
  - name: OTEL_SERVICE_NAME
    value: "orders-service"
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://otel-collector.observability:4317"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "grpc"           # 4317 is gRPC; use 4318 for http/protobuf
  - name: OTEL_TRACES_SAMPLER
    value: "parentbased_traceidratio"
  - name: OTEL_TRACES_SAMPLER_ARG
    value: "0.1"

val tracer = GlobalOpenTelemetry.getTracer("orders")
val span = tracer.spanBuilder("placeOrder")
  .setAttribute("order.account_id", accountId) // stable, non-PII identifier
  .setAttribute("order.symbol", symbol)
  .startSpan()
val scope = span.makeCurrent()
try placeOrder(req)
catch {
  case e: Throwable =>
    span.recordException(e)
    span.setStatus(StatusCode.ERROR)
    throw e
}
finally { scope.close(); span.end() }

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

span = trace.get_tracer("orders").start_span("placeOrder")
span.set_attribute("order.account_id", account_id)  # stable, non-PII identifier
span.set_attribute("order.symbol", symbol)
try:
    place_order(req)
except Exception as e:
    span.record_exception(e)
    span.set_status(Status(StatusCode.ERROR))
    raise
finally:
    span.end()

import { trace, SpanStatusCode } from "@opentelemetry/api";

const span = trace.getTracer("orders").startSpan("placeOrder");
span.setAttribute("order.account_id", accountId); // stable, non-PII identifier
span.setAttribute("order.symbol", symbol);
try {
  placeOrder(req);
} catch (e) {
  span.recordException(e as Error);
  span.setStatus({ code: SpanStatusCode.ERROR });
  throw e;
} finally {
  span.end();
}

flowchart LR
Cl[Client] --> Gw[Gateway] --> Svc[Services]
Gw -. traceparent .-> Svc
Svc --> Col[OTel collector] --> CT[Cloud Trace]

Structured logging

Logs use JSON with a fixed schema: timestamp, severity, service, trace_id/span_id, and a stable event field. This schema is the versioned interface named in the table. Connect logs to traces with the special Cloud Logging fields. Use ERROR only for failures that need action, so paid ingestion remains useful.

{
  "timestamp": "2026-08-22T14:03:11.482Z",
  "severity": "ERROR",
  "service": "orders-service",
  "event": "order.rejected",
  "logging.googleapis.com/trace": "projects/PROJECT_ID/traces/abc123",
  "logging.googleapis.com/spanId": "def456",
  "account_id": "91827",
  "reason": "insufficient_margin"
}

Control costs with routing sinks and exclusions for each environment. Keep metric label cardinality bounded. Never use user_id, full URLs, or other unbounded values as labels. See Cloud Logging structured logging.

Alerts and dashboards

• Alerts are code. They are scoped to an environment, sent to a team channel with an owner, and limited in frequency.
• Alerts focus on symptoms and user impact first. Classify each one as a page, ticket, or dashboard-only alert.
• Every paging alert links to a runbook, and the alert policy links to it. See release and incidents.
• A service builds and maintains dashboards, alerts, and runbooks for each environment. These are defined as code. Each dashboard covers the golden signals and links directly to traces and logs.

Governance

• Report only from staging, UAT, and production. Make the environment a main filter.
• Keep the symbols and source maps needed to read stack traces. An unreadable trace is a build bug.
• Identify users with a unique identifier. Assume that anything in an exception message can leak. Keep all PII out of logs and traces. This is a security control.

References

• OpenTelemetry documentation, SDKs, the Java agent, and samplers.
• Cloud Monitoring alerting, alert policies and log-based metrics.
• In-house: tradovate/servers wires the OTel SDK on the JVM; NT-NinjaTrader/gcp-infrastructure-base has a reusable alert-policy module; NT-NinjaTrader/innolab-ada ADR 0010 documents the sampling strategy.