Skip to main content

Telemetry (OpenTelemetry + Prometheus)

Status: baseline implementation (GAR-384). Instrumentation is additive — the gateway runs fine with telemetry off.

TL;DR​

  1. Start the observability stack: 
    docker compose -f docker-compose.yml -f ops/compose.otel.yml up -d
  2. Start the gateway with telemetry enabled: 
    GARRAIA_OTEL_ENABLED=true \
    GARRAIA_METRICS_ENABLED=true \
    cargo run -p garraia-gateway

    Production: see §1.1 TLS for OTLP exports before pointing at a non-loopback collector — tonic defaults to cleartext.

  3. Open Jaeger at http://localhost:16686, select service garraia-gateway, and click "Find Traces" to see the latest trace.

1. Configuration​

Env vars​

VariableDefaultPurpose
GARRAIA_OTEL_ENABLEDfalseEnable OTLP trace export
GARRAIA_OTEL_EXPORTER_OTLP_ENDPOINThttp://localhost:4317gRPC endpoint for the OTLP collector
GARRAIA_OTEL_SERVICE_NAMEgarraia-gatewayService name shown in Jaeger
GARRAIA_OTEL_SAMPLE_RATIO1.0Fraction of traces to sample (0.0–1.0)
GARRAIA_METRICS_ENABLEDfalseEnable Prometheus metrics scrape endpoint
GARRAIA_METRICS_BIND127.0.0.1:9464Socket address for the dedicated /metrics listener
GARRAIA_METRICS_TOKEN(unset)Bearer token required on /metrics (plan 0024 / GAR-412)
GARRAIA_METRICS_ALLOW(unset)Comma-separated CIDR allowlist for /metrics peers (plan 0024 / GAR-412)

.env example​

# Telemetry — enable for local dev / staging; off by default in prod
GARRAIA_OTEL_ENABLED=true
GARRAIA_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
GARRAIA_OTEL_SERVICE_NAME=garraia-gateway
GARRAIA_OTEL_SAMPLE_RATIO=1.0

GARRAIA_METRICS_ENABLED=true
GARRAIA_METRICS_BIND=127.0.0.1:9464

Disabled by default​

All variables default to a safe, zero-overhead state. A production binary with no telemetry env vars set emits no spans, makes no network connections to a collector, and exposes no metrics endpoint. Opt in explicitly per environment.

1.1 TLS for OTLP exports (production)​

⚠️ Plaintext by default. opentelemetry-otlp uses tonic as the gRPC transport, which defaults to HTTP/2 cleartext (http://). Any span attribute — including http.target, http.url, request IDs, or custom fields you attach — traverses the wire unencrypted unless you configure TLS explicitly.

Production recommendation:

# Force TLS by using the https:// scheme. tonic picks up the scheme
# automatically and negotiates TLS against the collector's cert.
GARRAIA_OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.example.com:4317

Your collector (Jaeger-as-OTLP, OpenTelemetry Collector, Grafana Tempo, etc.) must be configured to terminate TLS on that port. See the collector's docs for serving TLS — typically a tls: block under the OTLP receiver with cert_file / key_file paths or an ACME integration.

Development / loopback: http://localhost:4317 is fine when the collector runs on the same host — the traffic never leaves the local network interface.

Why no auto-TLS? opentelemetry-otlp defers to the scheme declared in the endpoint URL — http:// negotiates cleartext, https:// negotiates TLS against the collector's cert. A "TLS by default unless explicitly disabled" knob does not exist in the current opentelemetry-otlp API (0.14.x series, built on tonic 0.12) and would conflict with service-mesh sidecar deploys (where the sidecar terminates TLS and forwards cleartext on loopback). We rely on this docs warning to keep the matrix simple. Revisit when opentelemetry-otlp exposes a first-class TLS opt-in independent of the endpoint scheme (tracked in GAR-411 M2 follow-up notes).

2. Running locally with Jaeger + Prometheus + Grafana​

# Bring up Jaeger, Prometheus, and Grafana alongside the main compose services
docker compose -f docker-compose.yml -f ops/compose.otel.yml up -d
UIURLCredentials
Jaegerhttp://localhost:16686none
Prometheushttp://localhost:9090none
Grafanahttp://localhost:3000admin / garraia

Sanity check:

# Send a request to the gateway
curl -s http://localhost:3888/health

# Then open Jaeger, select service "garraia-gateway", and find the trace for
# the /health request — it should appear within a few seconds.

3. Reading a trace​

Finding your trace: Open Jaeger at http://localhost:16686, select garraia-gateway from the Service dropdown, and click "Find Traces". The most recent request appears at the top.

Span hierarchy:

http.request  [TraceLayer — root span, covers the full HTTP round trip]
  └─ AgentRuntime::process_message   [#[tracing::instrument] in garraia-agents]
       └─ SessionStore::append_message  [#[tracing::instrument] in garraia-db]
       └─ SessionStore::load_recent_messages

The root span comes from garraia_telemetry::http_trace_layer(). Nested spans are created automatically by #[tracing::instrument] on AgentRuntime::process_message*, SessionStore::append_message*, and SessionStore::load_recent_messages.

Correlating logs with a trace:

Every response includes an x-request-id header (added by garraia_telemetry::request_id_layer()). This ID appears in structured logs alongside the trace_id. Search Jaeger by trace ID or grep logs by request ID to correlate the two.

4. Adding a span to your code​

Pattern A: function-scoped span​

Use #[tracing::instrument] for async functions. The span is correctly attached across .await points.

#[tracing::instrument(skip(self), fields(provider = %provider_name), err)]
async fn do_thing(&self, provider_name: &str) -> Result<()> {
    // Every await point inside here is covered by this span.
    Ok(())
}

Pattern B: block-scoped span​

Use a manual span when you need to instrument a specific block rather than a whole function.

let span = tracing::info_span!("my.operation", key = %value);
let _enter = span.enter();
// Work happens here, inside the span.
// Span closes when _enter is dropped.

Guidance:

  • Use skip(self) or skip_all to prevent large or non-Debug arguments from being emitted as span fields.
  • Never put secrets, API keys, or PII in fields(...). See §5.
  • For async code, always prefer Pattern A — span.enter() in Pattern B does not survive .await.

5. PII safety​

  • garraia_telemetry::http_trace_layer() excludes request and response headers from spans by default. Authorization headers, cookies, and other sensitive headers are never recorded.
  • Do not add span fields with names such as: password, secret, token, api_key, jwt, passphrase, authorization, or cookie.
  • To correlate a request without leaking credentials, use the x-request-id header value — it is safe to log and trace.

6. Metrics​

When GARRAIA_METRICS_ENABLED=true, the gateway exposes four baseline metrics at 127.0.0.1:9464/metrics:

MetricTypeDescription
garraia_requests_total{route, status}counterTotal HTTP requests by route and status code
garraia_http_latency_seconds{route}histogramRequest duration in seconds, by route
garraia_errors_total{kind}counterError count by error kind
garraia_active_sessionsgaugeCurrent number of active sessions

Call sites use the helpers from garraia_telemetry:

garraia_telemetry::inc_requests(route, status);
garraia_telemetry::record_latency(route, elapsed_seconds);
garraia_telemetry::inc_errors(kind);
garraia_telemetry::set_active_sessions(n);

6.0 Memory metrics (#957)​

Four more, emitted by the agent runtime whenever the memory system is in use:

MetricTypeDescription
garraia_memory_embed_latency_seconds{provider, operation}histogramTime spent inside the embeddings provider. operation is document (ingestion, batched) or query (recall, on the critical path)
garraia_memory_embed_failures_total{provider, operation}counterEmbedding calls that failed
garraia_memory_recall_latency_secondshistogramWhole recall — query embedding plus the search
garraia_memory_ingested_total{outcome}counterTurns ingested, by outcome: embedded, noise, no_provider, failed
garraia_memory_entries{has_embedding}gaugeEntries in memory, with and without a vector
garraia_memory_vector_index_sizegaugeRows in the vector index (vec_id_map)
garraia_memory_gauge_errors_totalcounterFailed reads of the two gauges above

The one to alert on is garraia_memory_embed_failures_total. Issue #948 described embedding failure as silent: the entry was stored without a vector and became invisible to semantic search forever. #948 made it loud in the log; this makes it visible as a trend — and it is the trend that lets someone notice the provider went down before recall degrades.

no_provider and failed are deliberately different outcomes. They are the difference between "nobody configured embeddings" and "it is configured and broken", which is the first question an operator asks. noise exists because of the ingestion filter (#952): without it, the count of entries without a vector would climb and nobody could tell defect from policy.

The two gauges are meant to be read together, and the gap between them is the signal: an entry with a vector in its column but missing from the index does not show up in semantic search. That was invisible until someone ran garra memory stats — which only reports when asked. As a gauge it becomes a trend, which is what prompts the asking.

The error counter exists because of the worst failure mode a gauge has: when the read stops working, the value does not disappear — Prometheus keeps serving the last one as if it were current, and nobody can tell "database unreachable" from "the base really is that size". Alert on rate(garraia_memory_gauge_errors_total[15m]) > 0 to make that silence audible.

They come from a dedicated worker (memory_gauge_worker) that starts whenever memory exists and resamples every 5 minutes. It is not a branch of the retention worker: that one only runs with memory.retention.enabled=true, which is off by default, so the gauges would be dead on almost every install.

Useful queries:

# Falha de embedding por minuto, por provider
rate(garraia_memory_embed_failures_total[5m])

# p95 do recall (o que o usuario espera)
histogram_quantile(0.95, rate(garraia_memory_recall_latency_seconds_bucket[5m]))

# Entradas com vetor que nao estao no indice — deveria ser sempre 0
garraia_memory_entries{has_embedding="true"} - garraia_memory_vector_index_size

# Fracao dos turnos que o filtro de ruido descartou
rate(garraia_memory_ingested_total{outcome="noise"}[1h])
  / ignoring(outcome) group_left sum without(outcome) (rate(garraia_memory_ingested_total[1h]))

Unlike the four baseline metrics, these are emitted through garraia_common::metrics, not garraia_telemetry — the crates that emit them (garraia-agents) are linked by the CLI, which must not pull in the OpenTelemetry/OTLP/axum stack. They use the metrics facade directly: library emits, binary installs the recorder. In the CLI, where no recorder is installed, every call is a cheap no-op.

Every label comes from a closed set — provider is a provider id, operation and outcome are enums rendered as &'static str. There is no path by which a session id, a user id or memory content reaches a label; a test asserts exactly that.

The /metrics endpoint binds to 127.0.0.1 only by default. To expose it to a remote Prometheus scraper, set GARRAIA_METRICS_BIND explicitly (e.g., 0.0.0.0:9464) and configure either GARRAIA_METRICS_TOKEN or GARRAIA_METRICS_ALLOW per §6.1 below. Without one of those the dedicated listener refuses to spawn (fail-closed startup) and the embedded route returns 503 for every non-loopback caller.

6.1 Security (plan 0024 / GAR-412)​

The gateway exposes /metrics on two surfaces:

  • Dedicated listener — spawned when GARRAIA_METRICS_ENABLED=true. Serves handle.render() from the globally-installed metrics-exporter-prometheus recorder. Bound to GARRAIA_METRICS_BIND (default 127.0.0.1:9464).
  • Embedded route — GET /metrics on the main gateway listener (port 3888 by default). Serves the legacy observability::Metrics snapshot — always on, always behind the same auth middleware.

Both surfaces share one metrics_auth_layer (crate garraia-gateway::metrics_auth). The middleware enforces the following matrix:

PeerToken configuredAllowlist configuredOutcome
loopback (127.0.0.1, ::1)noempty200 — dev ergonomics
loopbackyes—200 only with Authorization: Bearer <token>; otherwise 401
loopbacknoconfigured200 when loopback is in the allowlist; otherwise 403
non-loopbacknoempty503 (metrics: auth not configured) — safety net
non-loopbackyes—200 only with correct token; otherwise 401
non-loopback—configured200 only when peer IP ∈ allowlist; otherwise 403

Fail-closed semantics — 2 listeners, 2 layers (intentional):

  • Dedicated listener fails closed at startup: if GARRAIA_METRICS_ENABLED=true and the bind is non-loopback and neither token nor allowlist is configured, spawn_dedicated_metrics_listener returns Err(MetricsExporterError::AuthNotConfigured) and no socket is opened. The gateway main listener stays healthy — telemetry is fail-soft by GAR-384 contract.
  • Embedded route fails closed at runtime: the main listener always binds (it serves /v1/*, /admin/*, /auth/*, /ws) but the middleware returns 503/401/403 for any caller that doesn't satisfy the matrix above.

Deploy recommendations:

ScenarioRecommended config
Local dev(leave unset) — loopback-only defaults
Prometheus scraper on same hostGARRAIA_METRICS_BIND=127.0.0.1:9464 (no auth needed)
Scraper behind VPN / private LANGARRAIA_METRICS_BIND=0.0.0.0:9464 + GARRAIA_METRICS_ALLOW=10.0.0.0/8,192.168.0.0/16
Scraper with internet pathGARRAIA_METRICS_BIND=0.0.0.0:9464 + GARRAIA_METRICS_TOKEN=<random hex> (and TLS terminator in front)
Escape hatch for a legacy scraperGARRAIA_METRICS_ALLOW=0.0.0.0/0 — explicit opt-in to the pre-0024 behavior

Secret hygiene: GARRAIA_METRICS_TOKEN is redacted in TelemetryConfig's Debug impl. It is listed alongside GARRAIA_JWT_SECRET in CLAUDE.md regra #6 — never log its value, never commit it to configs.

7. Sampling​

  • Default GARRAIA_OTEL_SAMPLE_RATIO=1.0 records every trace — suitable for local development.
  • Recommended for production: 0.1 (10%) or lower depending on request volume.
  • The sampler is TraceIdRatioBased, which makes a consistent per-trace decision. All spans belonging to a sampled trace are recorded; no orphan spans are produced.

8. Troubleshooting​

No traces appear in Jaeger: Check that GARRAIA_OTEL_ENABLED=true is set. Verify the OTLP endpoint is reachable — curl -v http://localhost:4317 will not return an HTTP response (gRPC), but the TCP connection should succeed. If it is refused, the collector is not running.

"Telemetry init failed" on startup: The gateway continues without telemetry and logs the underlying error to stderr. Fix the configuration error and restart. The process does not abort.

/metrics returns 404: GARRAIA_METRICS_ENABLED=true is required for the dedicated listener on GARRAIA_METRICS_BIND. The embedded /metrics route on the main listener (port 3888 by default) is always registered. Also confirm you are hitting 127.0.0.1:9464 and not 0.0.0.0:9464 — the dedicated listener binds to loopback by default.

/metrics returns 401 (metrics: invalid token): GARRAIA_METRICS_TOKEN is configured on the gateway side and the request is missing Authorization: Bearer <token> or the value does not match. Fix the scraper config.

/metrics returns 403 (metrics: peer not allowed): GARRAIA_METRICS_ALLOW is configured and the peer IP (the immediate TCP peer — not X-Forwarded-For) is not inside any CIDR on the allowlist. Add the scraper's IP or broaden the CIDR.

/metrics returns 503 (metrics: auth not configured): The main listener is serving the embedded route but the peer is non-loopback and neither token nor allowlist is configured. Set one of GARRAIA_METRICS_TOKEN or GARRAIA_METRICS_ALLOW and restart, or proxy the scrape through loopback.

Dedicated metrics subsystem refused to start (metrics auth not configured for non-loopback bind; listener disabled): GARRAIA_METRICS_BIND is non-loopback and neither GARRAIA_METRICS_TOKEN nor GARRAIA_METRICS_ALLOW is set — plan 0024's startup fail-closed check refuses to expose unauth'd metrics. Configure one and restart. The gateway main listener is unaffected.

9. Next steps (out of scope for GAR-384)​

  • Structured per-subsystem dashboards in Grafana (future issue).
  • Alerting rules in Prometheus / Grafana.
  • Cross-service trace propagation when a gateway-to-workspace-service boundary exists (Fase 3).
  • Instrumentation of garraia-channels adapters (not yet covered).

References​