Saltar al contenido
AKOS
Todos los packs
StableDevOpsobservability

Observability — Instrumenting for Questions Nobody Asked Yet

devops/sre owns the *targets* — SLOs, error budgets, alerting, on-call. This pack owns the *signals* those targets are measured from, and the debugging that starts once an alert fires: which signal answers which question, how trace context survives a queue, where an attribute may go, what sampling keeps, and what telemetry must never carry.

opentelemetrytracingmetricsstructured-loggingcardinalitysampling

Principios

  • · P1 — Observability is the ability to answer new questions without shipping new code. Monitoring answers questions you thought of in advance; observability handles the ones you didn't. The distinction is not a dashboard count — it is whether a novel hypothesis can be tested against data you already have.
  • · P2 — The interesting failures are the ones nobody predicted. Known failure modes get a check and an alert, and then they stop being interesting. What takes systems down is the combination nobody enumerated: this customer, on that plan, with this payload, after that deploy. Instrumentation is judged by whether it can isolate a combination.
  • · P3 — Pre-aggregation destroys the questions you haven't asked yet. A counter incremented at write time has already thrown away who, where, and with what. Aggregation is a query-time decision; doing it at write time trades tomorrow's unknown question for today's storage bill.
  • · P4 — Instrumentation is part of the feature, not a follow-up. Code shipped without it is code that cannot be debugged in production, and the moment it needs debugging is the worst moment to add it. "We'll add logging if it breaks" is a plan to be blind exactly once.
  • · P5 — Traces, metrics, and logs answer different questions and none replaces the others. A trace shows where time went *in one request*. A metric shows the shape of *many* requests. A log records what happened *at one point*. Reaching for the wrong one is the most common instrumentation mistake — a metric cannot tell you why one customer's request was slow, and a trace cannot tell you whether it usually is.

Patrones

  • · Don't build a telemetry platform for a system that needed a request identifier. Generate one at the edge, log it everywhere, return it to the client. On a small stack that is most of the value for almost none of the cost.
  • · Instrument the boundaries, then stop and see what's missing. Inbound, outbound, database, queue. Most incidents live at a boundary, and instrumenting everything up front produces volume nobody queries.
  • · If you're about to put an ID on a metric label, you wanted a span. This single substitution prevents most surprise telemetry bills.
  • · Ask "what dimension would I slice by at 3am?" and add that attribute now.
  • · Put the build identifier on everything. "Did this start with the last deploy" is.

Riesgos

  • · Ver anti-patrones específicos del dominio en el pack.

Checklist

  • [ ] No credential, token, session identifier, API key, or authorization header appears in any attribute, log field, or URL. (OBS31)
  • [ ] Personal data in telemetry is inventoried, pseudonymized where possible, given a retention period, and covered by the deletion path — the vendor is a processor. (OBS32)
  • [ ] No unbounded attribute (user ID, raw URL with identifiers, email) is used as a metric label. (OBS3, OBS13)
  • [ ] Trace context propagates across every hop, including queues and background jobs. (OBS6)
  • [ ] Propagation is verified end-to-end by an executed check, not assumed from the SDK being installed. (OBS7)
  • [ ] Span names are low-cardinality; identifiers live in attributes. (OBS9)

Fuentes

En la bibliografía AKOS