Observability Is Where the Product Stops Guessing

After removing a stale model name, the configuration told us which model production should use. It still could not prove which selection path a real request had taken, whether a fallback ran, or whether a guard rejected the result.

Observability closes that gap. It gives the product a record of what happened before the user has to report the damage.

Dashboards are the visible part

It is tempting to think of observability as a dashboard project. Add Grafana. Add Sentry. Add traces. Add logs. Add a page that looks serious enough to leave open on another monitor.

That is the visible part.

The real work is deciding what the system needs to explain.

For a wedding platform, the questions might be:

Did the guest RSVP flow complete?

Did the confirmation email send?

Did a webhook arrive twice?

Did a template render slowly?

Did a payment callback fail before or after state changed?

For a household assistant, the questions are different:

Did a scheduled task recommendation get generated?

Did the agent choose not to act because context was missing?

Did a calendar sync fail?

Did a budget signal update after a ledger change?

Did an LLM call time out, return invalid structure, or get skipped by a guard?

The dashboard is only useful if the instrumentation was shaped by questions like these.

Optional instrumentation still needs a contract

Small products often need instrumentation to be optional. Local development should work without a cloud token. A preview environment may not have full telemetry. A production deployment may start with error reporting before traces.

Optional does not mean vague.

It should be clear what happens when a DSN is missing. Is Sentry a no-op? Does the app still boot? Are traces disabled silently, or does startup log the absence? Which environment variables are public? Which are server-only? Which values are read at build time?

The best optional instrumentation has a boring contract:

If the key is absent, the feature is off.

If the key is present, the feature initializes once.

If initialization fails, the failure is visible without breaking unrelated product behavior.

That contract matters because observability code sits close to startup code. Careless instrumentation can make the thing that is supposed to explain production become another reason production does not boot.

AI makes traces more important, not less

AI product work increases the need for observability because many failures are not binary.

The request may succeed but produce the wrong shape.

The provider may respond, but too slowly for the workflow.

The model may be configured through one path in development and another in production.

A fallback may hide the original failure so well that nobody notices the product is degraded.

When an AI-backed feature breaks, a normal error log is often not enough. You need to know which provider was selected, which model name was used, whether the guardrail rejected output, whether retry happened, whether the response was cached, and whether the user saw a degraded but acceptable state.

The goal is to make the product debuggable without asking the user to reconstruct the request for us.

Building with AI: AI wrote much of the instrumentation across Django, Celery, Next.js, Nginx, and Compose after I defined the privacy and service boundaries. Two shortcuts survived the first pass: a local config check looked too much like proof of live telemetry, and an Alloy discovery rule could collect every container on a shared host. I separated compile-time proof from production evidence and narrowed discovery before calling the rollout complete.

Sentry tells you what hurt

Error monitoring has a different role.

Traces and metrics explain the shape of behavior. Sentry tells you where the product cut someone.

That distinction is useful. A metric can show an increase in failed requests. A trace can show a slow dependency. But an exception with request context, release version, and environment tells you which line of code actually broke under production pressure.

The mistake is treating Sentry as something to add after the product is stable.

Stability is one of the things Sentry helps you discover.

For small teams, this is especially important because support, engineering, and product are often the same person. The faster production can point to the real failure, the less time the builder spends reconstructing a crime scene from memory.

Instrument decisions as well as crashes

The most useful telemetry often records decisions, not crashes.

Why was this task assigned to this person?

Why was this email not sent?

Why did this signup get blocked?

Why did this agent refuse to make a recommendation?

Why did this provider get selected?

Those are product questions. They deserve technical evidence. A Task Count Cannot Tell You Who Is Free follows the first question into workload, calendar coverage, and the explanation shown to the household.

Good observability gives the product a memory of what it did and why. That record lets the builder start with evidence, decide whether the behavior was correct, and change the part that was not.

What production needs to explain

This became concrete in the Sivella and Vowframes deployments. Sivella names four traced processes separately: sivella-backend, sivella-frontend, sivella-celery-worker, and sivella-celery-beat. Vowframes adds vowframes-nginx to the corresponding backend, frontend, worker, and beat services.

The switches are explicit: OTEL_ENABLED, OTEL_METRICS_ENABLED, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME, DEPLOYMENT_ENVIRONMENT, SERVICE_VERSION, LOG_JSON, and the public Faro settings. Sentry remains enabled for exceptions; its performance tracing can be disabled after Grafana traces are verified rather than removed in the same change.

Trace context crosses Django into Celery. JSON application logs carry trace_id and span_id. Nginx logs include request and upstream timing, status, request ID, and trace headers where available. The host's existing Alloy instance receives OTLP over host.docker.internal; neither application installs a second collector.

Privacy is enforced in code rather than in a dashboard convention. A span processor drops SQL statement text, query strings, request headers, cookies, authorization, bodies, and other private household or wedding content. Metrics avoid user IDs, household IDs, task IDs, and raw URLs as labels. Faro does not enable session replay.

Where local checks stop

Before deployment, I rendered the Compose configuration, started the applications with telemetry disabled, and checked that each process received its intended service name and OTLP host. Vowframes also fixed an Alloy discovery rule that would otherwise have collected every container on a shared host; a keep rule now admits only Vowframes containers.

Those local checks could not show that Grafana Cloud had received live data. I still had to find one release in Loki, follow one Django-to-Celery trace in Tempo, see the HTTP and task metrics in Grafana, and join a browser trace to its backend request.

That gives me a faster path through provider, queue, HTTP, and browser failures without asking a user to reconstruct a private request. Distinct service names, privacy scrubbers, disabled-path startup checks, and the live Grafana walkthrough now travel with the deployment.