Emit an OpenTelemetry span for each HTTP request

[asg017]

Follow-up to #2862. Adds a single SERVER span per HTTP request, so the database spans from that
PR have a trace to belong to.

 datasette/app.py                 |  48 ++-
 datasette/telemetry.py           | 230 +++++++++++
 datasette/telemetry_registry.py  | 129 +++++-
 docs/internals.rst               |  53 ++-
 tests/conftest.py                |   1 +
 tests/test_http_span.py          | 855 +++++++++++++++++++++++++++++++++++++++
 tests/test_telemetry_registry.py | 192 +++++++--
 7 files changed, 1455 insertions(+), 53 deletions(-)

Why this is needed

Without it, nothing in Datasette creates a span for the request, so every span is a root span and
a trace UI shows dozens of unrelated single-span traces per page with no URL on any of them —
interleaved across concurrent requests.

Measured on this branch against fixtures.db, one warm request each (the first request of a process
also pays connection warm-up, so these are the second):

Request	Without this PR	With this PR
/fixtures/facetable?_facet=state&_facet=_city_id&_facet=on_earth	112 spans in 56 traces	113 spans in 1 trace
/fixtures/facetable/1	24 spans in 12 traces	25 spans in 1 trace
/fixtures/searchable.json	8 spans in 4 traces	9 spans in 1 trace

Every one of those "traces" without the PR is a two-span db.query/db.query.execute pair with no
URL, no status and no method on it, arriving interleaved with every other concurrent request.

opentelemetry-instrument does not fix this on its own. Auto-instrumentation only picks up
frameworks that ship an instrumentor entry point, and Datasette's raw ASGI app is not one. So the
alternative to this PR is telling every operator to install opentelemetry-instrumentation-asgi and
write a three-line asgi_wrapper plugin, which is a poor default.

Here is /fixtures/searchable.json in full, before and after. Same request, same spans; the only
difference is whether they have a parent.

Before - four unrelated root traces:

db.query                                     |====                              |   0.08ms
  db.query.execute                           | =                                |   0.02ms
db.query                                     |         =====                    |   0.12ms
  db.query.execute                           |          ===                     |   0.06ms
db.query                                     |                         ===      |   0.06ms
  db.query.execute                           |                          =       |   0.01ms
db.query                                     |                               ===|   0.05ms
  db.query.execute                           |                                = |   0.01ms

After - one trace, rooted in a span that says what was requested and how it ended:

GET /(?P<database>[^\/\.]+)/(?P<table>[^\... |==================================|   1.00ms
  db.query                                   |  ===                             |   0.08ms
    db.query.execute                         |   =                              |   0.02ms
  db.query                                   |          =====                   |   0.14ms
    db.query.execute                         |          ===                     |   0.07ms
  db.query                                   |                       ==         |   0.06ms
    db.query.execute                         |                        =         |   0.01ms
  db.query                                   |                            ==    |   0.06ms
    db.query.execute                         |                             =    |   0.01ms

Design

The middleware is applied outermost in Datasette.app(), after the asgi_wrapper plugin loop,
so plugin middleware runs inside the span rather than outside it. Putting it in
DatasetteRouter.__call__ instead would leave CSRF and every plugin middleware outside the span,
which reintroduces the orphan problem for instrumented plugins.

Only scope["type"] == "http" is wrapped; lifespan and websocket scopes pass straight through.

Everything needed is in opentelemetry-api — opentelemetry.propagate.extract, Getter,
SpanKind.SERVER. No new dependency, and core still never owns a provider.

Streaming works with a plain finally, which is why this is ~130 lines rather than the ~700 that
opentelemetry-instrumentation-asgi needs. Datasette's app never returns before the body is sent:
route_path awaits response.asgi_send(send), and for CSV AsgiStream.asgi_send runs stream_fn
inline. So no deferred-end machinery is required. Verified against ?_stream=1 exports, where
nearly all the time is in the body send.

Status code comes from sniffing http.response.start, not from Response objects — views bypass
Response freely (asgi_static, favicon, AsgiStream, AsgiFileDownload all call send
directly), and a wrapped send is the only thing that sees every response including 404 and 500
handlers.

Errors are driven by status_code >= 500, since semconv says 4xx is not an error for a SERVER span.
asyncio.CancelledError on client disconnect is caught explicitly — it's a BaseException, so
route_path's except Exception deliberately doesn't see it.

Attributes

http.request.method (clamped to the known set, else _OTHER), http.route,
http.response.status_code, url.path, url.scheme, server.address, user_agent.original,
error.type. Span name is {method} {route}, falling back to bare {method} for unrouted requests.

http.route is the compiled route pattern, e.g.
/(?P<database>[^\/\.]+)/(?P<table>[^\/\.]+)(\.(?P<format>\w+))?$. Ugly, but exact, bounded by the
route table and zero-parse. I deliberately did not write a regex→/{database}/{table} prettifier —
that transform accretes edge cases, and Django's instrumentation ships regex-flavoured routes too.
Happy to add one if you'd prefer prettier output.

Since routing happens in DatasetteRouter but the span starts at the edge, http.route is set and
the span renamed from the router, after resolve_routes.

The router finds that span through the ASGI scope, not through get_current_span(). That is a
correction I made while writing the tests: a plugin asgi_wrapper runs inside this middleware, so
an instrumented plugin makes its own span current for the whole request. Reading the current span in
the router therefore renamed the plugin's INTERNAL span to GET <route> and hung http.route off
it, while the actual request span kept a bare method name and never got the one attribute a trace UI
groups requests by. It reproduces with a five-line plugin. Going through the scope also keeps the
fallback: if no Datasette span is in the scope the router falls back to the current span, which is
how an externally-created SERVER span still gets enriched.

Deliberately not recorded:

• url.query — Datasette query strings carry user SQL (?sql=) and canned-query parameters.
  Recording it by default would export exactly the class of data the rest of this work is careful
  about.
• client.address — an IP is borderline PII.

The security question, stated plainly

This extracts W3C traceparent from inbound request headers by default, which means on a public
instance an arbitrary client can influence your traces:

1. Trace-ID pollution — the client dictates trace IDs.
2. Sampling control — the SDK's default sampler is parentbased_always_on, so under a
   parent-based ratio sampler a client's sampled flag can force 100% recording (a telemetry-cost
   DoS) or suppress recording.
3. Baggage injection, via the default composite propagator.

I've gone with extract-by-default because it's the ecosystem norm — Flask, Django, FastAPI and the
ASGI instrumentation all extract unconditionally, and edges that care strip or restart traces at the
proxy. Crucially it uses the global propagator, so the operator stays in control with no
Datasette-specific setting: OTEL_PROPAGATORS=none disables extraction entirely,
OTEL_PROPAGATORS=tracecontext drops baggage, and a non-parent-based sampler neutralises (2).

That's consistent with core never touching sampling or configuration. But it is a genuine decision
rather than an obvious one, and if you'd rather Datasette not trust inbound trace context by
default I'm happy to invert it.

Interactions

datasette.startup nests differently depending on how you run it. datasette serve calls
invoke_startup() before uvicorn starts, so the startup span stays its own trace. ASGI-hosted and
programmatic deployments hit AsgiRunOnFirstRequest cold, so startup nests under the first
request's span. That's benign and arguably more honest — it genuinely is that request's latency.

Double instrumentation is harmless. If an operator also installs
opentelemetry-instrumentation-asgi, its middleware lands inside this one and becomes a redundant
child SERVER span in the same trace. Nothing is re-orphaned. I haven't added an opt-out —
OTEL_SDK_DISABLED and the no-provider default already cover "turn it off".

No-provider cost is near zero. With no provider every span is a NonRecordingSpan, the
middleware short-circuits on span.is_recording(), and the send wrapper is never allocated.
Measured over 300 requests to /fixtures/searchable.json in a process with no SDK loaded,
interleaved in blocks to cancel out drift:

with middleware       n=300  median=1.226ms  mean=1.332ms  p90=1.729ms
without middleware    n=300  median=1.237ms  mean=1.314ms  p90=1.726ms

The median difference across four repeats of that benchmark was -11.4us, -4.8us, -2.8us and +2.3us -
it changes sign between runs, so it is smaller than the noise floor rather than merely small.

(The check has to be is_recording(), not get_span_context().is_valid. With no provider and an
inbound traceparent, the API's NoOpTracer returns a NonRecordingSpan carrying the remote
context: valid, sampled, and recording nothing. An is_valid guard would take the slow path for
every request arriving from an instrumented caller.)

How to test this

pip install -e '.[test]'
pytest -n auto -m "not serial" && pytest -m "serial"

OTEL_TRACES_EXPORTER=console OTEL_METRICS_EXPORTER=none OTEL_LOGS_EXPORTER=none \
  opentelemetry-instrument datasette mydb.db
# one request should now produce ONE trace, not dozens

Known limitations

• http.route is a regex pattern, not a pretty template.
• Startup work nests under the first request span in ASGI-hosted deployments.
• Inbound trace context is trusted by default; see above.

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}