From a41de04a553718240e41b06bee5abe7eddcdf269 Mon Sep 17 00:00:00 2001 From: KodeStar Date: Sat, 1 Aug 2026 19:28:04 +0100 Subject: [PATCH] docs(meta): metaserve readiness, degraded boot, and paged endpoints metaserve's deployment behaviour changed with the serve-at-scale work (KodeStar/audiosilo-meta#1261) and the pages described the old shape - one part dangerously so. api.md: - /healthz is a READINESS check, not liveness: 503 {"status":"starting"} with Retry-After until an artifact loads, 200 with built_at after. Adds a warning against wiring it as a liveness probe, which would restart-loop a server deliberately waiting out a GitHub outage. - New "Boot and degraded start" section replacing the "fetches the newest data release on boot so it never starts empty" claim: cached-artifact adoption by verified digest, the stale-cache fallback (log-only signal), and the nothing-cached 503 + honest Retry-After with 30s-to-interval backoff. - people/{id} and series/{id} document ?limit=/?offset= and the new authored_total / narrated_total / works_total / limit / offset fields. People page at 100 (max 500); series returns the whole list unless ?limit is given, because the player's series rail depends on it. - coverage/works ?q= is a word-prefix FTS match over title, subtitle, authors, narrators and series names, no longer a title/author substring. - --db described as optional; drops the now-false "build-time data" note. overview.md: - The image is the site build plus the metaserve binary with no data; adds "The published image" covering the cache volume, the valid dataless boot, and readiness probing. - release.yml also triggers on internal/build/** and cmd/metabuild/**. - Drops metabuild's retired --added flag. --- docs-developers/meta/api.md | 104 ++++++++++++++++++++++++++----- docs-developers/meta/overview.md | 32 ++++++++-- 2 files changed, 117 insertions(+), 19 deletions(-) diff --git a/docs-developers/meta/api.md b/docs-developers/meta/api.md index 9c238d9..24bc89b 100644 --- a/docs-developers/meta/api.md +++ b/docs-developers/meta/api.md @@ -17,12 +17,31 @@ routes are `GET`. ## `/healthz` -Liveness plus a cheap freshness signal. Always 200 while a snapshot is loaded: +A **readiness** check, not a liveness check. Until an artifact is loaded it +answers **503** with a `Retry-After` header carrying the real seconds until the +next fetch attempt: + +```json +{ "status": "starting" } +``` + +Once a snapshot is loaded it answers 200 with a cheap freshness signal - `built_at` +is the build time of the artifact currently being served, so a stuck poller shows +up as an ageing timestamp: ```json { "status": "ok", "built_at": "2026-07-15T...", "works": 1234 } ``` +:::warning Readiness probe yes, liveness probe no +Wire `/healthz` as the **readiness/startup probe** so an orchestrator holds traffic +back until the catalogue is in. Do **not** wire it as a liveness probe: that +restart-loops a server that is patiently waiting out a GitHub outage. The degraded +boot is deliberate (see [boot and degraded start](#boot-and-degraded-start)) - the +process is healthy, it simply has no data yet, and killing it only resets the +backoff it is already managing. +::: + ## `/api/v1/stats` Catalogue totals, precomputed once per loaded snapshot: @@ -75,27 +94,42 @@ The chapter list for one recording of a work: `{"chapters": [{title, start_ms, length_ms}]}`, ordered by chapter index. An unknown work/recording yields an empty list, not a 404. -## `/api/v1/people/{id}` +## `/api/v1/people/{id}?limit=&offset=` -A person plus their works, or 404 `person not found`: +A person plus their works, or 404 `person not found`. Both credit lists are +**paged**: ```json { "id": "...", "name": "...", "sort_name": "...", "authored": [workCard...], - "narrated": [{ "work": workCard, "recording_id": "..." }] } + "narrated": [{ "work": workCard, "recording_id": "..." }], + "authored_total": 0, "narrated_total": 0, + "limit": 100, "offset": 0 } ``` -## `/api/v1/series/{id}` +`limit` defaults to **100** and is clamped to a maximum of **500**; `offset` is a +non-negative row offset. An unparseable or non-positive value falls back to the +default rather than erroring. The window applies to `authored` and `narrated` +**independently**, and `authored_total` / `narrated_total` are the unpaged counts +of each. A client must page against those totals rather than assume the arrays are +complete - a prolific narrator will exceed one page. + +## `/api/v1/series/{id}?limit=&offset=` A series with its ordered member works, or 404 `series not found`: ```json { "id": "...", "name": "...", "authors": [personRef...], - "works": [{ "position": "2.5", "work": workCard }] } + "works": [{ "position": "2.5", "work": workCard }], + "works_total": 0, "limit": 0, "offset": 0 } ``` `works` is sorted by the numeric start of each `position` string (so `"1-3.5"` -sorts by 1). +sorts by 1). Paging here is **opt-in**: with no `?limit=` the whole member list is +returned and the echoed `limit` is `0` (the player's series rail depends on getting +the complete list). Pass `?limit=` - clamped to a maximum of **500** - with an +optional `?offset=` to window it. `works_total` is always the unpaged member count, +so it is the reliable "how long is this series" number either way. ## `/api/v1/lookup?asin=|isbn=` @@ -123,7 +157,11 @@ These back the site's contribute page and stay small at any catalogue size. - **`/api/v1/coverage/works?filter=&q=&limit=&offset=`** - the paginated, searchable per-work browser. `filter` selects the dimension - `missing` (missing any dimension) or `has_characters` / `has_recaps` / `has_recap_summary` - and an - unknown filter is 400 `unknown filter`. `q` matches title/author; `limit` + unknown filter is 400 `unknown filter`. `q` is a **full-text** match over the + work's title and subtitle, its authors, its recordings' narrators, and its series + names - it runs through the same escaped FTS path as `/api/v1/search`, so it + matches **whole words with the final token as a prefix**, not arbitrary + substrings (`tolki` matches "Tolkien"; `olkien` does not). `limit` defaults to 25, clamped to `[1, 100]`; `offset` is a non-negative row offset. The response carries a per-filter `available` flag that is false when the dimension is not evaluable at the artifact's schema version. @@ -172,7 +210,7 @@ is non-fatal - the fallback poller still discovers the release. | Flag / env | Default | Purpose | |---|---|---| | `--addr` | `:8080` | listen address | -| `--db` | (none) | a local `meta.sqlite` artifact to serve (dev) | +| `--db` | (none) | a local `meta.sqlite` artifact to serve immediately (dev; the published image ships none and relies on `--poll`) | | `--site` | (none) | a static site directory to serve at `/` | | `--poll` | `false` | fetch and hot-swap the newest data release from GitHub Releases | | `--repo` | `KodeStar/audiosilo-meta` | GitHub `owner/name` to poll | @@ -181,9 +219,43 @@ is non-fatal - the fallback poller still discovers the release. | `GITHUB_TOKEN` (env) | (none) | raises the GitHub API rate limit | | `METASERVE_WEBHOOK_SECRET` (env) | (none) | enables the signed release webhook (requires `--poll`) | -With `--poll` and no `--db`, metaserve fetches the newest data release on boot so -it never starts empty. With both, the baked `--db` serves immediately and the -poller still runs one refresh at startup. +At least one of `--db` and `--poll` is required (`New` refuses "nothing to serve" +otherwise). With `--poll` and no `--db` - the production shape - metaserve fetches +its catalogue at boot and **can legitimately start empty**; see below. With both, +the local `--db` serves immediately and the poller still runs one refresh at +startup. + +## Boot and degraded start + +The published image ships **no data** (see +[the overview](overview.md#the-published-image)), so a production boot always +fetches its catalogue. A fetch that fails does not stop the process - it degrades +visibly instead, in one of three states: + +- **GitHub reachable** - the newest data release loads and the server is ready. If + the `--cache` directory already holds that release's artifact, it is verified + against the release's `meta.sqlite.sha256` and adopted **without downloading** + (matched by digest, never trusted by filename). This is what makes a restart on a + persistent cache volume cheap. +- **GitHub unreachable, something cached** - the newest cached artifact is adopted + and served, logged loudly as stale, and replaced by the first poll that succeeds. + Serving slightly old data beats refusing to serve data that is on disk. The + staleness is a **log-only** signal: no endpoint reports it, and `built_at` on + `/healthz` or `/api/v1/stats` is the only hint a client gets. +- **GitHub unreachable, nothing cached** - the process listens anyway. A static + `--site` still serves, but `/healthz`, every `/api/v1` route and `/abs/search` + answer **503** with an honest `Retry-After` and the envelope: + + ```json + { "error": "no data loaded yet: the server is fetching the latest release" } + ``` + + Retries back off from **30 seconds**, doubling until they reach `--interval`, and + `Retry-After` always reports the wait actually scheduled. The moment a release + loads, the backoff resets and the server becomes ready without a restart. + +Every one of those states is a correctly-working process, which is why `/healthz` +must be a readiness probe and never a liveness one. ## Serving and refresh @@ -204,7 +276,9 @@ webhook keep it current: handle (closed after a grace delay). A rejected patch never swaps, and a poll failure only logs and retries - it never crashes the process. -The startup refresh means a recreated production container catches up to the -newest release within seconds instead of serving build-time data for a full -`--interval`. The release asset contract these steps rely on is described on +The startup refresh means a recreated production container reaches the newest +release within seconds rather than at the first `--interval` tick. Superseded cache +files are pruned on every adopt, sparing any artifact still draining its swap +grace, so the cache does not grow release by release. The release asset contract +these steps rely on is described on [the overview](overview.md#release-artifacts). diff --git a/docs-developers/meta/overview.md b/docs-developers/meta/overview.md index 317d3cf..e4a201c 100644 --- a/docs-developers/meta/overview.md +++ b/docs-developers/meta/overview.md @@ -69,7 +69,7 @@ internal/importer OpenAudible / Libation export -> canonical records (ASIN ded internal/issueform issue-form body -> canonical records + an ok/duplicate/needs-human/invalid verdict internal/build the deterministic SQLite builder (FTS5, ASIN/ISBN indexes, added_at) internal/serve the read-only HTTP API + ABS provider + GitHub-release poller/hot-swap -Dockerfile image: site build + metaserve + baked data +Dockerfile image: the site build + the metaserve binary - no data (see below) .github/ issue forms + CI workflows (check, release, image, intake, ai-verify) ``` @@ -86,7 +86,7 @@ run ./cmd/`. |---|---| | `metacheck` | Validates the whole `data/` tree - schema, id/shard agreement, referential integrity, uniqueness, chapter ordering, series positions. Prints one line per problem and exits 1 if any are found. | | `metafmt` | Enforces canonical JSON for `data/**/*.json` (sorted keys, 2-space indent, single trailing LF). `--check` lists non-canonical files and exits 1; `--write` rewrites them. | -| `metabuild` | Compiles `data/` into the SQLite artifact (`-o meta.sqlite`). Runs the full validation first and refuses to build invalid data; `--added` dates each work from a git-history-derived list. | +| `metabuild` | Compiles `data/` into the SQLite artifact (`-o meta.sqlite`). Runs the full validation first and refuses to build invalid data. Deterministic: identical data produces an identical artifact. | | `metaserve` | Serves the compiled artifact read-only over HTTP (and optionally the static site at `/`), hot-swapping newer GitHub releases. See [the HTTP API](api.md). | | `metascan` | Scans a local audiobook folder into an import JSON - see [contributing data](contributing-data.md#scanning-local-files-metascan). | | `metaimport` | Ingests an OpenAudible/Libation library export into `data/` - see [contributing data](contributing-data.md#bulk-importers-metaimport). | @@ -114,8 +114,10 @@ go run ./cmd/metaserve --db meta.sqlite --addr :8080 ## Release artifacts On merge to `main`, `.github/workflows/release.yml` publishes a **dated data -release** tagged `data-vYYYY.MM.DD-` when data or schema changes land. -The asset contract: +release** tagged `data-vYYYY.MM.DD-` when data or schema changes land - +and also when the builder itself changes (`internal/build/**`, `cmd/metabuild/**`), +so an artifact-shaping change such as a new index actually reaches a published +artifact. The asset contract: - `meta.sqlite.gz` + `meta.sqlite.gz.sha256` - the universal anchor every consumer verifies against. @@ -131,6 +133,28 @@ non-prerelease release carrying `meta.sqlite.gz` with the maximum `published_at` either kind). The [`metaserve` refresh loop](api.md#serving-and-refresh) applies the same rule. +## The published image + +`ghcr.io/kodestar/audiosilo-meta` is **the site build plus the `metaserve` binary, +and no data at all**. The catalogue is not baked in: the container fetches the +newest data release at boot and hot-swaps every release after that, which keeps +image size and image build time independent of how large the catalogue grows. + +The practical consequences for a deployment: + +- **Give it a cache volume.** The image runs with `--poll --cache /data/cache` and + declares `/data` as a volume. A restart that finds the current release's artifact + already in the cache verifies it against the release's `meta.sqlite.sha256` and + adopts it **without downloading** - so restarts stay cheap. Without a persistent + volume every restart re-downloads the catalogue. Budget for two artifacts at + peak, one in steady state. +- **A boot with no data is a valid state, not a crash.** The server stays up and + degrades visibly if GitHub is unreachable - full detail in + [boot and degraded start](api.md#boot-and-degraded-start). +- **Probe `/healthz` for readiness only.** It reports readiness, not liveness; a + liveness probe on it will restart-loop a server that is correctly waiting out an + outage. + ## How it connects to the rest of AudioSilo audiosilo-meta is the **upstream** of a three-repo metadata seam. `metaserve`