Skip to content

meta: metaserve deployment and API doc updates for the scale work #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
KodeStar merged 1 commit into from
Aug 1, 2026
Merged

meta: metaserve deployment and API doc updates for the scale work #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
104 changes: 89 additions & 15 deletions docs-developers/meta/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,31 @@ routes are `GET`.

## `/healthz`

Liveness plus a cheap freshness signal. Always 200 while a snapshot is loaded:
A **readiness** check, not a liveness check. Until an artifact is loaded it
answers **503** with a `Retry-After` header carrying the real seconds until the
next fetch attempt:

```json
{ "status": "starting" }
```

Once a snapshot is loaded it answers 200 with a cheap freshness signal - `built_at`
is the build time of the artifact currently being served, so a stuck poller shows
up as an ageing timestamp:

```json
{ "status": "ok", "built_at": "2026-07-15T...", "works": 1234 }
```

:::warning Readiness probe yes, liveness probe no
Wire `/healthz` as the **readiness/startup probe** so an orchestrator holds traffic
back until the catalogue is in. Do **not** wire it as a liveness probe: that
restart-loops a server that is patiently waiting out a GitHub outage. The degraded
boot is deliberate (see [boot and degraded start](#boot-and-degraded-start)) - the
process is healthy, it simply has no data yet, and killing it only resets the
backoff it is already managing.
:::

## `/api/v1/stats`

Catalogue totals, precomputed once per loaded snapshot:
Expand Down Expand Up @@ -75,27 +94,42 @@ The chapter list for one recording of a work: `{"chapters": [{title, start_ms,
length_ms}]}`, ordered by chapter index. An unknown work/recording yields an empty
list, not a 404.

## `/api/v1/people/{id}`
## `/api/v1/people/{id}?limit=&offset=`

A person plus their works, or 404 `person not found`:
A person plus their works, or 404 `person not found`. Both credit lists are
**paged**:

```json
{ "id": "...", "name": "...", "sort_name": "...",
"authored": [workCard...],
"narrated": [{ "work": workCard, "recording_id": "..." }] }
"narrated": [{ "work": workCard, "recording_id": "..." }],
"authored_total": 0, "narrated_total": 0,
"limit": 100, "offset": 0 }
```

## `/api/v1/series/{id}`
`limit` defaults to **100** and is clamped to a maximum of **500**; `offset` is a
non-negative row offset. An unparseable or non-positive value falls back to the
default rather than erroring. The window applies to `authored` and `narrated`
**independently**, and `authored_total` / `narrated_total` are the unpaged counts
of each. A client must page against those totals rather than assume the arrays are
complete - a prolific narrator will exceed one page.

## `/api/v1/series/{id}?limit=&offset=`

A series with its ordered member works, or 404 `series not found`:

```json
{ "id": "...", "name": "...", "authors": [personRef...],
"works": [{ "position": "2.5", "work": workCard }] }
"works": [{ "position": "2.5", "work": workCard }],
"works_total": 0, "limit": 0, "offset": 0 }
```

`works` is sorted by the numeric start of each `position` string (so `"1-3.5"`
sorts by 1).
sorts by 1). Paging here is **opt-in**: with no `?limit=` the whole member list is
returned and the echoed `limit` is `0` (the player's series rail depends on getting
the complete list). Pass `?limit=` - clamped to a maximum of **500** - with an
optional `?offset=` to window it. `works_total` is always the unpaged member count,
so it is the reliable "how long is this series" number either way.

## `/api/v1/lookup?asin=|isbn=`

Expand Down Expand Up @@ -123,7 +157,11 @@ These back the site's contribute page and stay small at any catalogue size.
- **`/api/v1/coverage/works?filter=&q=&limit=&offset=`** - the paginated,
searchable per-work browser. `filter` selects the dimension - `missing` (missing
any dimension) or `has_characters` / `has_recaps` / `has_recap_summary` - and an
unknown filter is 400 `unknown filter`. `q` matches title/author; `limit`
unknown filter is 400 `unknown filter`. `q` is a **full-text** match over the
work's title and subtitle, its authors, its recordings' narrators, and its series
names - it runs through the same escaped FTS path as `/api/v1/search`, so it
matches **whole words with the final token as a prefix**, not arbitrary
substrings (`tolki` matches "Tolkien"; `olkien` does not). `limit`
defaults to 25, clamped to `[1, 100]`; `offset` is a non-negative row offset. The
response carries a per-filter `available` flag that is false when the dimension
is not evaluable at the artifact's schema version.
Expand Down Expand Up @@ -172,7 +210,7 @@ is non-fatal - the fallback poller still discovers the release.
| Flag / env | Default | Purpose |
|---|---|---|
| `--addr` | `:8080` | listen address |
| `--db` | (none) | a local `meta.sqlite` artifact to serve (dev) |
| `--db` | (none) | a local `meta.sqlite` artifact to serve immediately (dev; the published image ships none and relies on `--poll`) |
| `--site` | (none) | a static site directory to serve at `/` |
| `--poll` | `false` | fetch and hot-swap the newest data release from GitHub Releases |
| `--repo` | `KodeStar/audiosilo-meta` | GitHub `owner/name` to poll |
Expand All @@ -181,9 +219,43 @@ is non-fatal - the fallback poller still discovers the release.
| `GITHUB_TOKEN` (env) | (none) | raises the GitHub API rate limit |
| `METASERVE_WEBHOOK_SECRET` (env) | (none) | enables the signed release webhook (requires `--poll`) |

With `--poll` and no `--db`, metaserve fetches the newest data release on boot so
it never starts empty. With both, the baked `--db` serves immediately and the
poller still runs one refresh at startup.
At least one of `--db` and `--poll` is required (`New` refuses "nothing to serve"
otherwise). With `--poll` and no `--db` - the production shape - metaserve fetches
its catalogue at boot and **can legitimately start empty**; see below. With both,
the local `--db` serves immediately and the poller still runs one refresh at
startup.

## Boot and degraded start

The published image ships **no data** (see
[the overview](overview.md#the-published-image)), so a production boot always
fetches its catalogue. A fetch that fails does not stop the process - it degrades
visibly instead, in one of three states:

- **GitHub reachable** - the newest data release loads and the server is ready. If
the `--cache` directory already holds that release's artifact, it is verified
against the release's `meta.sqlite.sha256` and adopted **without downloading**
(matched by digest, never trusted by filename). This is what makes a restart on a
persistent cache volume cheap.
- **GitHub unreachable, something cached** - the newest cached artifact is adopted
and served, logged loudly as stale, and replaced by the first poll that succeeds.
Serving slightly old data beats refusing to serve data that is on disk. The
staleness is a **log-only** signal: no endpoint reports it, and `built_at` on
`/healthz` or `/api/v1/stats` is the only hint a client gets.
- **GitHub unreachable, nothing cached** - the process listens anyway. A static
`--site` still serves, but `/healthz`, every `/api/v1` route and `/abs/search`
answer **503** with an honest `Retry-After` and the envelope:

```json
{ "error": "no data loaded yet: the server is fetching the latest release" }
```

Retries back off from **30 seconds**, doubling until they reach `--interval`, and
`Retry-After` always reports the wait actually scheduled. The moment a release
loads, the backoff resets and the server becomes ready without a restart.

Every one of those states is a correctly-working process, which is why `/healthz`
must be a readiness probe and never a liveness one.

## Serving and refresh

Expand All @@ -204,7 +276,9 @@ webhook keep it current:
handle (closed after a grace delay). A rejected patch never swaps, and a poll
failure only logs and retries - it never crashes the process.

The startup refresh means a recreated production container catches up to the
newest release within seconds instead of serving build-time data for a full
`--interval`. The release asset contract these steps rely on is described on
The startup refresh means a recreated production container reaches the newest
release within seconds rather than at the first `--interval` tick. Superseded cache
files are pruned on every adopt, sparing any artifact still draining its swap
grace, so the cache does not grow release by release. The release asset contract
these steps rely on is described on
[the overview](overview.md#release-artifacts).
32 changes: 28 additions & 4 deletions docs-developers/meta/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ internal/importer OpenAudible / Libation export -> canonical records (ASIN ded
internal/issueform issue-form body -> canonical records + an ok/duplicate/needs-human/invalid verdict
internal/build the deterministic SQLite builder (FTS5, ASIN/ISBN indexes, added_at)
internal/serve the read-only HTTP API + ABS provider + GitHub-release poller/hot-swap
Dockerfile image: site build + metaserve + baked data
Dockerfile image: the site build + the metaserve binary - no data (see below)
.github/ issue forms + CI workflows (check, release, image, intake, ai-verify)
```

Expand All @@ -86,7 +86,7 @@ run ./cmd/<name>`.
|---|---|
| `metacheck` | Validates the whole `data/` tree - schema, id/shard agreement, referential integrity, uniqueness, chapter ordering, series positions. Prints one line per problem and exits 1 if any are found. |
| `metafmt` | Enforces canonical JSON for `data/**/*.json` (sorted keys, 2-space indent, single trailing LF). `--check` lists non-canonical files and exits 1; `--write` rewrites them. |
| `metabuild` | Compiles `data/` into the SQLite artifact (`-o meta.sqlite`). Runs the full validation first and refuses to build invalid data; `--added` dates each work from a git-history-derived list. |
| `metabuild` | Compiles `data/` into the SQLite artifact (`-o meta.sqlite`). Runs the full validation first and refuses to build invalid data. Deterministic: identical data produces an identical artifact. |
| `metaserve` | Serves the compiled artifact read-only over HTTP (and optionally the static site at `/`), hot-swapping newer GitHub releases. See [the HTTP API](api.md). |
| `metascan` | Scans a local audiobook folder into an import JSON - see [contributing data](contributing-data.md#scanning-local-files-metascan). |
| `metaimport` | Ingests an OpenAudible/Libation library export into `data/` - see [contributing data](contributing-data.md#bulk-importers-metaimport). |
Expand Down Expand Up @@ -114,8 +114,10 @@ go run ./cmd/metaserve --db meta.sqlite --addr :8080
## Release artifacts

On merge to `main`, `.github/workflows/release.yml` publishes a **dated data
release** tagged `data-vYYYY.MM.DD-<shortsha>` when data or schema changes land.
The asset contract:
release** tagged `data-vYYYY.MM.DD-<shortsha>` when data or schema changes land -
and also when the builder itself changes (`internal/build/**`, `cmd/metabuild/**`),
so an artifact-shaping change such as a new index actually reaches a published
artifact. The asset contract:

- `meta.sqlite.gz` + `meta.sqlite.gz.sha256` - the universal anchor every
consumer verifies against.
Expand All @@ -131,6 +133,28 @@ non-prerelease release carrying `meta.sqlite.gz` with the maximum `published_at`
either kind). The [`metaserve` refresh loop](api.md#serving-and-refresh) applies
the same rule.

## The published image

`ghcr.io/kodestar/audiosilo-meta` is **the site build plus the `metaserve` binary,
and no data at all**. The catalogue is not baked in: the container fetches the
newest data release at boot and hot-swaps every release after that, which keeps
image size and image build time independent of how large the catalogue grows.

The practical consequences for a deployment:

- **Give it a cache volume.** The image runs with `--poll --cache /data/cache` and
declares `/data` as a volume. A restart that finds the current release's artifact
already in the cache verifies it against the release's `meta.sqlite.sha256` and
adopts it **without downloading** - so restarts stay cheap. Without a persistent
volume every restart re-downloads the catalogue. Budget for two artifacts at
peak, one in steady state.
- **A boot with no data is a valid state, not a crash.** The server stays up and
degrades visibly if GitHub is unreachable - full detail in
[boot and degraded start](api.md#boot-and-degraded-start).
- **Probe `/healthz` for readiness only.** It reports readiness, not liveness; a
liveness probe on it will restart-loop a server that is correctly waiting out an
outage.

## How it connects to the rest of AudioSilo

audiosilo-meta is the **upstream** of a three-repo metadata seam. `metaserve`
Expand Down
Loading