diff --git a/docs-developers/architecture/cross-repo-contract.md b/docs-developers/architecture/cross-repo-contract.md index 12d75cf..fa03486 100644 --- a/docs-developers/architecture/cross-repo-contract.md +++ b/docs-developers/architecture/cross-repo-contract.md @@ -265,6 +265,27 @@ handler **and** `serverapi`, with tests on both. Changes to `pkg/match` or against them before calling it done. See [manager server integration](../manager/server-integration.md). +## 14. Community metadata: a three-repo seam + +**What couples:** enriched book metadata (description, characters, recaps, "more +in this series") originates in a fourth repo, `audiosilo-meta`, and flows through +the server to the player. + +**Upstream (`audiosilo-meta`):** `metaserve` serves the community metadata +read-only (`GET /lookup`, `/works/{id}`, `/series/{id}`) - see the +[metadata database developer pages](../meta/overview.md). +**Server:** `internal/meta` resolves a book's ASIN/ISBN against `metaserve` and +composes an enrichment envelope, returned at `GET /libraries/{id}/meta` behind an +admin off-switch and a bounded cache; the `metadata` capability on `GET /server` +reflects whether the lookup is live. +**Frontend:** the `BookMeta` envelope (hand-mirrored in `src/api/types.ts`) is +fetched by `client.bookMeta` and rendered capability-gated on the book screen. + +**A change requires:** because the server consumes `metaserve`'s response shapes, +a change to those shapes ripples audiosilo-meta -> the server's `internal/meta` -> +the player (only if the server's outward `/meta` envelope changes). Keep it +additive, same as every other seam here. + ## The wire-change checklist Every change to the wire format follows the same shape (the worked example in diff --git a/docs-developers/contributing/documentation.md b/docs-developers/contributing/documentation.md index f42d838..f74ef98 100644 --- a/docs-developers/contributing/documentation.md +++ b/docs-developers/contributing/documentation.md @@ -60,6 +60,7 @@ mappings: | Scanner, detection, metadata | [server/scanner.md](../server/scanner.md) + `/users/getting-started/organizing-your-library` | | Auth, invites, shares | [server/auth-and-security.md](../server/auth-and-security.md) + `/users/admin/users-and-invites`, `/users/admin/sharing` | | Manager features | `/users/manager/*` + [manager developer pages](../manager/overview.md) + `manager/` screenshots | +| Meta schemas, `metaserve` API, or intake tooling | [meta developer pages](../meta/overview.md) (data model, API, contributing) - and the [cross-repo contract](../architecture/cross-repo-contract.md) when the server's `/meta` envelope is affected | | Build / release / distribution | [release-pipeline.md](../architecture/release-pipeline.md) + [releasing.md](./releasing.md) | | A capability flag | [server/api/index.md](../server/api/index.md) + the feature's user page | diff --git a/docs-developers/contributing/workspace.md b/docs-developers/contributing/workspace.md index f3d69a8..f232a62 100644 --- a/docs-developers/contributing/workspace.md +++ b/docs-developers/contributing/workspace.md @@ -3,7 +3,7 @@ title: Workspace setup description: "How the multi-repo workspace is laid out, the toolchain you need, and how to run the whole stack locally." --- -AudioSilo is one product across four repositories, and they are developed +AudioSilo is one product across several repositories, and they are developed together from a single **workspace folder**. Get the layout right first - some of the build machinery assumes it. @@ -22,6 +22,9 @@ it, gitignored by the meta repo: ├── audiosilo-server/ github.com/KodeStar/audiosilo-server (Go) ├── audiosilo-frontend/ github.com/KodeStar/audiosilo-frontend (Expo / React Native) ├── audiosilo-manager/ github.com/KodeStar/audiosilo-manager (Wails desktop) +├── audiosilo-meta/ github.com/KodeStar/audiosilo-meta (community metadata DB) +├── audiosilo-sidecars/ github.com/KodeStar/audiosilo-sidecars (contributor extraction tool) +├── audiosilo-site/ github.com/KodeStar/audiosilo-site (marketing site) └── audiosilo-docs/ this documentation site (Docusaurus) ``` diff --git a/docs-developers/index.md b/docs-developers/index.md index d75b2f9..34c9643 100644 --- a/docs-developers/index.md +++ b/docs-developers/index.md @@ -137,6 +137,10 @@ The server's priorities, in order - when they conflict, the earlier one wins: - **[Desktop manager](manager/overview.md)** - the Wails app: [server integration](manager/server-integration.md), the [Audible backup pipeline](manager/audible.md), and [transfers](manager/transfers.md). +- **[Metadata database](meta/overview.md)** - `audiosilo-meta`, the + community audiobook metadata database behind meta.audiosilo.app that the server + and player enrich books from: the [data model](meta/data-model.md), the + [`metaserve` HTTP API](meta/api.md), and [contributing data](meta/contributing-data.md). - **[Contributing](contributing/workspace.md)** - the [workspace setup](contributing/workspace.md), [per-repo gates and CI](contributing/gates-and-ci.md), [how to land a cross-repo change](contributing/cross-repo-changes.md), diff --git a/docs-developers/meta/api.md b/docs-developers/meta/api.md new file mode 100644 index 0000000..9c238d9 --- /dev/null +++ b/docs-developers/meta/api.md @@ -0,0 +1,210 @@ +--- +title: Meta HTTP API +description: "The metaserve read-only JSON API reference: every /api/v1 route, the Audiobookshelf provider at /abs/search, the production release webhook, CORS behavior, and how the server refreshes its artifact from GitHub Releases." +--- + +`metaserve` (`cmd/metaserve` over `internal/serve`) is a **read-only** JSON API +over the compiled SQLite artifact. All data is public, so there is **no auth**; +every `/api/v1` route (and `/abs/search`) responds with permissive CORS +(`Access-Control-Allow-Origin: *`, `Vary: Origin`), and responses are +gzip-compressed. Cross-origin `GET`s work from any browser; there is no +preflight handling, so requests must stay CORS-simple (no custom headers). It +can also serve a static site at `/` (`--site`) and hot-swaps a newer release +artifact without a restart. + +Errors are a JSON envelope `{"error": "..."}` with the matching HTTP status. All +routes are `GET`. + +## `/healthz` + +Liveness plus a cheap freshness signal. Always 200 while a snapshot is loaded: + +```json +{ "status": "ok", "built_at": "2026-07-15T...", "works": 1234 } +``` + +## `/api/v1/stats` + +Catalogue totals, precomputed once per loaded snapshot: + +```json +{ "works": 0, "recordings": 0, "people": 0, "series": 0, + "total_runtime_min": 0, "total_chapters": 0, "built_at": "..." } +``` + +## `/api/v1/search?q=&limit=` + +Full-text search over works, people, and series. `q` is **required** (400 `q is +required` when empty). `limit` defaults to 20, clamped to `[1, 50]`. Returns +`{"results": [...]}`, best-ranked first; each result is one of three shapes +distinguished by `kind`: + +- **work**: `{kind, id, title, authors[], series, cover_url, added_at, narrators[]}` +- **person**: `{kind, id, name}` +- **series**: `{kind, id, name, works}` (`works` = member count) + +FTS input is escaped defensively (every token quoted, the final token +prefixed with `*`), so no user input can break the underlying `MATCH`. + +## `/api/v1/works/latest?limit=` + +The newest works, for the site's landing grid. `limit` defaults to 12, clamped to +`[1, 50]`. Returns `{"works": [workCard...]}` ordered by `added_at` descending +(then title), with at most two works from any one series so a bulk import sharing +one date can't fill the grid. A **workCard** is the compact shape reused across +lists and lookups: `{id, title, authors[], series, cover_url, added_at}`. + +## `/api/v1/works/{id}` + +The full work document, or 404 `work not found`. It carries the work's +identifiers, its `authors[]` and `series[]`, every `recordings[]` entry (with +narrators, ASINs, ISBNs, and a `chapter_count`), and - when the loaded artifact is +new enough and the work has them - the inline expressive layer: + +- `characters[]` - `{id, name, aliases?, role?, reveal:{chapter}, description?, xref?}` +- `recaps[]` - `{through:{chapter}, scope?, text}` +- `recap_summary` - `{in_short?, ending?}` + +All three are `omitempty` and gated on the artifact `schema_version` +(characters/recaps at 2, recap summary at 3), so an older artifact simply omits +them (see [the data model](data-model.md#the-compiled-artifact-and-schema-versioning)). + +## `/api/v1/works/{id}/recordings/{rid}/chapters` + +The chapter list for one recording of a work: `{"chapters": [{title, start_ms, +length_ms}]}`, ordered by chapter index. An unknown work/recording yields an empty +list, not a 404. + +## `/api/v1/people/{id}` + +A person plus their works, or 404 `person not found`: + +```json +{ "id": "...", "name": "...", "sort_name": "...", + "authored": [workCard...], + "narrated": [{ "work": workCard, "recording_id": "..." }] } +``` + +## `/api/v1/series/{id}` + +A series with its ordered member works, or 404 `series not found`: + +```json +{ "id": "...", "name": "...", "authors": [personRef...], + "works": [{ "position": "2.5", "work": workCard }] } +``` + +`works` is sorted by the numeric start of each `position` string (so `"1-3.5"` +sorts by 1). + +## `/api/v1/lookup?asin=|isbn=` + +Resolve one identifier to a work. At least one of `asin` / `isbn` is **required** +(400 `asin or isbn is required`); a miss is 404 `not found`. On a hit: + +```json +{ "work": workCard, "recording_id": "..." } +``` + +ASIN resolves against recording ASINs; ISBN resolves against recording ISBNs and +then falls back to a work's print ISBN (pointing at its first recording). This is +the entry point the AudioSilo server's `internal/meta` uses to enrich a book by +its `asin`/`isbn`. + +## Coverage endpoints + +These back the site's contribute page and stay small at any catalogue size. + +- **`/api/v1/coverage`** - the top-line totals only: + `{"totals": {works, with_characters?, with_recaps?, with_recap_summary?}}`. The + three sidecar counts are **omitted** (not zero) when the loaded artifact's + `schema_version` predates that dimension's table, so an unknowable count is + never reported as a misleading 0. +- **`/api/v1/coverage/works?filter=&q=&limit=&offset=`** - the paginated, + searchable per-work browser. `filter` selects the dimension - `missing` (missing + any dimension) or `has_characters` / `has_recaps` / `has_recap_summary` - and an + unknown filter is 400 `unknown filter`. `q` matches title/author; `limit` + defaults to 25, clamped to `[1, 100]`; `offset` is a non-negative row offset. The + response carries a per-filter `available` flag that is false when the dimension + is not evaluable at the artifact's schema version. +- **`/api/v1/coverage/series-gaps?q=&limit=&offset=`** - the paginated, + name-searchable list of series with interior position gaps (integer positions + absent between the lowest and highest present integer). No schema-version + dependency, so it is always available. + +## `GET /abs/search` (Audiobookshelf provider) + +`metaserve` doubles as an **Audiobookshelf custom metadata provider**. An ABS +admin configures the base URL `https://meta.audiosilo.app/abs` (no auth; ABS +v2.8.0+), and ABS appends `/search`. ABS sends `?mediaType=book&query=` +with optional `&author=` and `&isbn=`, and **never** an ASIN. + +- `query` is **required** (400 `query is required`); the endpoint **never 404s** - + a no-match is a 200 with an empty array. +- Resolution: if an ISBN is present, an exact identifier lookup runs first (the + hyphens ABS sends are stripped to the bare stored form); otherwise, or on an + ISBN miss, an FTS work search runs, with works whose authors loosely match + `author` boosted ahead of the rest (a wrong author boosts rather than filters, + so it never empties results). +- The response is `{"matches": [...]}`, **one entry per recording** (a recording is + what ABS matches a local audiobook against), capped at 10. Each match carries + `title` (the only required field) plus, when present, `subtitle`, `author`, + `narrator`, `publisher`, `publishedYear` (a string), `description`, `cover`, + `isbn`, `asin`, `series[]` (`{series, sequence}`), `language`, and `duration` + **in minutes**. `genres` and `tags` are **deliberately never returned** - the + data model does not carry publisher genres/tags. + +## Production release webhook (optional) + +When `METASERVE_WEBHOOK_SECRET` (at least 32 bytes) is set **and** `--poll` is +enabled, metaserve registers `POST /hooks/github/release`, authenticated by the +standard `X-Hub-Signature-256: sha256=...` HMAC header. `release.yml` calls it +only after every release asset has finished uploading. The request body is only a +**trigger**: metaserve re-queries GitHub and goes through the same verified +refresh path as polling, never trusting or installing data from the request body. +The endpoint is not registered when the secret is absent, and a missed delivery +is non-fatal - the fallback poller still discovers the release. + +## Flags + +`cmd/metaserve` is flag wiring only; every knob maps onto `internal/serve.Config`: + +| Flag / env | Default | Purpose | +|---|---|---| +| `--addr` | `:8080` | listen address | +| `--db` | (none) | a local `meta.sqlite` artifact to serve (dev) | +| `--site` | (none) | a static site directory to serve at `/` | +| `--poll` | `false` | fetch and hot-swap the newest data release from GitHub Releases | +| `--repo` | `KodeStar/audiosilo-meta` | GitHub `owner/name` to poll | +| `--interval` | `1h` | fallback poll interval | +| `--cache` | `./cache` | directory for downloaded artifacts | +| `GITHUB_TOKEN` (env) | (none) | raises the GitHub API rate limit | +| `METASERVE_WEBHOOK_SECRET` (env) | (none) | enables the signed release webhook (requires `--poll`) | + +With `--poll` and no `--db`, metaserve fetches the newest data release on boot so +it never starts empty. With both, the baked `--db` serves immediately and the +poller still runs one refresh at startup. + +## Serving and refresh + +The current artifact lives behind an atomic pointer (a `snapshot`); readers load +the pointer once per request. With `--poll`, a background loop plus the optional +webhook keep it current: + +- It selects the newest **data** release (the selection rule is on + [the overview](overview.md#release-artifacts)) and fetches conditionally + (`If-None-Match` / 304). +- On a new release it first tries a `--patch-from` binary delta against the + currently-loaded artifact (zstd, `--long=31` window), verifying the reconstructed + file byte-for-byte against `meta.sqlite.sha256` before installing it; it falls + back unconditionally to a full `meta.sqlite.gz` download (verified against + `meta.sqlite.gz.sha256`) whenever a patch is unavailable or fails. The first + refresh after boot is always full. +- Either way it hot-swaps the pointer; in-flight requests finish on the old + handle (closed after a grace delay). A rejected patch never swaps, and a poll + failure only logs and retries - it never crashes the process. + +The startup refresh means a recreated production container catches up to the +newest release within seconds instead of serving build-time data for a full +`--interval`. The release asset contract these steps rely on is described on +[the overview](overview.md#release-artifacts). diff --git a/docs-developers/meta/contributing-data.md b/docs-developers/meta/contributing-data.md new file mode 100644 index 0000000..3f61c01 --- /dev/null +++ b/docs-developers/meta/contributing-data.md @@ -0,0 +1,170 @@ +--- +title: Contributing data to Meta +description: "How metadata enters audiosilo-meta: the six GitHub issue forms and the intake automation that turns them into validated bot pull requests, the advisory ai-verify layer, the OpenAudible/Libation importers, metascan, the in-browser site tools, and the authoring rules." +--- + +## The rules that govern everything + +Two non-negotiable rules apply to every contribution, whichever path it takes. +They are enforced by tooling where possible and by review everywhere else (the +full policy is in the repo's `LICENSING.md` and `GOVERNANCE.md`): + +- **Facts only, never fabricated.** Contributed data must be real and verifiable. + If a fact can't be verified, the (optional) field is **omitted rather than + guessed** - no publisher blurbs, no invented ASINs, no cover files (covers are + URLs). Every record carries a `sources[]` provenance entry. +- **Own words, never copied.** Descriptions and the CC BY-SA characters/recaps + are community-authored and length-capped for the reference-guide tier; verbatim + or near-verbatim phrasing from a source is a separate publish-pipeline failure + (see [the extraction docs](#authoring-the-expressive-layer)). + +Because the GitHub repository is the database, **all writes go through GitHub** - +there are no server-side accounts. A contribution is either a direct pull request +editing `data/**`, or an issue form that the intake automation turns into one. + +## The six issue forms + +`.github/ISSUE_TEMPLATE/*.yml` are structured forms (machine-parseable field ids) +so a non-programmer can contribute without touching JSON. Each carries a +`data:<kind>` routing label that the intake workflow branches on: + +| Form | Routing label | For | +|---|---|---| +| Add a work (book) and its first recording | `data:add-work` | a new book plus its first narration | +| Add a recording (narration) | `data:add-recording` | another narration of a work already in the database | +| Correct data | `data:correction` | a single-field fix to an existing record | +| Add characters (the cast) | `data:characters` | the per-work characters sidecar (CC BY-SA) | +| Add recaps (story so far) | `data:recaps` | the per-work recaps sidecar (CC BY-SA) | +| Import a library export | `data:import` | an OpenAudible / Libation / Audiobookshelf / metascan export to bulk-import | + +## Intake automation: issue form to bot pull request + +`.github/workflows/intake.yml` converts a submitted form into a validated bot +pull request. On a data-labelled issue it runs `metaissue`, which parses the +rendered form body into canonical records (or a single-field correction, or a +placed sidecar), **deduplicates against the existing catalogue**, and emits a +machine-readable verdict the workflow branches on: + +| Verdict | Meaning | Workflow action | +|---|---|---| +| `ok` | valid new/changed records produced | opens a PR on branch `intake/issue-<n>` | +| `duplicate` | everything already exists (requires at least one skip) | labels + comments, no PR | +| `needs-human` | ambiguous - e.g. an import that produced and deduped nothing | labels for maintainer attention | +| `invalid` | the submission fails schema/validation | labels + comments with the errors | + +Two behaviors are worth knowing: + +- **Envelope sniffing.** For an import, `metaissue` sniffs a self-identifying + `audiosilo-books` envelope and routes it to that importer regardless of the + form's export-type dropdown - the file is trusted over the form. +- **The `labeled` trigger is load-bearing.** The GitHub API silently drops labels + on issues opened by non-collaborators (the sibling `audiosilo-sidecars` + contributor tool creates intake issues over the API), so such an issue arrives + label-less and the `opened`/`edited` runs skip it. When a maintainer later + applies the routing label, the `labeled` trigger admits it. The job gate + excludes the workflow's own outcome labels (`data:invalid` / `data:needs-human` + / `data:duplicate`) so outcome-labeling can't re-fire intake. + +:::note Intake runs on `issues`, not fork code +`intake.yml` triggers on the `issues` event, so there is **no fork code +execution**. The only untrusted input is the issue body and any attachment: it is +written to a file via an environment variable (never interpolated into a shell +command), parsed by `metaissue`, and never executed. Attachments are fetched +HTTPS-only from GitHub's user-attachment hosts with a size cap. The security +posture is deliberate - see [gates and CI](../contributing/gates-and-ci.md) for +the workspace-wide CI rules. +::: + +## Two validation layers on a pull request + +Every pull request touching `data/**` is checked twice: + +- **`check.yml` (mechanical, blocking).** Runs `go build`/`vet`/`test`, + `metacheck` (schema, id/shard agreement, referential integrity, uniqueness, + chapter/series rules), and `metafmt --check` (canonical JSON). A red pull + request never merges. It uses the plain `pull_request` trigger, so fork pull + requests run with a read-only token and no secrets. +- **`ai-verify.yml` (advisory, never blocking).** An AI judgement layer on top of + the mechanical check: it posts a `PASS` / `FLAG` comment and label but **never + blocks a merge**. It also uses the plain `pull_request` trigger by design (not + `pull_request_target`, which is forbidden here as the "pwn request" pattern), so + a fork pull request gets a neutral skip notice until a maintainer pushes its + branch to the repo or re-runs it - fork secrets are never reached. The diff is + passed to the model as untrusted data and never executed. + +## Bulk importers: metaimport + +`metaimport` ingests an external library export into `data/` as reviewable +records, for contributors who already have a library manager's export: + +```sh +go run ./cmd/metaimport openaudible <books.json> [--dry-run] [--date YYYY-MM-DD] +go run ./cmd/metaimport libation <export.json> [--dry-run] [--date YYYY-MM-DD] +``` + +It imports **factual fields only** (dropping publisher copy, genres, ratings, and +personal state), maps one export entry to a work + recording (+ people + series), +and **deduplicates by ASIN** against the catalogue. `--dry-run` prints the plan +without writing; a real run writes the files, then validates the whole tree and +exits non-zero if that fails. The identity rules are careful: a person slug is the +identity (name variants merge, no numbered duplicates), a work is (title slug + +author set) with per-volume disambiguation so distinct series volumes never merge, +a trailing `(Unabridged)`/`(Abridged)` marker is stripped before identity (and +seeds the recording's tri-state `abridged` when the source didn't state it), and a +same-work/same-narrator entry whose only new fact is another ASIN **merges that +ASIN into the existing recording** (guarded by runtime and abridged checks) rather +than minting a sibling. + +## Scanning local files: metascan + +`metascan` is the low-friction path when you have only audio files - no export. +It walks a folder locally and **sends nothing anywhere**, emitting an import JSON +the site's `/import` page accepts: + +```sh +go run ./cmd/metascan /path/to/audiobooks -o scan.json +``` + +Per book it gathers embedded tags (via `dhowden/tag`), the folder structure +treated as a first-class source (`Author/Book`, `Author/Series/Book`, and name +patterns like `01 - Title` or `Jack Reacher 03 - Title`), an ASIN hunted in tag +atoms and file/folder names, and - if `ffprobe` is on `PATH` - runtime and chapter +counts. Every field records where it came from (`tag` / `path` / `filename`) in +the book's `sources`, and unknown fields are omitted rather than guessed. Without +`ffprobe` the scan still works; embedded ASIN/series extraction is just more +limited (pass `-ffprobe ""` to skip it entirely). + +## The in-browser site tools + +meta.audiosilo.app hosts two client-side helpers so a contributor never has to run +Go tooling. The Developer Docs only note them; the end-user walkthrough is the +User Guide's [community metadata site page](/users/community/meta-site): + +- **[`/import`](https://meta.audiosilo.app/import)** - parses an OpenAudible, + Libation, or Audiobookshelf export, or a `metascan` folder scan, entirely in the + browser and **diffs it against the live catalogue**, so a contributor sees + exactly what is new before submitting. +- **[`/build`](https://meta.audiosilo.app/build)** - a guided builder that walks a + contributor through writing the characters or recaps sidecar for a work already + in the catalogue. + +## Authoring the expressive layer + +The CC BY-SA characters/recaps layer has its own documented process and tooling, +all at the root of the `audiosilo-meta` repository: + +- `AUTHORING.md` - the reusable authoring process for characters/recaps: + positions, the spoiler model, the copyright length caps, and the submission + checklist. +- `EXTRACTION.md` - the epub source-to-sidecar pipeline (rolling fact pass, + notes-only synthesis, adversarial spoiler audit), supported by + `metaextract split` + `ngram`. +- `EXTRACTION-AUDIO.md` - the audio-only variant (chapter-isolated ASR + + proper-noun verification), the process the `audiosilo-sidecars` tool automates. +- `GOVERNANCE.md` - the merge policy and contributor trust tiers + (schema/tooling/`.github` changes always need maintainer review via + CODEOWNERS). + +Source material and transcripts **never enter the repository** - only the derived +CC BY-SA sidecars are committed, so the near-verbatim `ngram` check is run locally +against the source you hold, by design. diff --git a/docs-developers/meta/data-model.md b/docs-developers/meta/data-model.md new file mode 100644 index 0000000..2fef18e --- /dev/null +++ b/docs-developers/meta/data-model.md @@ -0,0 +1,166 @@ +--- +title: Meta data model +description: "The audiosilo-meta entities and their on-disk layout: path-is-identity slugs and sharding, works/recordings/people/series, the characters and recaps sidecars, the position model, provenance, and the JSON Schema contract." +--- + +## Path is identity + +Every entity is addressed by a **slug** matching `^[a-z0-9]+(-[a-z0-9]+)*$` +(`common.schema.json` `$defs/slug`, max 100 chars), and its file lives in a +**shard directory** named for the first two characters of the slug. Sharding +keeps any one directory small as the catalogue grows. The slug is the identity - +there is no numeric id anywhere in the data, and the file path encodes exactly +where a record lives. + +The JSON Schemas in `schema/*.schema.json` (JSON Schema draft 2020-12, every +object `additionalProperties: false`) are the **authoritative, public contract**. +They are embedded into the tooling via `schema.go`, so a schema edit is a code +change that ships with tests. The field lists below are drawn straight from those +schemas. + +## The factual core (CC0) + +### work - `data/works/<shard>/<slug>/work.json` + +The abstract book, independent of any particular narration. + +| Field | Required | Notes | +|---|---|---| +| `id` | yes | slug | +| `title` | yes | | +| `subtitle` | no | | +| `authors` | yes | array of person slugs, at least one | +| `language` | yes | BCP-47-ish (`^[a-z]{2,3}(-[a-z0-9]{2,8})*$`) | +| `first_published` | no | `YYYY` or `YYYY-MM-DD` | +| `description` | no | community-written, never a publisher blurb | +| `xref` | no | `wikidata` (`Q\d+`), `openlibrary` (`OL\d+W`), `goodreads`, print `isbn[]` | +| `license` | yes | `CC0-1.0` | +| `sources` | yes | provenance (below) | + +### recording - `data/works/<shard>/<slug>/recordings/<rec-slug>.json` + +A specific narration/production of a work. **One work, many recordings** - the +canonical example is *Harry Potter and the Philosopher's Stone*, one work with a +Stephen Fry recording and a Jim Dale recording, each carrying its own ASINs. The +shard is the **parent work's** slug shard. + +| Field | Required | Notes | +|---|---|---| +| `id` | yes | slug | +| `work` | yes | parent work slug | +| `narrators` | yes | array of person slugs, at least one | +| `abridged` | no | **tri-state**: absence means *unknown*, so importers omit it rather than guess | +| `language` | yes | | +| `runtime_min` | no | integer minutes, > 0 | +| `release_date` | no | `YYYY`, `YYYY-MM`, or `YYYY-MM-DD` | +| `publisher` | no | | +| `asin` | no | array of `{region, asin}`; `region` is one of 11 storefronts (`us`, `uk`, `ca`, `au`, `de`, `fr`, `es`, `it`, `jp`, `in`, `br`); `asin` is `[A-Z0-9]{10}` | +| `isbn` | no | array of bare 10/13-digit ISBNs | +| `cover_url` | no | must be an `https://` URL | +| `chapters` | no | array of `{title, start_ms, length_ms}` | +| `license` | yes | `CC0-1.0` | +| `sources` | yes | | + +### person - `data/people/<shard>/<slug>.json` + +One human, shared across roles: authors on works and narrators on recordings are +the same entity type, and a person can be both. Fields: `id`, `name`, optional +`sort_name`, optional `description`, optional `xref` (`wikidata`, `openlibrary` +`OL\d+A`, `audible` ASIN), `license` (`CC0-1.0`), `sources`. + +### series - `data/series/<shard>/<slug>.json` + +A named, ordered set of works. Each entry is `{work, position}`, where +**`position` is a string** so it can express decimals and omnibus ranges: +`"1"`, `"2.5"`, or a range spanning several entries `"1-3.5"` (pattern +`^\d+(\.\d+)?(-\d+(\.\d+)?)?$`). `metacheck` enforces that **no two works share a +position** within a series. Fields: `id`, `name`, optional `authors`, `works`, +optional `xref` (`wikidata`, `goodreads`), `license` (`CC0-1.0`), `sources`. + +## The expressive layer (CC BY-SA) + +Two **per-work sidecars** carry the community-authored, spoiler-tagged content. +They are structurally separated from the core: their `license` field accepts only +`CC-BY-SA-3.0` (`$defs/license_content`). Authoring them is documented in the +repo's `AUTHORING.md` (see [contributing data](contributing-data.md)). + +### characters - `data/works/<shard>/<slug>/characters.json` + +An array of character entries under a `work` slug. Each character has: + +- `id` - unique **within the file**, not globally (two works may each have a + `bilbo-baggins`); +- `name`, optional `aliases[]`, optional `role` (`protagonist` / `antagonist` / + `supporting` / `minor`); +- `reveal` - a [position](#the-position-model), the spoiler gate: a consumer only + shows the entry once the listener has passed it (the Kindle X-Ray model); +- optional `description` - own-words, length-capped at 1500 chars (a card + without one simply has nothing to reveal); +- optional `xref` (`wikidata`, `goodreads`) - a shared `wikidata` QID links a + recurring character across a series' per-work files. + +Recurring characters are **re-described per book**, so what a reader sees stays +bounded by which book they are currently in. + +### recaps - `data/works/<shard>/<slug>/recaps.json` + +Position-keyed "story so far" summaries under a `work` slug, plus two optional +whole-book summaries. Each recap entry has: + +- `through` - a [position](#the-position-model): the recap is safe to show once + the listener has finished that chapter. **No two recaps in a file share a + `through` chapter.** +- optional `scope` (`book` / `series`) - a `chapter: 0` + `series` entry is the + "previously, in earlier books" recap; +- `text` - own-words, length-capped at 3000 chars. + +The file also carries two optional whole-book fields for a reader who has +finished the book: `in_short` (the whole arc in one paragraph, ending included, +cap 1500) and `ending` (how the book closes, stated plainly, cap 2000 - +deliberately tighter than a chaptered recap entry, a crisp sequel-handoff). + +## The position model + +Spoiler positions use one shape everywhere (`common.schema.json` `$defs/position`): + +```json +{ "chapter": 3 } +``` + +`chapter` is a non-negative integer: the **logical, edition-independent** work +chapter (1-based), where `0` means front matter or knowledge carried in from +earlier books in a series. A consumer maps its own recording-chapter timeline +onto these ordinals; text-to-audio alignment is a consumer concern, out of schema +scope. The object shape is deliberately extensible - a later `paragraph` or +`offset_ms` can be added without a breaking change. + +## Provenance on every entity + +Every record carries a `sources[]` array (`$defs/sources`, at least one entry). +Each source is `{type, ref?, imported_at?}`, where `type` is one of a fixed enum +(`user`, `openaudible-import`, `libation-import`, `audiosilo-books-import`, +`audible-lookup`, `openlibrary`, `wikidata`, `inventaire`, `community`) and +`imported_at` is `YYYY-MM-DD`. Because every fact records where it came from, a +whole source can be audited or retracted. + +## The compiled artifact and schema versioning + +`metabuild` compiles the tree into a deterministic SQLite artifact (`internal/build`), +inserting rows in sorted id order so identical data always produces an identical +file. It stamps a `schema_version` into the artifact's `meta` table, and the +expressive layer was added in later versions: + +| artifact `schema_version` | Adds | +|---|---| +| 1 | the factual core (works, recordings, people, series, FTS5 search index) | +| 2 | the `characters`, `character_aliases`, and `recaps` tables | +| 3 | the per-work `recap_summaries` table (the `in_short` / `ending` fields) | + +`metaserve` returns characters and recaps inline on `GET /works/{id}` +(`characters` / `recaps` / `recap_summary`, all `omitempty`). The serve queries +**degrade gracefully** when a newer binary briefly serves an older release: the +characters/recaps queries no-op below `schema_version` 2 and the recap summary +below 3, so a missing table reads as "no data", never a 500. The same versioning +drives the [coverage endpoints](api.md#coverage-endpoints), which omit a +dimension's count rather than report it as a misleading zero when the artifact +predates its table. diff --git a/docs-developers/meta/overview.md b/docs-developers/meta/overview.md new file mode 100644 index 0000000..317d3cf --- /dev/null +++ b/docs-developers/meta/overview.md @@ -0,0 +1,159 @@ +--- +title: Meta overview +description: "Architecture of audiosilo-meta: the GitHub repo that IS the community audiobook metadata database, validated by Go tooling, compiled to a SQLite artifact, and served read-only by metaserve to the AudioSilo player and Audiobookshelf." +--- + +## What audiosilo-meta is + +`audiosilo-meta` is the community audiobook metadata database behind +[meta.audiosilo.app](https://meta.audiosilo.app). Its defining idea: **the GitHub +repository is the database.** There is no application database and no server-side +accounts - metadata lives as plain JSON files under `data/`, one file per entity, +edited by pull request or issue form, validated by Go tooling in CI, and compiled +into a single SQLite artifact that consumers download and serve read-only. + +The reason it exists is a gap no other open database fills: none is both +**audiobook-specific** and **community-editable**. Open Library, Wikidata, and +BookBrainz are open but carry essentially no narrator, recording, or chapter +structure; Audnexus is a read-only cache over Audible's own catalogue. So +narrators, specific recordings, and chapters are **first-class data** here - the +fields every other database lacks. + +Module path: `github.com/kodestar/audiosilo-meta`. The code is **AGPL-3.0**; the +data is split across two licence layers (below). + +``` + contributor + │ pull request / issue form + ▼ + data/*.json ──► CI validation ──► meta.sqlite ──► metaserve (API + site) + (the database) (metacheck + (release read-only, hot-swaps + metafmt, Go) artifact) new releases +``` + +## The two licence layers + +Data lives in two layers, and the boundary is **enforced structurally by the JSON +Schema**, not by convention (see `LICENSING.md` in the repo): the `license` field +of a core record accepts only `CC0-1.0`, and a sidecar's only `CC-BY-SA-3.0` +(`common.schema.json` `$defs/license` vs `$defs/license_content`). + +| Layer | Entities | Licence | +|---|---|---| +| **Factual core** | works, recordings, people, series | **CC0-1.0** public-domain dedication | +| **Expressive layer** | per-work characters + recaps sidecars | **CC BY-SA 3.0** | + +Publisher blurbs and cover art are referenced, never copied: descriptions are +community-written and covers are URLs. Every entity also carries a `sources[]` +provenance array so any source can be audited or retracted wholesale. Full detail +is in [the data model](data-model.md). + +## Package layout + +The tooling follows the same discipline as the rest of the workspace: thin CLIs +under `cmd/`, all logic in reusable packages, and a clean split between the +public `pkg/*` (consumed by the sibling `audiosilo-sidecars` module as ordinary +dependencies) and the private `internal/*`. + +``` +data/ the database: works/, people/, series/ (sharded JSON) + per-work sidecars +schema/ JSON Schemas (one per entity) - the public contract, embedded via schema.go +cmd/ thin CLIs: metacheck, metafmt, metabuild, metaserve, metascan, + metaimport, metaissue, metaextract (flag wiring only) +pkg/model PUBLIC entity structs, slug/shard rules, location parsing +pkg/canonical PUBLIC canonical JSON (sorted keys, 2-space indent, trailing LF) +pkg/check PUBLIC schema validation + integrity/uniqueness/chapter/series rules +pkg/extract PUBLIC epub split + the word-shingle near-verbatim check +pkg/scan PUBLIC local folder scanner (tags + path/filename heuristics + ffprobe) +internal/importer OpenAudible / Libation export -> canonical records (ASIN dedup) +internal/issueform issue-form body -> canonical records + an ok/duplicate/needs-human/invalid verdict +internal/build the deterministic SQLite builder (FTS5, ASIN/ISBN indexes, added_at) +internal/serve the read-only HTTP API + ABS provider + GitHub-release poller/hot-swap +Dockerfile image: site build + metaserve + baked data +.github/ issue forms + CI workflows (check, release, image, intake, ai-verify) +``` + +Dependency direction mirrors the server's "transport is logic-free" rule: +`cmd/metaserve` is flag wiring only, and all business logic lives in +`internal/serve`. + +## The CLI tool set at a glance + +Each command is a thin `cmd/*` wrapper over a package; run any of them with `go +run ./cmd/<name>`. + +| Command | What it does | +|---|---| +| `metacheck` | Validates the whole `data/` tree - schema, id/shard agreement, referential integrity, uniqueness, chapter ordering, series positions. Prints one line per problem and exits 1 if any are found. | +| `metafmt` | Enforces canonical JSON for `data/**/*.json` (sorted keys, 2-space indent, single trailing LF). `--check` lists non-canonical files and exits 1; `--write` rewrites them. | +| `metabuild` | Compiles `data/` into the SQLite artifact (`-o meta.sqlite`). Runs the full validation first and refuses to build invalid data; `--added` dates each work from a git-history-derived list. | +| `metaserve` | Serves the compiled artifact read-only over HTTP (and optionally the static site at `/`), hot-swapping newer GitHub releases. See [the HTTP API](api.md). | +| `metascan` | Scans a local audiobook folder into an import JSON - see [contributing data](contributing-data.md#scanning-local-files-metascan). | +| `metaimport` | Ingests an OpenAudible/Libation library export into `data/` - see [contributing data](contributing-data.md#bulk-importers-metaimport). | +| `metaissue` | Issue-form body to canonical records + verdict, for the intake bot - see [contributing data](contributing-data.md#intake-automation-issue-form-to-bot-pull-request). | +| `metaextract` | Supports the source-to-sidecar extraction pipeline: `split` (epub -> chapter text + manifest) and `ngram` (near-verbatim overlap check against the source text). | + +## Build, validate, and serve locally + +Requires **Go 1.25+** (pure Go, no cgo, no external services). The full gate, +matching CI (`.github/workflows/check.yml`): + +```sh +go build ./... && go vet ./... && go test -race ./... && golangci-lint run +go run ./cmd/metacheck # validate the data tree +go run ./cmd/metafmt --check # canonical formatting (--write to fix) +``` + +Build the artifact and serve it: + +```sh +go run ./cmd/metabuild -o meta.sqlite +go run ./cmd/metaserve --db meta.sqlite --addr :8080 +``` + +## Release artifacts + +On merge to `main`, `.github/workflows/release.yml` publishes a **dated data +release** tagged `data-vYYYY.MM.DD-<shortsha>` when data or schema changes land. +The asset contract: + +- `meta.sqlite.gz` + `meta.sqlite.gz.sha256` - the universal anchor every + consumer verifies against. +- `meta.sqlite.sha256` - the raw-file digest, used to verify a patched artifact. +- `meta.sqlite.patch.from-<PREV_TAG>.zst` - a **best-effort** zstd `--patch-from` + binary delta against the previous data release (`--long=31`; a stock zstd CLI + consumer must also pass `--long=31` at decompression time). + +The repo also cuts code/image `v*` releases with no data assets, so consumers +select the newest **data** release by asset presence: the non-draft, +non-prerelease release carrying `meta.sqlite.gz` with the maximum `published_at` +(GitHub's release list order is not publish-chronological, and its "latest" can be +either kind). The [`metaserve` refresh loop](api.md#serving-and-refresh) applies +the same rule. + +## How it connects to the rest of AudioSilo + +audiosilo-meta is the **upstream** of a three-repo metadata seam. `metaserve` +serves the community data; the AudioSilo server composes a book's enrichment from +it; the player renders that enrichment. + +```mermaid +flowchart LR + meta["audiosilo-meta<br/>metaserve (this repo)"] + srv["audiosilo-server<br/>internal/meta"] + player["audiosilo-frontend<br/>book screen"] + meta -->|"GET /lookup, /works/{id}, /series/{id}"| srv + srv -->|"GET /libraries/{id}/meta<br/>(composed envelope)"| player +``` + +The seam of record is +[the cross-repo contract's community-metadata section](../architecture/cross-repo-contract.md#14-community-metadata-a-three-repo-seam) - +what couples, which files sit on each side, and what a `metaserve` response-shape +change ripples into. The consumer sides are documented on +[server configuration](../server/configuration.md), the server's +[HTTP API reference](../server/api/reference.md), and the frontend's +[state and data](../frontend/state-and-data.md). + +`metaserve` additionally doubles as an **Audiobookshelf custom metadata +provider** (`GET /abs/search`) for that competitor's users - see +[the HTTP API](api.md#get-abssearch-audiobookshelf-provider). diff --git a/docs-users/community/meta-site.md b/docs-users/community/meta-site.md new file mode 100644 index 0000000..5009828 --- /dev/null +++ b/docs-users/community/meta-site.md @@ -0,0 +1,118 @@ +--- +title: The community metadata site +description: Browse, check and contribute to AudioSilo Meta - the free, community-built audiobook database that powers the "About this book" extras in the player. +--- + +**AudioSilo Meta** at [meta.audiosilo.app](https://meta.audiosilo.app) is an +open, community-built database of audiobooks: the books themselves, each +narration (recording), the narrators and authors behind them, and the series +they belong to. It is the catalogue that powers the +[About this book](../listening/browsing.md#about-this-book) extras in the +player - and it is free for anyone to browse, with no account and no sign-up. + +The database treats the details other catalogues skip as first-class +information: every recording lists its **narrators**, and one book can hold many +recordings (for example a Stephen Fry reading and a Jim Dale reading), each with +its own runtime, publisher, chapters and store identifiers. + +![The AudioSilo Meta home page](/img/screenshots/meta/home.png) + +## Browsing the database + +The home page has a single search box. It searches **books, people (authors and +narrators), and series** at once, and if you paste an ASIN or ISBN it jumps +straight to the exact recording. Below the search box you'll find the latest +additions and a running count of what the community has catalogued so far. + +![Search results for a series query](/img/screenshots/meta/search.png) + +- A **book page** lists every catalogued narration of that title. Each recording + card shows its narrators, runtime, release date, publisher, any region-specific + ASINs and ISBNs (tap one to copy it), and an expandable chapter list. If the + book belongs to a series, the page links to the series and offers previous and + next buttons plus a "more in this series" row of covers. +- A **series page** lists the member books in reading order, with their positions + (including half-numbers like 2.5). +- A **person page** collects everything a narrator or author is credited on. + +![A work page with its recordings](/img/screenshots/meta/work.png) + +![A series page in reading order](/img/screenshots/meta/series.png) + +## Characters and story-so-far recaps + +On a book page, two extra tabs appear once the community has written them - the +same material the player shows under [About this book](../listening/browsing.md#about-this-book): + +- **Characters** - a card for each person in the book, written by readers in + their own words. A card's name, role and aliases are always visible, along with + the chapter where the character first appears; the description stays hidden + behind the card until you open it, so you choose when to read on. +- **Story so far** - short "what's happened up to here" recaps, each labelled + with the chapter it is safe to read after. Every recap stays closed until you + open it, so you can catch up to exactly where you've listened without spoiling + what's ahead. Some books also carry a whole-book summary and an ending recap + for readers who have finished; those are marked as full spoilers. + +![Characters and story-so-far recaps](/img/screenshots/meta/characters.png) + +## Check your own library + +The [import page](https://meta.audiosilo.app/import) tells you which of your +audiobooks are already in the database and which are new. Drop in a library +export and it sorts your books into "In the database", "New - you can contribute +these", and "Cannot auto-match". + +It accepts an export from **OpenAudible**, **Libation**, or **Audiobookshelf**, +or a scan of a plain folder of audiobooks. The page explains how to produce each +one. + +Your file is read **entirely in your browser** and never uploaded. Only the book +identifiers (ASINs and ISBNs) are sent to the database to look for a match, plus +the author names of books that didn't match, so it can tell a brand-new book from +a new narration of a book that's already listed. Personal fields - purchase +history, ratings, file paths - never leave your device. + +![Checking a library export against the database](/img/screenshots/meta/import.png) + +## Contributing + +Everything in AudioSilo Meta is added by its users, and you don't have to be a +programmer to help. Contributions are made through **guided forms** that open a +prefilled entry on the project's GitHub page; a bot then checks the submission +and files it. You need a free GitHub account to submit one. + +There are forms to add a book or a narration, to write a book's characters or its +story-so-far recaps, to send in a correction, and to submit a whole library at +once. The import page above turns each new book it finds into a one-click "add +this book" link. + +The [contribute page](https://meta.audiosilo.app/contribute) shows where help is +most needed: which books still need characters or recaps, and which series are +missing volumes. From there, a **guided builder** walks you through writing the +characters and recaps for a book you know well, checking the spoiler positions +and length limits as you go, and hands you a finished submission at the end. + +![The contribute coverage browser](/img/screenshots/meta/contribute.png) + +## How the data stays honest + +Two rules keep the catalogue clean, and they're worth knowing before you +contribute: + +- **Facts only, in your own words.** The catalogue records verifiable facts and + community-written descriptions. It never copies a publisher's blurb or text + from another site, and it stores cover art as a link rather than a copy. +- **Two open licences.** The factual catalogue - books, recordings, narrators, + series - is dedicated to the public domain (CC0), free to use in any app for + any purpose. The community-written characters and recaps are shared under + CC BY-SA (credit the source and share your own additions on the same terms). + +## Works with Audiobookshelf + +If you use Audiobookshelf, AudioSilo Meta can act as a **custom metadata +provider** for it, so your Audiobookshelf library can pull narrators, recordings, +series order and cover art from the community database. The +[Audiobookshelf page](https://meta.audiosilo.app/audiobookshelf) has the setup +steps, and also shows how to send your Audiobookshelf library back to help fill +in the catalogue. diff --git a/docs-users/faq.md b/docs-users/faq.md index 1c761db..c021c9a 100644 --- a/docs-users/faq.md +++ b/docs-users/faq.md @@ -100,6 +100,15 @@ your files at any time - but the database also holds the things that favourites, shares, and folder overrides. That's why the data directory belongs in your backups. +### Where does the extra "About this book" information come from? + +The description, characters, story-so-far recaps and series links that some +books show in the player come from **AudioSilo Meta**, a free, community-built +audiobook database at [meta.audiosilo.app](https://meta.audiosilo.app). It's +optional (your admin switches the lookup on or off), and you can browse and add +to the catalogue yourself - see +[The community metadata site](./community/meta-site.md). + ## Users & sharing ### How many users can I have, and can I limit what each one sees? diff --git a/docs-users/listening/browsing.md b/docs-users/listening/browsing.md index 5099d60..17f90ba 100644 --- a/docs-users/listening/browsing.md +++ b/docs-users/listening/browsing.md @@ -61,7 +61,7 @@ Some books show an extra section below the chapters with more about the title: - A **More in this series** row - other books in the same series, each opening its page on the metadata site so you can see what to read next. - A **View on AudioSilo Meta** link to the book's full entry. -This information comes from the **AudioSilo community metadata database** at [meta.audiosilo.app](https://meta.audiosilo.app), a free, community-run catalogue of audiobook details. It appears only when the book can be matched (it carries an ASIN or ISBN) and your server has the metadata lookup switched on. The **Characters** and **Story so far** sections are contributed by the community, so a matched book shows them only once someone has written them - many books will have the About and series details but not these yet. If a book has no extra section, it simply isn't matched or your admin has turned the feature off - nothing else on the page changes. A server admin can switch the lookup on or off at any time from the admin console's Overview section (see the [console tour](../admin/console-tour.md#community-metadata-lookup)); when it is off, no book shows this section. +This information comes from the **AudioSilo community metadata database** at [meta.audiosilo.app](https://meta.audiosilo.app), a free, community-run catalogue of audiobook details that you can also browse and contribute to yourself (see [The community metadata site](../community/meta-site.md)). It appears only when the book can be matched (it carries an ASIN or ISBN) and your server has the metadata lookup switched on. The **Characters** and **Story so far** sections are contributed by the community, so a matched book shows them only once someone has written them - many books will have the About and series details but not these yet. If a book has no extra section, it simply isn't matched or your admin has turned the feature off - nothing else on the page changes. A server admin can switch the lookup on or off at any time from the admin console's Overview section (see the [console tour](../admin/console-tour.md#community-metadata-lookup)); when it is off, no book shows this section. ## Search diff --git a/docusaurus.config.ts b/docusaurus.config.ts index bb1e6be..c0e8a36 100644 --- a/docusaurus.config.ts +++ b/docusaurus.config.ts @@ -111,6 +111,7 @@ const config: Config = { {label: 'audiosilo-server', href: 'https://github.com/KodeStar/audiosilo-server'}, {label: 'audiosilo-frontend', href: 'https://github.com/KodeStar/audiosilo-frontend'}, {label: 'audiosilo-manager', href: 'https://github.com/KodeStar/audiosilo-manager'}, + {label: 'audiosilo-meta', href: 'https://github.com/KodeStar/audiosilo-meta'}, {label: 'audiosilo-docs', href: 'https://github.com/KodeStar/audiosilo-docs'}, ], }, @@ -132,6 +133,7 @@ const config: Config = { {label: 'Server', href: 'https://github.com/KodeStar/audiosilo-server'}, {label: 'Player app', href: 'https://github.com/KodeStar/audiosilo-frontend'}, {label: 'Desktop manager', href: 'https://github.com/KodeStar/audiosilo-manager'}, + {label: 'Metadata database', href: 'https://github.com/KodeStar/audiosilo-meta'}, ], }, { diff --git a/screenshots/README.md b/screenshots/README.md index d844b40..38e80e5 100644 --- a/screenshots/README.md +++ b/screenshots/README.md @@ -18,7 +18,10 @@ npm install && npx playwright install chromium # first time only LibriVox library (cached in `.cache/library`; `MAX_FILES=3` chapter files per book keeps it ~100 MB), starts a demo-mode server on `:8790` serving the frontend's web export (plus a `--setup` instance on `:8791` for the wizard -shot), then runs the Playwright captures and backfills placeholders. +shot), builds the `audiosilo-meta` data artifact + site and starts a +`metaserve` on `:8795` for the meta-site shots, then runs the Playwright +captures and backfills placeholders. `SKIP_META=1` skips the meta stack (for a +web/admin-only run). Every capture is optimized in place with **pngquant** (`brew install pngquant`) - a lossy-palette pass that shrinks the retina PNGs ~60% with no perceptible @@ -56,6 +59,27 @@ semi-manually instead: Anything not replaced stays a labelled placeholder - visible in the docs as "regenerate me", never a broken image. +## AudioSilo Meta site captures + +The `meta/` shots come from `capture-meta.mjs`, which drives the +meta.audiosilo.app site (the sibling `audiosilo-meta` repo: an Astro static +site served same-origin by the Go `metaserve` together with its read-only +`/api/v1` JSON). `run.sh` handles it end-to-end: it builds the data artifact +(`metabuild` into `.cache/meta.sqlite`), builds the site once if +`audiosilo-meta/site/dist` is missing, and starts `metaserve` on `:8795`. +Because the site build is reused while it exists, **delete +`audiosilo-meta/site/dist` after changing the meta site's UI** so the next run +rebuilds it - otherwise the shots silently show the old UI. + +Needs: Go 1.25+, and (only for the site build) yarn + Node 24. After the caches +are warm (node_modules, Go modules, `site/dist`), a run only touches the network +for remote cover images - and the capture waits on rendered content, not on +covers, so missing cover art never fails a shot. + +`capture-meta.mjs` is standalone: point `META_BASE` at any `metaserve` (or the +live site) and run `META_BASE=http://127.0.0.1:8795 node capture-meta.mjs` - it +needs no other services (the /import fixture is vendored in `fixtures/`). + ## The store/marketing pipeline is separate `~/dev/audiosilo/store/tools` + `SCREENSHOTS.md` produce the app-store and diff --git a/screenshots/capture-admin.mjs b/screenshots/capture-admin.mjs index 13b542a..d8d0b84 100644 --- a/screenshots/capture-admin.mjs +++ b/screenshots/capture-admin.mjs @@ -7,7 +7,7 @@ // Before capturing it provisions a little demo state through the admin API // (a listener account, an invite, a share) so the console looks lived-in. import {chromium} from 'playwright'; -import {sleep, shoot, step} from './lib.mjs'; +import {sleep, shoot, step, DESKTOP_CONTEXT} from './lib.mjs'; const ORIGIN = (process.env.AS_ORIGIN || 'http://127.0.0.1:8790').replace(/\/$/, ''); const PASSWORD = process.env.ADMIN_PASSWORD; @@ -77,12 +77,7 @@ await step('provision listener + share', async () => { // ── Capture ──────────────────────────────────────────────────────────────── const browser = await chromium.launch(); -const ctx = await browser.newContext({ - viewport: {width: 1440, height: 900}, - deviceScaleFactor: 2, - colorScheme: 'dark', - reducedMotion: 'reduce', -}); +const ctx = await browser.newContext(DESKTOP_CONTEXT); const page = await ctx.newPage(); const nav = async (section) => { diff --git a/screenshots/capture-meta.mjs b/screenshots/capture-meta.mjs new file mode 100644 index 0000000..9877309 --- /dev/null +++ b/screenshots/capture-meta.mjs @@ -0,0 +1,157 @@ +// Captures the meta.audiosilo.app site documentation screenshots against a +// locally-running metaserve (run.sh builds the data artifact + site and starts +// it). Env: +// META_BASE the site/API origin (default http://127.0.0.1:8795) +// IMPORT_FIXTURE library export dropped on /import (default: the vendored +// fixtures/openaudible-books.json, synthetic data) +// +// The site is an Astro static export whose React islands fetch /api/v1 on the +// CLIENT, so every wait here is content-based (a rendered element the island +// produced), never a bare load event. Remote cover images may be slow or 404; +// steps wait on text/layout and give images a short bounded settle rather than +// blocking on them, so a missing cover never fails a shot. +// +// Standalone: point META_BASE at any metaserve (or the live site) and run +// `node capture-meta.mjs` - it needs no other services. +import path from 'node:path'; +import {fileURLToPath} from 'node:url'; +import {chromium} from 'playwright'; +import {sleep, shoot, step, DESKTOP_CONTEXT} from './lib.mjs'; + +const HERE = path.dirname(fileURLToPath(import.meta.url)); +const BASE = (process.env.META_BASE || 'http://127.0.0.1:8795').replace(/\/$/, ''); + +// The work referenced by the developer docs: two recordings (Jim Dale + Stephen +// Fry), and a fully-populated community layer (characters + a story-so-far +// recap), so the same book anchors both the work and characters shots. +const WORK_ID = 'harry-potter-and-the-philosophers-stone'; +const SERIES_ID = 'harry-potter'; +// An OpenAudible books.json sample (synthetic ASINs, vendored here) that drives +// the /import diff to a populated results view without matching real catalogue +// entries. +const IMPORT_FIXTURE = + process.env.IMPORT_FIXTURE || path.resolve(HERE, 'fixtures/openaudible-books.json'); + +const browser = await chromium.launch(); +const ctx = await browser.newContext(DESKTOP_CONTEXT); +const page = await ctx.newPage(); + +// domcontentloaded, not networkidle: a hanging remote cover request would time +// the navigation out even though the page content rendered long before. Each +// step's own content wait (+ a bounded settle for cover paint) gates the shot. +const goto = (route) => page.goto(`${BASE}${route}`, {waitUntil: 'domcontentloaded', timeout: 45000}); + +// ── Home: search hero + stats band + latest additions ─────────────────────── +await step('home', async () => { + await goto('/'); + await page.waitForSelector('input[role="combobox"]', {timeout: 20000}); + // The stats band and latest-additions grid are client:visible islands - they + // only hydrate + fetch once scrolled into view. A single jump to the bottom + // skips past them (the observer never sees the crossing), so step down one + // viewport at a time to arm each island, wait for the work-card links (proof + // the API resolved), then anchor back to the top so the shot leads with the + // search hero. + await page.evaluate(async () => { + const step = Math.round(window.innerHeight * 0.8); + for (let y = 0; y <= document.body.scrollHeight; y += step) { + window.scrollTo(0, y); + await new Promise((r) => setTimeout(r, 150)); + } + }); + await page.waitForSelector('a[href^="/work?id="]', {timeout: 20000}); + await page.evaluate(() => window.scrollTo(0, 0)); + await sleep(2000); // let the stats count-up settle + covers paint (bounded) + await shoot(page, 'meta/home.png'); +}); + +// ── Search: the grouped dropdown for a seed-data query ─────────────────────── +await step('search', async () => { + // Continues on the home page; re-navigate if the home step failed partway. + if (page.url() !== `${BASE}/`) { + await goto('/'); + await page.waitForSelector('input[role="combobox"]', {timeout: 20000}); + } + const box = page.locator('input[role="combobox"]').first(); + await box.click(); + await box.fill('harry potter'); + // Results render as role=option rows inside the listbox once the debounced + // fetch returns; wait for the first one rather than a fixed sleep. + await page.waitForSelector('[role="option"]', {timeout: 15000}); + await sleep(1200); // covers in the option rows settle + await shoot(page, 'meta/search.png'); +}); + +// ── Work detail: cover, metadata, recordings ───────────────────────────────── +await step('work', async () => { + await goto(`/work?id=${WORK_ID}`); + await page.getByRole('heading', {level: 1}).first().waitFor({state: 'visible', timeout: 20000}); + // Recordings are the client island's main payload; wait for a recording card + // (the cards only render inside the Recordings section). + await page.waitForSelector('article', {timeout: 20000}); + await sleep(1800); // cover art settle (bounded) + await shoot(page, 'meta/work.png'); +}); + +// ── Characters: the community layer with one card opened ───────────────────── +// Continues on the work page the previous step loaded (same book by design); +// re-navigates only if that step failed partway, so one transient failure +// cannot cost both shots. +await step('characters', async () => { + if (!page.url().includes(`/work?id=${WORK_ID}`)) { + await goto(`/work?id=${WORK_ID}`); + await page.getByRole('heading', {level: 1}).first().waitFor({state: 'visible', timeout: 20000}); + } + // The Characters tab only exists when the work has a characters sidecar. + const tab = page.getByRole('tab', {name: /characters/i}).first(); + await tab.waitFor({state: 'visible', timeout: 20000}); + await tab.click(); + await page.waitForSelector('#panel-characters article', {timeout: 15000}); + // Open the first character card that has a description (its disclosure button + // starts collapsed) so the shot shows the spoiler-gated body revealed. + const disclosure = page.locator('#panel-characters button[aria-expanded="false"]').first(); + await disclosure.waitFor({state: 'visible', timeout: 10000}); + await disclosure.click(); + await page.waitForSelector('#panel-characters p[id^="char-desc-"]', {timeout: 10000}); + // Bring the section (intro + grid + opened card) to the top of the viewport. + await page.locator('#panel-characters').scrollIntoViewIfNeeded().catch(() => {}); + await sleep(1200); + await shoot(page, 'meta/characters.png'); +}); + +// ── Series: the ordered list of volumes ────────────────────────────────────── +await step('series', async () => { + await goto(`/series?id=${SERIES_ID}`); + await page.getByRole('heading', {level: 1}).first().waitFor({state: 'visible', timeout: 20000}); + // Each volume is a work link in an ordered list; wait for the list to fill. + await page.waitForSelector('ol li a[href^="/work?id="]', {timeout: 20000}); + await sleep(1500); // thumbnail covers settle (bounded) + await shoot(page, 'meta/series.png'); +}); + +// ── Contribute: the coverage browser ───────────────────────────────────────── +await step('contribute', async () => { + await goto('/contribute'); + // The coverage island renders stat cards + the browser heading once /coverage + // resolves; wait for the heading text rather than a load event. + await page.getByText('Browse books by coverage', {exact: false}).first().waitFor({timeout: 20000}); + await sleep(1200); + await shoot(page, 'meta/contribute.png'); +}); + +// ── Import: drop a library export, capture the diff results ─────────────────── +await step('import', async () => { + await goto('/import'); + // The file input is visually hidden (sr-only); setInputFiles drives it + // directly, which triggers the in-browser diff against the API. + const input = page.locator('input[type="file"]').first(); + await input.waitFor({state: 'attached', timeout: 20000}); + await input.setInputFiles(IMPORT_FIXTURE); + // The diff finishes on the results stat tiles ("In the database" etc.). + await page.getByText('In the database', {exact: false}).first().waitFor({timeout: 30000}); + await sleep(1000); + await shoot(page, 'meta/import.png'); +}); + +await ctx.close(); +await browser.close(); +console.log('capture-meta: done.'); diff --git a/screenshots/fixtures/openaudible-books.json b/screenshots/fixtures/openaudible-books.json new file mode 100644 index 0000000..0f2a402 --- /dev/null +++ b/screenshots/fixtures/openaudible-books.json @@ -0,0 +1,57 @@ +[ + { + "asin": "B0SYNTH001", + "product_id": "ignored-123", + "title": "The Iron Ledger: Book One of the Ledger Wars", + "title_short": "The Iron Ledger", + "author": "Mara Quill, Devon Ashe", + "narrated_by": "Priya Lund, Tomas Reed", + "series_name": "The Ledger Wars", + "series_sequence": "1", + "duration": "12:03:40", + "seconds": 43420, + "release_date": "2025-04-15", + "publisher": "Vantage Audio", + "language": "english", + "region": "US", + "abridged": false, + "image_url": "https://covers.example.com/iron-ledger.jpg", + "description": "PUBLISHER BLURB THAT MUST NOT BE IMPORTED.", + "summary": "MORE PUBLISHER COPY.", + "genre": "Fantasy", + "rating_average": 4.6, + "chapters": [ + { "start_offset_ms": 0, "length_ms": 60000, "start_offset_sec": 0, "title": "Prologue\t" }, + { "start_offset_ms": 60000, "length_ms": 120000, "start_offset_sec": 60, "title": " Chapter One " }, + { "start_offset_ms": 180000, "length_ms": 90000, "start_offset_sec": 180, "title": "Chapter Two" } + ] + }, + { + "asin": "B0SYNTH002", + "title_short": "The Bronze Ledger", + "author": "Mara Quill, Devon Ashe", + "narrated_by": "Priya Lund, Tomas Reed", + "series_name": "The Ledger Wars", + "series_sequence": "2", + "seconds": 50000, + "release_date": "2025-09-01", + "publisher": "Vantage Audio", + "language": "english", + "region": "US", + "abridged": null, + "image_url": "https://covers.example.com/bronze-ledger.jpg" + }, + { + "asin": "B0SYNTH003", + "title_short": "Grenzland", + "author": "Ingrid Falk", + "narrated_by": "Klaus Berger", + "series_name": "The Ledger Wars", + "series_sequence": "1-3.5", + "seconds": 72000, + "release_date": "2024-02-10", + "publisher": "Nord Hörbuch", + "language": "german", + "region": "DE" + } +] diff --git a/screenshots/lib.mjs b/screenshots/lib.mjs index 84a2e63..cf0b160 100644 --- a/screenshots/lib.mjs +++ b/screenshots/lib.mjs @@ -12,6 +12,16 @@ export const CACHE = path.resolve(here, '.cache'); export const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); +// The standard desktop capture context (retina, dark, motion-reduced) shared by +// the capture scripts. capture-web builds its own parameterized contexts (phone +// profile, connect/demo variants) on purpose. +export const DESKTOP_CONTEXT = { + viewport: {width: 1440, height: 900}, + deviceScaleFactor: 2, + colorScheme: 'dark', + reducedMotion: 'reduce', +}; + // Raw Playwright PNGs at deviceScaleFactor 2 are large (a full desktop shot is // ~0.5-1.3 MB); pngquant's lossy-palette pass shrinks them ~60% with no // perceptible loss on UI captures. It's a deterministic transform run on every diff --git a/screenshots/manifest.mjs b/screenshots/manifest.mjs index eb720cc..cfdfb64 100644 --- a/screenshots/manifest.mjs +++ b/screenshots/manifest.mjs @@ -9,6 +9,7 @@ // `capture` says which script produces it: // 'web' - capture-web.mjs (web player via a warmed demo session) // 'admin' - capture-admin.mjs (admin console + public pages) +// 'meta' - capture-meta.mjs (the meta.audiosilo.app site via a local metaserve) // 'manager' - capture-manager.mjs (Wails dev server; falls back to placeholder) export const SHOTS = [ @@ -39,6 +40,15 @@ export const SHOTS = [ {file: 'server/connect-page.png', capture: 'admin', title: 'Connect page', hint: 'public connect page (auth-code box / QR)'}, {file: 'server/setup-wizard.png', capture: 'admin', title: 'Setup wizard', hint: 'first-run --setup wizard page'}, + // ── AudioSilo Meta site (meta.audiosilo.app; desktop 1440x900, dark) ──── + {file: 'meta/home.png', capture: 'meta', title: 'Meta - Home', hint: 'search hero, live stats band, latest additions grid'}, + {file: 'meta/search.png', capture: 'meta', title: 'Meta - Search', hint: 'search dropdown with grouped work/person/series results'}, + {file: 'meta/work.png', capture: 'meta', title: 'Meta - Work detail', hint: 'a work page with its cover, metadata and multiple recordings'}, + {file: 'meta/characters.png', capture: 'meta', title: 'Meta - Characters', hint: 'the community characters section, one card opened'}, + {file: 'meta/series.png', capture: 'meta', title: 'Meta - Series', hint: 'a series page listing its ordered volumes'}, + {file: 'meta/contribute.png', capture: 'meta', title: 'Meta - Contribute', hint: 'the coverage browser (what still needs characters/recaps)'}, + {file: 'meta/import.png', capture: 'meta', title: 'Meta - Import', hint: 'the in-browser library-export diff results'}, + // ── Desktop manager (Wails) ───────────────────────────────────────────── {file: 'manager/servers.png', capture: 'manager', title: 'Manager - Servers', hint: 'server list home'}, {file: 'manager/server-detail.png', capture: 'manager', title: 'Manager - Server detail', hint: 'a connected server with its libraries'}, diff --git a/screenshots/run.sh b/screenshots/run.sh index e85e25a..283f822 100755 --- a/screenshots/run.sh +++ b/screenshots/run.sh @@ -5,28 +5,36 @@ # 2. seeds a small public-domain library (LibriVox; cached in .cache/library), # 3. starts a demo-mode server (port 8790) serving the frontend's web export, # plus a second --setup instance (port 8791) for the wizard shot, -# 4. runs the Playwright captures (web player + admin console + public pages), -# 5. backfills placeholders for anything not captured (e.g. the desktop +# 4. builds the AudioSilo Meta data artifact + site and starts a local +# metaserve (port 8795) for the meta.audiosilo.app site captures, +# 5. runs the Playwright captures (web player + admin console + public pages +# + the meta site), +# 6. backfills placeholders for anything not captured (e.g. the desktop # manager on a headless run - see README.md for manager captures). # # Prereqs: Go 1.25+, Node 24, ffmpeg/ffprobe, `npm install` + # `npx playwright install chromium` in this directory, and a web export at # ../audiosilo-frontend/dist (run audiosilo-server/scripts/build-web.sh once). +# The meta section also needs yarn (for the sibling audiosilo-meta site build); +# once its caches are warm, only remote cover images touch the network. # # Env knobs: MAX_FILES (chapter files per seeded book, default 3), -# SKIP_SEED=1 (reuse the cached library as-is). +# SKIP_SEED=1 (reuse the cached library as-is), +# SKIP_META=1 (skip the meta site stack + its captures). set -euo pipefail HERE="$(cd "$(dirname "$0")" && pwd)" WORKSPACE="$(cd "$HERE/../.." && pwd)" SERVER="$WORKSPACE/audiosilo-server" FRONTEND="$WORKSPACE/audiosilo-frontend" +META="$WORKSPACE/audiosilo-meta" CACHE="$HERE/.cache" LIBRARY="$CACHE/library" DATA="$CACHE/data" SETUP_DATA="$CACHE/setup-data" PORT=8790 SETUP_PORT=8791 +META_PORT=8795 mkdir -p "$CACHE" @@ -54,9 +62,18 @@ fi cleanup() { [ -n "${MAIN_PID:-}" ] && kill "$MAIN_PID" 2>/dev/null || true [ -n "${SETUP_PID:-}" ] && kill "$SETUP_PID" 2>/dev/null || true + [ -n "${META_PID:-}" ] && kill "$META_PID" 2>/dev/null || true } trap cleanup EXIT +wait_healthy() { # <url> <logfile> <name> + for i in $(seq 1 60); do + curl -fsS "$1" >/dev/null 2>&1 && return 0 + [ "$i" = 60 ] && { echo "$3 never became healthy; see $2"; exit 1; } + sleep 1 + done +} + echo "==> starting demo server on :$PORT (fresh data dir)" rm -rf "$DATA" && mkdir -p "$DATA" cat > "$DATA/config.yaml" <<EOF @@ -83,11 +100,7 @@ AUDIOSILO_BIND="127.0.0.1:$SETUP_PORT" AUDIOSILO_TLS_MODE=off \ SETUP_PID=$! echo "==> waiting for the demo server" -for i in $(seq 1 60); do - curl -fsS "http://127.0.0.1:$PORT/healthz" >/dev/null 2>&1 && break - [ "$i" = 60 ] && { echo "server never became healthy; see $CACHE/server.log"; exit 1; } - sleep 1 -done +wait_healthy "http://127.0.0.1:$PORT/healthz" "$CACHE/server.log" "server" sleep 8 # let the startup scan index the seeded books ADMIN_PASSWORD="$(grep 'Admin password' "$CACHE/server.log" | awk -F': ' '{print $2}' | tr -d ' ')" @@ -96,7 +109,28 @@ if [ -z "$ADMIN_PASSWORD" ]; then fi SETUP_URL="$(grep -o "http://[^ ]*/setup#token=[^ ]*" "$CACHE/setup.log" | head -1 || true)" -# ── 4. Captures ───────────────────────────────────────────────────────────── +# ── 4. AudioSilo Meta site (data artifact + site build + metaserve) ───────── +if [ "${SKIP_META:-0}" != "1" ]; then + echo "==> building the meta data artifact" + (cd "$META" && go run ./cmd/metabuild -o "$CACHE/meta.sqlite") + + if [ ! -f "$META/site/dist/index.html" ]; then + echo "==> no meta site build found; building (yarn, Node 24)" + (cd "$META/site" && yarn install --frozen-lockfile && yarn build) + fi + + echo "==> starting metaserve on :$META_PORT" + (cd "$META" && go build -o bin/metaserve ./cmd/metaserve) + "$META/bin/metaserve" --db "$CACHE/meta.sqlite" \ + --site "$META/site/dist" --addr "127.0.0.1:$META_PORT" \ + > "$CACHE/metaserve.log" 2>&1 & + META_PID=$! + + echo "==> waiting for metaserve" + wait_healthy "http://127.0.0.1:$META_PORT/healthz" "$CACHE/metaserve.log" "metaserve" +fi + +# ── 5. Captures ───────────────────────────────────────────────────────────── cd "$HERE" echo "==> capturing web player" AS_BASE="http://127.0.0.1:$PORT/web/" node capture-web.mjs @@ -105,7 +139,12 @@ echo "==> capturing admin console + public pages" AS_ORIGIN="http://127.0.0.1:$PORT" ADMIN_PASSWORD="$ADMIN_PASSWORD" \ SETUP_URL="$SETUP_URL" node capture-admin.mjs -# ── 5. Backfill placeholders for anything missing ────────────────────────── +if [ "${SKIP_META:-0}" != "1" ]; then + echo "==> capturing the meta site" + META_BASE="http://127.0.0.1:$META_PORT" node capture-meta.mjs +fi + +# ── 6. Backfill placeholders for anything missing ────────────────────────── echo "==> backfilling placeholders" node placeholders.mjs diff --git a/sidebars-developers.ts b/sidebars-developers.ts index 0717a74..9893bec 100644 --- a/sidebars-developers.ts +++ b/sidebars-developers.ts @@ -55,6 +55,16 @@ const sidebars: SidebarsConfig = { 'manager/transfers', ], }, + { + type: 'category', + label: 'Metadata database (audiosilo-meta)', + items: [ + 'meta/overview', + 'meta/data-model', + 'meta/api', + 'meta/contributing-data', + ], + }, { type: 'category', label: 'Contributing', diff --git a/sidebars-users.ts b/sidebars-users.ts index 8f0f3a2..c4b2c40 100644 --- a/sidebars-users.ts +++ b/sidebars-users.ts @@ -50,6 +50,11 @@ const sidebars: SidebarsConfig = { 'manager/audible-backup', ], }, + { + type: 'category', + label: 'Community metadata', + items: ['community/meta-site'], + }, 'demo', 'troubleshooting', 'faq', diff --git a/static/img/screenshots/meta/characters.png b/static/img/screenshots/meta/characters.png new file mode 100644 index 0000000..a226638 Binary files /dev/null and b/static/img/screenshots/meta/characters.png differ diff --git a/static/img/screenshots/meta/contribute.png b/static/img/screenshots/meta/contribute.png new file mode 100644 index 0000000..4e2c86d Binary files /dev/null and b/static/img/screenshots/meta/contribute.png differ diff --git a/static/img/screenshots/meta/home.png b/static/img/screenshots/meta/home.png new file mode 100644 index 0000000..b4f9e06 Binary files /dev/null and b/static/img/screenshots/meta/home.png differ diff --git a/static/img/screenshots/meta/import.png b/static/img/screenshots/meta/import.png new file mode 100644 index 0000000..410430d Binary files /dev/null and b/static/img/screenshots/meta/import.png differ diff --git a/static/img/screenshots/meta/search.png b/static/img/screenshots/meta/search.png new file mode 100644 index 0000000..f108f68 Binary files /dev/null and b/static/img/screenshots/meta/search.png differ diff --git a/static/img/screenshots/meta/series.png b/static/img/screenshots/meta/series.png new file mode 100644 index 0000000..73de745 Binary files /dev/null and b/static/img/screenshots/meta/series.png differ diff --git a/static/img/screenshots/meta/work.png b/static/img/screenshots/meta/work.png new file mode 100644 index 0000000..02d4dc3 Binary files /dev/null and b/static/img/screenshots/meta/work.png differ