audiosilo-meta

The community audiobook metadata database behind meta.audiosilo.app.

The GitHub repository is the database. Metadata lives as plain JSON pack files - many records to a file, addressed by slug range (PACK-SPEC.md) - edited by pull request or issue form, validated by Go tooling in CI, and compiled into a single SQLite artifact that servers consume.

Status: Phase 1 shipped, Phase 2 landed. Governance, schemas, the validation/build pipeline, the OpenAudible and Libation importers, the read-only API server (metaserve), the website (site/, served by metaserve at meta.audiosilo.app), the Docker image, the in-browser /import diff, the Audiobookshelf metadata-provider endpoint, and issue-form intake automation are all built. The characters/recaps CC BY-SA layer (Phase 2) is live across the schema, tooling, and data. Still open: Open Library / Wikidata crosswalk seeding. Only claim what exists.

Why this exists

No existing open database is both audiobook-specific and community-editable:

• Open Library, Wikidata and Inventaire are open (CC0) but have essentially no audiobook structure - no narrator field of substance, no chapters, no runtime.
• BookBrainz has no audiobook format and no narrator field.
• Audnexus (the backend Audiobookshelf and Plex read from) is a read-only cache over Audible's own catalogue; its own README says it exists only "during the interim of waiting for a community driven audiobook database."

So narrators, specific recordings, and chapters are treated here as first-class data - the fields every other database lacks, and the reason this project exists.

How the pipeline works

  contributor
      │  pull request  /  issue form
      ▼
  data/*.json  ──►  CI validation  ──►  meta.sqlite  ──►  API server + site
  (the database)    (metacheck +        (release          (metaserve,
                     metafmt, Go)        artifact)          read-only)

1. Contributors add or edit JSON files in data/ (by pull request, or via issue forms that become pull requests).
2. CI validates every pull request - schema, canonical formatting, referential integrity, and identifier uniqueness. A red pull request never merges.
3. On merge to main, a release workflow builds meta.sqlite and attaches it (gzipped, with a checksum) to a dated GitHub Release.
4. Servers download that artifact and serve a read-only API. All writes go through GitHub; there are no server-side accounts.

The data model

• Work - the abstract book (title, authors, language, series membership).
• Recording - a specific narration of a work (narrators, abridged flag, runtime, release date, publisher, region-scoped ASINs, ISBNs, cover URL, chapters). One work, many recordings. A production released in several marketplaces stays one recording: an ISBN may be written as {"isbn": ..., "region": ...} when the region is known, and publishers[] records the other regions' imprints beside the publisher of record.
• Person - a human shared across roles (author on works, narrator on recordings; can be both).
• Series - a named series with an ordered list of member works (string positions such as "2.5").

Harry Potter and the Philosopher's Stone is one work with two recordings - Stephen Fry and Jim Dale - each carrying its own ASINs. That is the shape this project is built around. Full details in CONTRIBUTING.md.

Repository layout

data/          the database: works/, works-community/, people/, series/ (range-packed JSON, see PACK-SPEC.md)
schema/        JSON Schemas (one per entity) - authoritative field definitions
cmd/           Go tooling: metacheck (validate), metafmt (canonicalise), metabuild (SQLite), metaserve (API server)
internal/      shared Go packages behind the tooling (build, check, serve, ...)
Dockerfile     container image: API server + static site (no baked data)
.github/       issue forms + CI workflows (check, release, image)
CONTRIBUTING.md  GOVERNANCE.md  LICENSING.md

Quickstart

Requires Go 1.25+ (pure Go, no cgo, no external services).

git clone https://github.com/kodestar/audiosilo-meta
cd audiosilo-meta

# The gate - run before opening a pull request:
go build ./... && go vet ./... && go test ./... && \
  go run ./cmd/metacheck && go run ./cmd/metafmt --check

# Build the SQLite artifact locally (validates first):
go run ./cmd/metabuild -o meta.sqlite

• metacheck - schema, referential integrity, and uniqueness validation.
• metafmt --check / --write - canonical JSON (sorted keys, 2-space indent, trailing newline) plus pack placement: --write moves an entry to the pack its slug belongs in and splits a pack that has outgrown its caps, so nobody has to work placement out by hand.
• metabuild -o meta.sqlite - compile the database into a SQLite file. A work's added_at comes from the record; one that carries none falls back to the newest sources[].imported_at.
• metamigrate - the one-off conversion of a pre-pack, file-per-record tree into the pack layout (--out <dir> to rehearse without touching the source).

Running the API server

metaserve is a read-only JSON API over the compiled artifact - FTS search, work/person/series detail, chapter lists, and ASIN/ISBN lookup. All data is public, so there is no auth; the API sends permissive CORS.

# Dev: build the artifact, then serve it locally.
go run ./cmd/metabuild -o meta.sqlite
go run ./cmd/metaserve --db meta.sqlite --addr :8080

curl localhost:8080/healthz
curl 'localhost:8080/api/v1/search?q=dragon'
curl 'localhost:8080/api/v1/works/latest'
curl 'localhost:8080/api/v1/lookup?asin=B08G9PRS1K'

Key endpoints (all under /api/v1): stats, search?q=&limit=, works/latest?limit=, works/{id}, works/{id}/recordings/{rid}/chapters, people/{id}, series/{id}, lookup?asin=|isbn=, coverage, coverage/works, coverage/series-gaps, plus /healthz.

people/{id} and series/{id} take an optional ?limit=&offset= window and report the unpaged totals (authored_total/narrated_total, works_total). They differ in default: a person's credit lists are paged (100 per page, 500 max) because they are unbounded - a corporate credit such as "Full Cast" narrates thousands of works - while a series returns every member work unless a window is asked for, because the player's series rail is composed from the full list.

The coverage endpoints back the site's contribute page. coverage returns the top-line band only - how many works carry characters/recaps/whole-book recap summaries - so it stays tiny at any catalogue size. coverage/works?filter=&q= &limit=&offset= is the paginated, searchable browser: filter is missing (missing any dimension) or has_characters/has_recaps/has_recap_summary; q matches a work's title, authors, narrators and series through the same full-text index as search (whole words and word prefixes, not mid-word substrings); the response carries a per-filter available flag that is false when the dimension is not evaluable at the artifact's schema version. coverage/series-gaps?q=&limit=&offset= is the paginated, name-searchable list of series with interior position gaps.

Flags: --db (local artifact), --site <dir> (serve a static site at /), --poll (fetch and hot-swap the latest published data release from GitHub), --repo (default KodeStar/audiosilo-meta), --interval (default 1h), --cache (download dir). With --poll and no --db - what the container image runs - the server fetches the newest data release (the newest release carrying meta.sqlite.gz; code/image v* releases are skipped) on boot, streaming it to the cache directory and verifying the published checksum before it counts. If that first fetch fails the process does not exit. It first falls back to the newest artifact already in the cache directory - what the previous container was serving - and serves that, logged as stale, until a release loads. Only with nothing cached does it come up empty (static site served, /healthz and every API route 503 with a clear message, and a Retry-After reporting the wait actually in force). Either way it retries after 30s and then backs off, doubling up to --interval, so an unreachable GitHub is a logged outage rather than a container crash loop. A restart while GitHub is reachable does not re-download what the cache already holds either: the cached artifact for the newest release is verified against that release's published checksum and adopted as is. With --db as well, the local artifact serves immediately and the poller still runs a refresh at startup (not just once per --interval). Superseded artifacts are pruned from the cache after each hot-swap. Set GITHUB_TOKEN to raise the API rate limit.

For immediate production refreshes, set METASERVE_WEBHOOK_SECRET to a random value of at least 32 bytes while keeping --poll enabled. This registers POST /hooks/github/release, authenticated with the standard X-Hub-Signature-256: sha256=... HMAC header. The release workflow calls the endpoint only after every release asset has uploaded, then metaserve re-queries GitHub, verifies the published checksums, and uses the same atomic hot-swap path as polling. The webhook never trusts or installs data from its request body. Hourly polling stays enabled as a fallback for a missed delivery.

Configure the deployment and GitHub repository with the same secret:

1. Generate a secret, for example with openssl rand -hex 32, and expose it to the container as METASERVE_WEBHOOK_SECRET.
2. Add the Actions secret METASERVE_WEBHOOK_SECRET with that value.
3. Add the Actions secret METASERVE_WEBHOOK_URL with the public endpoint, for example https://meta.audiosilo.app/hooks/github/release.

The endpoint is not registered when the deployment secret is absent. A missing workflow configuration or failed delivery is non-fatal because the fallback poller will still discover the release.

Docker

The image bundles the server and the static site (built from site/) - and no data. On boot it downloads the newest published data release into the cache volume, then accepts signed release refreshes when configured and polls as a fallback. Keeping the catalogue out of the image means a site or server change neither rebuilds nor ships the database, and the image size is independent of how large the catalogue grows.

docker build -t audiosilo-meta .
docker run -p 8080:8080 -v audiosilo-meta-cache:/data ghcr.io/kodestar/audiosilo-meta:latest

For a real deployment use the committed docker-compose.yml: loopback-only port (a TLS reverse proxy such as nginx/Ploi fronts it), a named volume for the release-download cache, an optional signed release webhook, and hourly fallback polling. The image entrypoint carries all required flags; command: appends extras (for example command: ["--interval", "15m"]). Health check endpoint: /healthz - a readiness check, so it answers 503 {"status":"starting"} until the first artifact has loaded.

The image workflow builds and pushes ghcr.io/kodestar/audiosilo-meta on a v* tag.

Scanning a local library (metascan)

metascan is the low-friction way to contribute a library when you have only audio files - no OpenAudible or Libation export. Point it at a folder and it walks the tree, gathers whatever metadata it can find, and writes a JSON file the meta.audiosilo.app import page accepts.

go run github.com/kodestar/audiosilo-meta/cmd/metascan@main /path/to/audiobooks -o scan.json

Then drop scan.json onto meta.audiosilo.app/import.

It runs entirely on your machine and sends nothing anywhere - it only reads files. What it gathers, per book:

• Embedded tags (via dhowden/tag): title, authors, narrators, and an ASIN when Libation/OpenAudible embedded one.
• The folder structure, treated as a first-class source (tags are often missing or wrong, series data especially): Author/Book, Author/Series/Book, and name patterns like 01 - Title, Book 3 - Title, Title, Book 3, and Jack Reacher 03 - Title yield author/series/position/ title.
• An ASIN - the anchor that makes a sparse book matchable - hunted in tag atoms and in file/folder names (for example Title [B076HYPQLK]).
• Runtime and chapter counts, if ffprobe is on your PATH. Without it the scan still works; those two fields are simply omitted.

Have ffprobe installed if you can: it is also the deeper tag reader. The pure Go reader covers the common title/author/narrator tags plus MP3 user frames, but several audiobook-specific fields are reachable only through ffprobe - Audible/Libation freeform MP4 atoms (ASIN and friends), m4b stream language, and various container extras. Without ffprobe, embedded series/ASIN extraction is limited (especially for m4b files); the folder-structure heuristics still work in full.

Every field records where it came from (tag / path / filename) in the book's sources map, and unknown fields are omitted rather than guessed. Grouping follows the workspace convention: a folder that directly contains audio is one book (its files are the parts), and loose files at the scan root are individual single-file books. One evidence-gated exception: when a folder's files carry mutually distinct album tags (or distinct, non-generic title tags that each match their own filename), that is a flat folder of separate single-file books - the common Series/01 - A.m4b, 02 - B.m4b layout - and each file becomes its own book, with the folder feeding the series/author heuristics. Without tag evidence the folder is always kept as one book (never split on filenames alone), and a multi-file folder with no signal either way is counted in the summary so you know where to check for collections. The JSON goes to stdout by default (or -o <file>); a human-readable summary goes to stderr. Pass -ffprobe "" to skip ffprobe enrichment.

Audiobookshelf

metaserve doubles as an Audiobookshelf custom metadata provider. An ABS admin adds it under Settings -> Item Metadata Utils -> Custom Metadata Providers with the URL https://meta.audiosilo.app/abs and no authentication (the data is public); it needs ABS v2.8.0 or newer. Once added it appears in each book's Match / Quick Match picker (it is not used by the background scanner), for book libraries only.

The endpoint is GET /abs/search. ABS sends ?mediaType=book&query=<title> (with optional &author= and &isbn=) and never an ASIN; the server returns {"matches": [...]} with one entry per recording, up to 10 - an exact ISBN lookup first, otherwise an FTS title search with loose author boosting. Each match carries title, subtitle, author, narrator, publisher, publishedYear, description, cover, ISBN, ASIN, series + sequence, language, duration (in minutes), and genres when the work carries them (values from the project's own normalized genre vocabulary - never a retailer's category tree). Tags are deliberately never returned. The /audiobookshelf site page walks through both directions (adding the provider, and exporting an ABS library into /import).

Contributing

New contributions are welcome - by direct pull request or by issue form (no JSON required). Start with CONTRIBUTING.md; the merge policy and trust tiers are in GOVERNANCE.md.

Data rules in brief: factual fields only, no publisher blurbs, covers as URLs, own-words descriptions, sources[] on every record, and a CC0 dedication for every submission.

Licensing

What	Licence
Code (tooling, schemas, CI, future server)	AGPL-3.0-only (LICENSE)
Data - factual core (works, recordings, people, series)	CC0-1.0 public domain dedication
Data - derived layer (characters and recaps)	CC BY-SA 3.0

Publisher blurbs and cover art are referenced, never copied. Full policy, including the takedown / rightsholder opt-out channel, in LICENSING.md.

Roadmap

• Phase 0 (done) - governance, schemas, CI validation, the SQLite builder, and hand-curated records that prove the pipeline (including multi-recording works).
• Phase 1 - the read-only Go API server (metaserve, FTS search + ASIN/ISBN lookup), the website (site/ - search-first landing, stats, latest additions, work/person/series pages), the Docker image, the OpenAudible and Libation importers, the public deployment (meta.audiosilo.app), the in-browser import page (/import - parses an OpenAudible export, a Libation export, an Audiobookshelf library export, or a metascan folder scan client-side and diffs it against the live catalogue), and issue-form intake automation (a data issue form becomes a validated bot pull request - see GOVERNANCE.md) have landed. Still planned: seeding from Open Library / Wikidata identifier crosswalks.
• Phase 1.5 - AudioSilo player integration - the AudioSilo server (GET /libraries/{id}/meta) and player surface enriched metadata from this database, capability-gated and behind an admin off-switch. This priority integration has landed, and the Audiobookshelf metadata-provider facade (GET /abs/search, see above) has now shipped on top of it.
• Phase 2 - community-authored characters and recaps under strict length and originality rules, in a separately-tagged CC BY-SA layer, have landed: the schema enforces the layer structurally, metacheck validates it, metabuild/metaserve ship it, and the seed tree already carries characters/recaps sidecars. Authoring guide: AUTHORING.md.
• Phase 3+ - the source-to-sidecar extraction tooling (metaextract plus the documented agent process) has landed; deeper player rendering gated by the listener's progress position is in progress.

Community: questions, contribution help, and coordination happen on the AudioSilo Discord and in GitHub Discussions.

Design basis: the workspace feasibility study (../META-FEASIBILITY.md).