Add connector sync hub for 3rd party services

Dockerfile

@@ -15,6 +15,7 @@ WORKDIR /app
 COPY --from=build /app/node_modules ./node_modules
 COPY --from=build /app/dist ./dist
 COPY migrations ./migrations
+COPY assets ./assets
 COPY package.json ./
 RUN mkdir -p /data && chown node:node /data
 USER node

README.md

@@ -51,6 +51,7 @@ DATABASE_PATH=./data/crosspoint.db PORT=8080 node dist/index.js
 | `DATABASE_PATH` | `/data/crosspoint.db` | SQLite file (parent dirs auto-created) |
 | `REGISTRATION_DISABLED` | `false` | Set `true` to lock down a private instance |
 | `AUTH_RATE_LIMIT_PER_MINUTE` | `30` | Per-IP limit on registration (0 disables) |
+| `TOKEN_ENC_KEY` | _(unset)_ | Enables connectors (Hardcover/Readwise sync). 64 hex chars, a base64 32-byte key, or a ≥32-char passphrase. Encrypts stored connector credentials at rest; unset = connectors disabled. |
 
 ## Point your reader at it
 

docs/API.md

@@ -374,6 +374,62 @@ weighted by `pace_n`, `start_date` = earliest non-zero, `finished_date` = latest
 `GET /api/v1/documents` lists them. Most clients don't need this endpoint — progress-PUT
 `metadata` capture populates the same table.
 
+### Connectors (master sync hub)
+
+Pair external services (Hardcover, Readwise, …) to the account; reading activity fans out to them
+server-side. See docs/design/sync-hub.md. All under `/api/v1`, same auth headers. Requires the
+server to have `TOKEN_ENC_KEY` set (credentials are encrypted at rest) — otherwise these endpoints
+report `encryption: "disabled"` and linking returns 403.
+
+#### GET /api/v1/connectors
+
+Lists available connectors and this account's link status.
+
+```json
+{
+  "encryption": "enabled",
+  "connectors": [
+    {"id": "hardcover", "name": "Hardcover", "tier": 1, "experimental": false,
+     "carries": ["progress", "finished"], "capabilities": {"read": false, "write": true},
+     "credential_kind": "token", "linked": true, "status": "ok", "account": "julia",
+     "queue": {"pending": 0, "dead": 0}},
+    {"id": "readwise", "name": "Readwise", "tier": 1, "experimental": false,
+     "carries": ["highlight"], "capabilities": {"read": true, "write": true},
+     "credential_kind": "token", "linked": false, "status": null, "account": null}
+  ]
+}
+```
+
+#### PUT /api/v1/connectors/{id}
+
+Link/re-link by validating and storing a credential. Body: `{"credential": { ... }}` — shape is
+connector-specific (`{"token": "..."}` for Hardcover and Readwise). The server validates against the
+service before storing; returns `400` if rejected. `{"id": "hardcover", "linked": true,
+"account": "julia"}` on success.
+
+#### DELETE /api/v1/connectors/{id}
+
+Unlink; wipes the stored credential, all matches, and queued work.
+
+#### GET /api/v1/connectors/{id}/matches
+
+Lists resolved book matches (for a review UI): `{"connector": "hardcover", "matches": [{"document",
+"external_id", "confidence", "source": "auto|manual|none", "query_used", "updated_at"}]}`.
+
+#### PUT /api/v1/connectors/{id}/matches/{document}
+
+Manually set a match (sticky — never auto-recomputed). Body `{"external_id": "42"}`, or
+`{"external_id": null}` to mark "never sync this document".
+
+#### POST /api/v1/connectors/{id}/rematch/{document}
+
+Force (re)matching now; returns the resolved match or null. Preserves manual overrides.
+
+**Matching** is server-side from the document's title/author (the EPUB metadata the firmware sends —
+so connectors need "Send Metadata" on). **Fan-out** is automatic: a progress PUT enqueues a
+progress/finished event to write-connectors that carry it; a clippings PUT enqueues highlight events
+to highlight-connectors (Readwise). A background worker delivers them with retry/backoff.
+
 ### GET /healthz
 
 Unauthenticated. `{"status": "ok", "version": "0.1.0"}`.

docs/BUILD_STATUS.md

@@ -0,0 +1,86 @@
+# Build status — connector hub (overnight session)
+
+Snapshot of the master-sync-hub build. **Nothing committed or pushed.** Everything below is in the
+working tree of `crosspoint-sync` for your review.
+
+## What's done and tested
+
+The full connector framework from `docs/design/sync-hub.md`, plus both Tier-1 connectors.
+`npm test` = **74 passing** (was 40); `npx tsc --noEmit` clean; `npm run build` clean.
+
+### Framework
+- **Encrypted secret vault** (`src/crypto/secrets.ts`) — AES-256-GCM, key from `TOKEN_ENC_KEY`
+  (hex / base64 / passphrase). Connectors disabled when unset; never stores plaintext. Tested
+  (roundtrip, tamper-detection, key formats, disabled mode).
+- **Schema** (`migrations/0003_connectors.sql`) — `connector_accounts`, `connector_matches`,
+  `connector_queue`, keyed by `connector_id`.
+- **Connector interface + registry** (`src/connectors/types.ts`, `registry.ts`) — one module per
+  service; `HttpTransport` is injectable so everything is testable without network.
+- **Matcher** (`src/connectors/matching.ts`) — server-side title/author matching: normalize,
+  strip subtitles/series, `Last, First` handling, Jaccard scoring, auto-accept only on a clear
+  winner (same-book editions OK, title collisions rejected). Filename is a last resort only.
+  Heavily unit-tested.
+- **Coalescing retry queue** (`src/connectors/queue.ts`) — one pending row per
+  (user, connector, document, coalesceKey); progress collapses to latest, highlights keyed per
+  clipping; exponential backoff, dead-letter after 8 tries. Tested.
+- **Runner/worker** (`src/connectors/runner.ts`) — resolves match (cached/manual-sticky/auto),
+  pushes, handles reauth/backoff; `startQueueWorker` drains every 15s (started in `index.ts` only
+  when `TOKEN_ENC_KEY` is set).
+- **Fan-out** (`src/connectors/fanout.ts`) — progress PUT → progress/finished events; clippings
+  PUT → highlight events. Wired into both kosync and v1 progress routes and the clippings route.
+  Best-effort, never blocks the request.
+- **Management API** (`src/routes/v1/connectors.ts`) — list / link / unlink / list-matches /
+  manual-match / rematch. Documented in `docs/API.md`. Endpoint + fan-out tests in
+  `test/connectors.test.ts` (fake transport).
+
+### Connectors
+- **Hardcover** (`src/connectors/hardcover.ts`, Tier 1) — token validate, search-based match,
+  status mutation (reading/read). Write-only, carries progress+finished.
+- **Readwise** (`src/connectors/readwise.ts`, Tier 1) — token validate, highlight push (fan-out),
+  and `exportHighlights()` for the fan-in "aggregator hop" (Kindle highlights via Readwise).
+  Carries highlights.
+
+## ⚠️ Live-verify gates before enabling in production
+
+Both connectors are built against **documented** API shapes but not verified against live accounts
+(I have no tokens, and shouldn't hit third-party APIs from here). Search the code for `GATE`.
+
+- **Hardcover** (beta API — highest risk): confirm against the live GraphQL explorer
+  (hardcover.app/account/api): the `me` query shape, `search` result shape (see `extractSearchHits`,
+  which defensively handles a few shapes), the `insert_user_book` mutation name/args, and the
+  **status ids** (`STATUS_READING`/`STATUS_READ` are guesses). Wire a recorded-fixture test once
+  confirmed so schema drift breaks CI.
+- **Readwise** (stable public API — lower risk): confirm `POST /api/v2/highlights/` field names and
+  the `GET /api/v2/export/` cursor field (`nextPageCursor`). Mind the ~20 req/min limit on
+  create/export.
+
+## Not built (deliberately deferred)
+
+- **Fan-in wiring** — `exportHighlights()` exists but isn't hooked into a canonical-clippings
+  importer or a poll loop yet. That's the next chunk if you want Kindle-via-Readwise highlights
+  landing in the clippings store / on-device.
+- **Tier 2/3 connectors** (Goodreads/StoryGraph cookie-replay, Kindle) — framework supports them
+  (`credentialKind: 'cookies'`, `experimental` flag) but none implemented; they need the CSRF
+  handshake + live capture work described in sync-hub.md, behind an experimental opt-in.
+- **Web UI** — token paste / OAuth / match-review screen. The API is UI-ready; the UI is the
+  natural next project and the real prerequisite for non-technical pairing.
+- **Bidirectional/fan-in merge rules** beyond the design notes.
+
+## To run locally with connectors on
+
+```sh
+export TOKEN_ENC_KEY=$(openssl rand -hex 32)
+DATABASE_PATH=./data/dev.db npm run dev
+# GET /api/v1/connectors shows encryption: "enabled" and both connectors unlinked
+```
+
+Railway note: to enable connectors in prod, set `TOKEN_ENC_KEY` as a service variable (generate
+once, keep it stable — rotating it invalidates all stored connector credentials).
+
+## Suggested review order
+
+1. `docs/design/sync-hub.md` (the plan) → `src/connectors/types.ts` (the shape).
+2. `matching.ts` + `test/matching.test.ts` (the only tricky pure logic).
+3. `runner.ts` + `queue.ts` + `test/queue.test.ts` (delivery semantics).
+4. `hardcover.ts` / `readwise.ts` (check the GATEs against live docs).
+5. `test/connectors.test.ts` (end-to-end link → sync → fan-out with a fake API).

docs/design/hardcover-sync.md

@@ -0,0 +1,175 @@
+# Design: Hardcover Sync Connector
+
+Status: **draft / not implemented**
+
+Forward reading activity from crosspoint-sync to [Hardcover](https://hardcover.app) so a user's
+shelf reflects what they read on their e-reader — automatically. The connector lives entirely
+server-side: devices keep speaking plain kosync and never talk to Hardcover.
+
+## Goals
+
+- When a user reads on any synced device, their Hardcover profile updates: book moves to
+  "Currently Reading" on first progress, progress percentage stays current, and the book flips to
+  "Read" (with a finish date) on completion.
+- Zero firmware changes. Works for CrossPoint, CrossInk, and stock KOReader (once it ships the
+  metadata PR) alike.
+- Hardcover being slow, down, or rate-limiting must never affect device sync.
+
+Non-goals (v1): syncing clippings/notes to Hardcover, importing Hardcover state back to devices,
+StoryGraph (no public API), ratings/reviews.
+
+## Architecture
+
+```
+device --kosync PUT--> crosspoint-sync --enqueue--> forwarder --GraphQL--> Hardcover
+                            |                          ^
+                            +-- matcher (title/author search, once per document)
+```
+
+Three pieces, all in this codebase:
+
+1. **Token link** — user attaches their Hardcover API token to their account.
+2. **Matcher** — resolves our opaque document hash to a Hardcover book, using the
+   title/author/filename metadata we already capture. Runs once per `(user, document)`, result
+   cached; re-runs when metadata improves or the user overrides.
+3. **Forwarder** — turns progress events into Hardcover GraphQL mutations, with a retry queue.
+
+### Why matching is server-side (decision)
+
+Title/author already arrive via the progress-PUT `metadata` object, so every existing client feeds
+the matcher with no firmware release. Hardcover's search API (Typesense-backed) handles the fuzzy
+matching; we only rank/accept results. Server-side logic can be improved and re-run against
+historical documents at any time. Firmware-extracted ISBN was considered and rejected for the
+critical path: it needs a firmware release to exist, and `dc:identifier` is missing or wrong in
+enough real-world EPUBs (DRM-stripped, self-published, Calibre-converted) that we'd need the
+fuzzy path anyway. ISBN remains a possible future confidence booster.
+
+## Hardcover API notes
+
+- Endpoint: `https://api.hardcover.app/v1/graphql`, header `Authorization: Bearer <token>`.
+- Tokens are user-generated (hardcover.app/account/api) and currently expire ~yearly; the API is
+  officially beta. Treat every request as fallible and every schema detail as subject to change.
+- **Implementation gate:** before building the forwarder, verify against the live schema (the API
+  has a GraphQL explorer): search query shape, `user_books` status ids
+  (want-to-read / currently-reading / read), and the exact mutations for inserting/updating a
+  user book and its progress (book-level vs edition-level, pages vs percentage). Encode those in
+  one `src/connectors/hardcover.ts` module with a recorded-fixture test so schema drift breaks CI,
+  not production.
+- Rate limits are modest (beta). The forwarder must coalesce: at most one progress mutation per
+  (user, book) per N minutes (default 15), always keeping only the latest value. Reading sessions
+  on e-ink produce sparse syncs anyway (sync is user-initiated over WiFi), so this is cheap.
+
+## Schema additions
+
+```sql
+CREATE TABLE hardcover_accounts (
+  user_id     INTEGER PRIMARY KEY REFERENCES users(id),
+  token_enc   TEXT NOT NULL,          -- encrypted at rest (see Security)
+  enabled     INTEGER NOT NULL DEFAULT 1,
+  created_at  INTEGER NOT NULL,
+  updated_at  INTEGER NOT NULL
+);
+
+CREATE TABLE hardcover_matches (
+  user_id       INTEGER NOT NULL REFERENCES users(id),
+  document      TEXT NOT NULL,
+  book_id       INTEGER,              -- Hardcover book id (NULL = unmatched)
+  edition_id    INTEGER,              -- optional, when confidently known
+  confidence    REAL NOT NULL DEFAULT 0,
+  source        TEXT NOT NULL,        -- 'auto' | 'manual' | 'none'
+  query_used    TEXT,                 -- what we searched, for debugging/review UI
+  updated_at    INTEGER NOT NULL,
+  PRIMARY KEY (user_id, document)
+);
+
+CREATE TABLE hardcover_queue (
+  id          INTEGER PRIMARY KEY,
+  user_id     INTEGER NOT NULL REFERENCES users(id),
+  document    TEXT NOT NULL,
+  kind        TEXT NOT NULL,          -- 'progress' | 'finished'
+  payload     TEXT NOT NULL,          -- JSON: percentage, timestamp, finished_date...
+  attempts    INTEGER NOT NULL DEFAULT 0,
+  next_try_at INTEGER NOT NULL,
+  UNIQUE (user_id, document, kind)    -- coalesce: newest payload replaces
+);
+```
+
+## API additions (same x-auth headers)
+
+- `PUT /api/v1/connectors/hardcover` — `{token: "..."}`; server validates it with a cheap `me`
+  query before storing. `DELETE` unlinks and wipes the queue.
+- `GET /api/v1/connectors/hardcover` — `{enabled, linked_at, matched: n, unmatched: n}`.
+- `GET /api/v1/connectors/hardcover/matches` + `PUT .../matches/:document` — list and manually
+  set/override matches. Powers the future web-UI "review matches" screen; usable via curl until
+  then. Manual matches are sticky (`source = 'manual'`, never auto re-matched).
+
+Token entry realistically requires a browser, not an e-ink keyboard — this lands with (or just
+before) the first web UI. The endpoints don't depend on the UI, though.
+
+## Matching algorithm
+
+Input: the EPUB's **`documents.title` + `documents.author`** — the real signal, which the firmware
+already extracts and sends in the progress `metadata` object. This is the primary/expected path.
+Filename is only a last resort for a title-less EPUB, and we don't guess `Title - Author` vs
+`Author - Title` order (drop separators, use the whole string as a fuzzy query). Filename can't
+rescue the no-metadata case — title/author and filename ship in the *same* metadata object — so
+**matching effectively requires "Send Metadata" on**, which is the reason to default that toggle on
+for connector users.
+
+1. Normalize: strip subtitle after `:`, series suffixes like `(Book 2)`, diacritics folded,
+   `Last, First` → `First Last`.
+2. Query Hardcover search with `title author`.
+3. Score the top results: normalized-title similarity + author-name overlap. Accept the top hit
+   when it clears a threshold **and** clearly beats the runner-up (title collisions like
+   "Circe" resolve on author; same-title-same-author editions are all correct at book level, so
+   ambiguity between them is acceptable — take the most popular edition).
+4. Store the result either way (`source: 'auto'` or `'none'` with the query kept for review).
+
+Re-match triggers: metadata for the document changes (e.g. the user flips Send Metadata on after
+the fact), or a manual re-match request. Never re-match over `source = 'manual'`.
+
+Documents with no metadata at all simply stay unmatched — which is also the user-facing nudge to
+enable the Send Metadata toggle.
+
+## Forwarding rules
+
+On progress PUT for a linked user with a matched document, upsert into the queue:
+
+- percentage > 0 → `progress` event: ensure user_book exists with status currently-reading,
+  update progress percentage. First event for a book is what shelves it.
+- completion → `finished` event: status read + finish date. Completion = percentage ≥ 0.98, or —
+  once CrossInk stats sync ships — `isCompleted` / non-zero `finished_date` from
+  `stats_device_book` (the stronger signal wins; stats-based finish dates are real dates, the
+  percentage heuristic uses the sync timestamp).
+
+A small worker drains the queue (setInterval in the Node process — no external infra), with
+exponential backoff on failure and a dead-letter state after ~a week of retries. `401` from
+Hardcover (expired token) disables the link and surfaces in the status endpoint rather than
+retrying forever.
+
+Kosync GET/downstream sync is never blocked or delayed by any of this.
+
+## Security
+
+- Hardcover tokens are bearer credentials to a third-party account: encrypt at rest
+  (AES-256-GCM with a key from a `TOKEN_ENC_KEY` env var), never log them, redact in errors.
+  Self-hosters without `TOKEN_ENC_KEY` set get the connector disabled, not plaintext storage.
+- Outbound requests go only to the Hardcover endpoint; no user-controlled URLs (no SSRF surface).
+
+## Rollout
+
+1. Schema + token link + matcher, with `GET .../matches` for inspection. Verify match quality on
+   real libraries via curl before any forwarding exists.
+2. Forwarder for progress + finished, behind per-user `enabled`.
+3. Web UI: token entry + review-matches screen.
+4. Later: ISBN confidence boost, richer status mapping (owned/DNF), maybe Bookwyrm/Calibre-Web
+   via the same connector pattern.
+
+## Open questions
+
+- Exact Hardcover mutation set and status ids (resolve at the implementation gate above).
+- Should percentage-completion threshold be user-configurable? (Default 0.98; back-matter skews
+  short books.)
+- Multi-device: progress forwarding uses the newest-across-devices row (same rule as kosync GET),
+  so no per-device fan-out — confirm that's the desired UX for people who read the same book on
+  two devices at different points.