Phase 27: Watcher Service & User-Initiated Scan

.env.example

@@ -1,3 +1,17 @@
+# =====================================================================
+# Phaze .env example
+#
+# Host vs Container
+# -----------------
+# DATABASE_URL and REDIS_URL below use the DOCKER SERVICE NAMES `postgres`
+# and `redis`. That's correct when the api/worker/watcher run via
+# `docker compose up`. If you instead run any service directly on the
+# HOST via `uv run`, switch to:
+#   DATABASE_URL=postgresql+asyncpg://phaze:phaze@localhost:5432/phaze
+#   REDIS_URL=redis://localhost:6379/0
+# (or use an SSH tunnel to the remote home server).
+# =====================================================================
+
 # Database
 DATABASE_URL=postgresql+asyncpg://phaze:phaze@postgres:5432/phaze
 POSTGRES_USER=phaze
@@ -15,6 +29,64 @@ API_PORT=8000
 # File discovery - mounted music directory for scanning
 SCAN_PATH=/data/music
 
+# =====================================================================
+# Bring-up (Phase 27 UAT Gap 2 / Gap 3)
+# =====================================================================
+# Run alembic upgrade head in the api lifespan on every startup.
+# Set to false in production environments where you want to gate
+# migrations behind a manual maintenance window.
+# PHAZE_AUTO_MIGRATE=true
+#
+# On a fresh agents table, seed a single dev-agent row so the watcher
+# can authenticate on first start. Production deployments should keep
+# this `false` (default) and provision agents via the management CLI.
+# PHAZE_DEV_SEED_AGENT=false
+#
+# Optional fixed bearer for the dev-seeded agent. If unset, the api
+# generates a random one and logs it at INFO -- scrape it from
+# `docker compose logs api` and paste into PHAZE_AGENT_TOKEN below.
+# Format: phaze_agent_<32 urlsafe-base64 bytes>.
+# PHAZE_DEV_AGENT_TOKEN=
+
+# =====================================================================
+# Agent Mode (required when PHAZE_ROLE=agent)
+# =====================================================================
+# The watcher container runs with PHAZE_ROLE=agent and needs all three:
+#
+# 1. URL of the application server. When running in docker compose:
+#    http://api:8000 (service DNS). When running on the host:
+#    http://localhost:8000 or your tunnel URL.
+# PHAZE_AGENT_API_URL=http://api:8000
+#
+# 2. Bearer issued at agent registration. MUST match the sha256(token_hash)
+#    stored in the `agents` table for the calling agent. For dev bring-up
+#    on a fresh DB, copy the token logged by `docker compose logs api`
+#    after PHAZE_DEV_SEED_AGENT=true triggers ensure_dev_agent. Format:
+#    phaze_agent_<32 urlsafe-base64 bytes>.
+# PHAZE_AGENT_TOKEN=
+#
+# 3. Comma-separated list of absolute filesystem paths the agent is
+#    permitted to read/write. Used by execute_approved_batch for path-
+#    traversal containment.
+# PHAZE_AGENT_SCAN_ROOTS=/data/music
+
+# =====================================================================
+# Watcher tunables (optional -- defaults shown below)
+# =====================================================================
+# Seconds a file's mtime must be stable before the watcher posts it (D-01)
+# PHAZE_WATCHER_SETTLE_SECONDS=10
+# Stuck-file cap: entries older than this are evicted from the pending set (D-02)
+# PHAZE_WATCHER_MAX_PENDING_SECONDS=3600
+# How often the watcher's sweep task checks for settled files (D-01)
+# PHAZE_WATCHER_SWEEP_INTERVAL_SECONDS=2
+# Use watchdog's PollingObserver instead of native inotify. REQUIRED on macOS
+# docker bind mounts (rancher-desktop / Docker Desktop) where inotify events
+# do not propagate through 9p/virtiofs. Leave false in production Linux
+# deployments where inotify works natively.
+# PHAZE_WATCHER_POLLING_MODE=false
+# Number of FileUpsertRecord rows per chunk in scan_directory (D-11)
+# PHAZE_SCAN_CHUNK_SIZE=500
+
 # Docker (for volume permissions)
 UID=1000
 GID=1000

.planning/PROJECT.md

@@ -25,15 +25,16 @@ Get 200K messy music and concert files properly named, organized into logical fo
 
 ## Current State
 
-**v3.0 shipped 2026-04-04.** Cross-service intelligence and file enrichment complete.
+**v4.0 in progress.** Phases 24–27 complete; Phase 28 (Distributed Execution Dispatch) and 29 (Deployment Hardening & Agents Admin) remaining.
 
-- 8,000+ lines of Python across 23 phases, 52 plans total (v1.0-v3.0)
-- 650+ tests passing, 53/53 cumulative requirements satisfied (all milestones)
-- Tech stack: FastAPI, SQLAlchemy (async), SAQ, litellm, essentia-tensorflow, mutagen, rapidfuzz, httpx, HTMX + Tailwind
-- Docker Compose: api, worker, postgres, redis, audfprint, panako containers
-- 12 Alembic migrations, 12 SQLAlchemy models, 3 fingerprint service containers
-- Admin UI: proposals, duplicates, tracklists, pipeline dashboard, directory tree preview, unified search, Discogs linking, tag review, CUE management
-- v3.0 added: unified FTS search with faceted filtering, Discogs cross-service linking with fuzzy matching and bulk-link, format-aware tag writing with 4-layer cascade (Discogs > tracklist > metadata > filename), CUE sheet generation with Discogs REM enrichment
+- 8,000+ lines of Python across 27 phases, 56+ plans total (v1.0–v4.0 in progress)
+- 1,070 tests passing on phase-27 branch; 58/63 cumulative requirements satisfied (DIST-02, SCAN-01..04 newly satisfied in Phase 27)
+- Tech stack: FastAPI, SQLAlchemy (async), SAQ, litellm, essentia-tensorflow, mutagen, rapidfuzz, httpx, watchdog, HTMX + Tailwind
+- Docker Compose: api, worker, postgres, redis, audfprint, panako, **watcher** containers
+- 13 Alembic migrations, 13 SQLAlchemy models (Agents added in Phase 24), 3 fingerprint service containers
+- Admin UI: proposals, duplicates, tracklists, pipeline dashboard with **Trigger Scan card**, directory tree preview, unified search, Discogs linking, tag review, CUE management
+- v3.0 (shipped 2026-04-04): unified FTS search with faceted filtering, Discogs cross-service linking with fuzzy matching and bulk-link, format-aware tag writing with 4-layer cascade, CUE sheet generation with Discogs REM enrichment
+- v4.0 (in progress, Phases 24–27): Agents table + token-based auth; internal HTTP API (`/api/internal/agent/*`) with bearer auth + cross-tenant 403-before-state-machine guards; `phaze.tasks.controller` vs `phaze.tasks.agent_worker` task code split; per-agent SAQ queue (`phaze-agent-<id>`); always-on `phaze-agent-watcher` service with watchdog + settle/debounce + LIVE-sentinel ScanBatch; user-initiated `scan_directory` task with chunked HTTP upserts; admin UI to trigger scans on any agent
 
 ## Previous State
 
@@ -169,4 +170,4 @@ This document evolves at phase transitions and milestone boundaries.
 4. Update Context with current state
 
 ---
-*Last updated: 2026-05-11 starting v4.0 milestone — Distributed Agents*
+*Last updated: 2026-05-14 — Phase 27 (Watcher Service & User-Initiated Scan) complete; 4/6 v4.0 phases done*

.planning/REQUIREMENTS.md

@@ -10,7 +10,7 @@ Requirements for Distributed Agents. Each maps to roadmap phases.
 ### Topology & Boundary
 
 - [ ] **DIST-01**: The application server runs the API, UI, Postgres, Redis, and a fileless SAQ worker; it has no `SCAN_PATH` or `MODELS_PATH` filesystem mounts and cannot read or write file content
-- [ ] **DIST-02**: Each file server runs one or more agents (SAQ worker + watcher + audfprint + panako sidecars) that hold local files and execute all file-bearing work locally
+- [x] **DIST-02**: Each file server runs one or more agents (SAQ worker + watcher + audfprint + panako sidecars) that hold local files and execute all file-bearing work locally
 - [x] **DIST-03**: Each agent pulls jobs from a per-agent SAQ queue named `phaze-agent-<agent_id>` on the application server's Redis; the application server enqueues file-bound jobs onto the correct queue using `FileRecord.agent_id`
 - [ ] **DIST-04**: Agents have zero direct Postgres access; every state change (file discovered, analysis result, fingerprint, execution log, heartbeat) is an authenticated HTTPS call to `/api/internal/agent/*` on the application server
 - [ ] **DIST-05**: Every `/api/internal/agent/*` endpoint is idempotent on retry; natural keys (`(agent_id, original_path)`, `file_id`, `proposal_id`, agent-generated log UUIDs) guarantee replay safety
@@ -31,10 +31,10 @@ Requirements for Distributed Agents. Each maps to roadmap phases.
 
 ### Scan & Watcher
 
-- [ ] **SCAN-01**: The administrator can trigger a scan of a specific path on a specific agent from the admin UI; the application server enqueues `scan_directory(scan_path, batch_id)` onto the chosen agent's queue
-- [ ] **SCAN-02**: As an agent walks the scan path, it streams discovered file records to the application server in chunks (e.g., 500 records per request); the application server upserts each chunk and enqueues `extract_file_metadata` per new music/video file before the scan completes
-- [ ] **SCAN-03**: Each file server runs an always-on `phaze-agent-watcher` service that observes its configured roots with the `watchdog` library; new file events stream to the application server via the same scan-batch upsert endpoint, attributed to a per-agent sentinel `ScanBatch`
-- [ ] **SCAN-04**: The watcher waits for a file's `mtime` to be stable for a configurable settle period (default 10s) before computing SHA-256 and posting it; partial / in-progress writes are not propagated
+- [x] **SCAN-01**: The administrator can trigger a scan of a specific path on a specific agent from the admin UI; the application server enqueues `scan_directory(scan_path, batch_id)` onto the chosen agent's queue
+- [x] **SCAN-02**: As an agent walks the scan path, it streams discovered file records to the application server in chunks (e.g., 500 records per request); the application server upserts each chunk and enqueues `extract_file_metadata` per new music/video file before the scan completes
+- [x] **SCAN-03**: Each file server runs an always-on `phaze-agent-watcher` service that observes its configured roots with the `watchdog` library; new file events stream to the application server via the same scan-batch upsert endpoint, attributed to a per-agent sentinel `ScanBatch`
+- [x] **SCAN-04**: The watcher waits for a file's `mtime` to be stable for a configurable settle period (default 10s) before computing SHA-256 and posting it; partial / in-progress writes are not propagated
 
 ### Task Execution
 
@@ -95,7 +95,7 @@ Explicitly excluded. Documented to prevent scope creep.
 | Requirement | Phase | Status |
 |-------------|-------|--------|
 | DIST-01 | Phase 29 — Deployment Hardening & Agents Admin | Pending |
-| DIST-02 | Phase 27 — Watcher Service & User-Initiated Scan | Pending |
+| DIST-02 | Phase 27 — Watcher Service & User-Initiated Scan | Complete |
 | DIST-03 | Phase 26 — Task Code Reorg & HTTP-Backed Agent Worker | Complete |
 | DIST-04 | Phase 25 — Internal Agent HTTP API & Bearer Auth | Pending |
 | DIST-05 | Phase 25 — Internal Agent HTTP API & Bearer Auth | Pending |
@@ -107,10 +107,10 @@ Explicitly excluded. Documented to prevent scope creep.
 | AUTH-02 | Phase 29 — Deployment Hardening & Agents Admin | Pending |
 | AUTH-03 | Phase 29 — Deployment Hardening & Agents Admin | Pending |
 | AUTH-04 | Phase 25 — Internal Agent HTTP API & Bearer Auth | Pending |
-| SCAN-01 | Phase 27 — Watcher Service & User-Initiated Scan | Pending |
-| SCAN-02 | Phase 27 — Watcher Service & User-Initiated Scan | Pending |
-| SCAN-03 | Phase 27 — Watcher Service & User-Initiated Scan | Pending |
-| SCAN-04 | Phase 27 — Watcher Service & User-Initiated Scan | Pending |
+| SCAN-01 | Phase 27 — Watcher Service & User-Initiated Scan | Complete |
+| SCAN-02 | Phase 27 — Watcher Service & User-Initiated Scan | Complete |
+| SCAN-03 | Phase 27 — Watcher Service & User-Initiated Scan | Complete |
+| SCAN-04 | Phase 27 — Watcher Service & User-Initiated Scan | Complete |
 | TASK-01 | Phase 26 — Task Code Reorg & HTTP-Backed Agent Worker | Complete |
 | TASK-02 | Phase 26 — Task Code Reorg & HTTP-Backed Agent Worker | Complete |
 | TASK-03 | Phase 26 — Task Code Reorg & HTTP-Backed Agent Worker | Complete |

.planning/ROADMAP.md

@@ -61,7 +61,7 @@ Full details: `.planning/milestones/v3.0-ROADMAP.md`
 - [ ] **Phase 24: Schema Foundation & Agent Registry** — `agents` table, `agent_id` columns on FileRecord/ScanBatch, two-step Alembic migration with legacy backfill
 - [x] **Phase 25: Internal Agent HTTP API & Bearer Auth** — `/api/internal/agent/*` endpoints, token-hash auth middleware deriving `agent_id` from token, idempotent upserts on natural keys, rotatable tokens (completed 2026-05-12)
 - [x] **Phase 26: Task Code Reorg & HTTP-Backed Agent Worker** — split `phaze.tasks.controller` (fileless) from `phaze.tasks.agent_worker` (file-bound), `PHAZE_ROLE` env-driven startup, per-agent SAQ queue (`phaze-agent-<id>`), self-contained job payloads (completed 2026-05-12)
-- [ ] **Phase 27: Watcher Service & User-Initiated Scan** — new `phaze-agent-watcher` compose service, watchdog with mtime settle/debounce, sentinel `LIVE` ScanBatch per agent, admin-triggered scan form
+- [x] **Phase 27: Watcher Service & User-Initiated Scan** — new `phaze-agent-watcher` compose service, watchdog with mtime settle/debounce, sentinel `LIVE` ScanBatch per agent, admin-triggered scan form (completed 2026-05-13)
 - [ ] **Phase 28: Distributed Execution Dispatch** — group-by-agent approval dispatch, per-operation ExecutionLog PATCH, unified SSE progress aggregating across agents, per-agent fingerprint sidecars in execution path
 - [ ] **Phase 29: Deployment Hardening & Agents Admin** — strip `SCAN_PATH`/`MODELS_PATH` from application-server compose, self-signed HTTPS w/ internal CA, Redis `requirepass` + LAN binding, `docker-compose.agent.yml`, per-file-server model download, heartbeat + Agents admin page
 
@@ -140,7 +140,14 @@ Full details: `.planning/milestones/v3.0-ROADMAP.md`
   3. A file whose `mtime` is still changing is **not** posted; only after the configured settle period (default 10s) of stable `mtime` does the watcher compute SHA-256 and stream the record (verified by writing a file slowly and observing no early upsert)
   4. From the admin UI, an administrator can choose `(agent, scan_path)` and trigger a scan; this enqueues `scan_directory(scan_path, batch_id)` onto the chosen agent's queue and the agent streams discovered files back in chunks (e.g., 500 records per request), with `extract_file_metadata` enqueued per new music/video file before the scan completes
   5. The same upsert endpoint serves both bulk scans and per-file watcher events, and a re-walked path produces no duplicate FileRecord rows
-**Plans**: TBD
+**Plans**: 7 plans
+- [x] 27-01-PLAN.md — Foundation: watchdog dep, AgentSettings watcher knobs, _shared/agent_bootstrap refactor, test scaffolding + extended import-boundary tests (Wave 0)
+- [x] 27-02-PLAN.md — Schemas: FileUpsertChunk.batch_id, ScanBatchPatch/Response, ScanDirectoryPayload, TriggerScanForm (Wave 1)
+- [x] 27-03-PLAN.md — Endpoints: PATCH /api/internal/agent/scan-batches + batch_id resolution in POST /files + patch_scan_batch client method + main.py wiring + contract tests (Wave 2)
+- [x] 27-04-PLAN.md — Agent task: scan_directory(scan_path, batch_id) with chunking, per-chunk PATCH, terminal PATCH; registered in agent_worker.settings.functions (Wave 3)
+- [x] 27-05-PLAN.md — Watcher package: phaze.agent_watcher (Debouncer, WatcherEventHandler, Poster, __main__); 16+ unit tests covering thread bridge, stuck-file cap, OSError vanish, LIVE-sentinel resolution (Wave 3)
+- [x] 27-06-PLAN.md — Admin UI: routers/pipeline_scans.py (POST + GET progress + GET agent-roots HTMX swap), 6 partial templates, dashboard.html extension + 10 contract tests (Wave 3)
+- [x] 27-07-PLAN.md — Deployment + docs: docker-compose watcher service, .env.example knobs, per-service README, STATE.md accumulation (Wave 5)
 **UI hint**: yes
 
 ### Phase 28: Distributed Execution Dispatch
@@ -200,6 +207,6 @@ Full details: `.planning/milestones/v3.0-ROADMAP.md`
 | 24. Schema Foundation & Agent Registry | v4.0 | 0/5 | Not started | - |
 | 25. Internal Agent HTTP API & Bearer Auth | v4.0 | 8/8 | Complete    | 2026-05-12 |
 | 26. Task Code Reorg & HTTP-Backed Agent Worker | v4.0 | 13/13 | Complete   | 2026-05-12 |
-| 27. Watcher Service & User-Initiated Scan | v4.0 | 0/? | Not started | - |
+| 27. Watcher Service & User-Initiated Scan | v4.0 | 7/7 | Complete    | 2026-05-14 |
 | 28. Distributed Execution Dispatch | v4.0 | 0/? | Not started | - |
 | 29. Deployment Hardening & Agents Admin | v4.0 | 0/? | Not started | - |