Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
ebbe6e0
docs(28): capture phase context
SimplicityGuy May 14, 2026
9e7850d
docs(state): record phase 28 context session
SimplicityGuy May 14, 2026
f5e11d2
docs(28): add validation strategy
SimplicityGuy May 14, 2026
848303b
docs(28): UI design contract
SimplicityGuy May 15, 2026
8299271
docs(28): create phase plan
SimplicityGuy May 15, 2026
9cc5ab8
docs(28): revise plans per checker feedback
SimplicityGuy May 15, 2026
0f6545d
docs(28): add D-NN citation tags to plan truths
SimplicityGuy May 15, 2026
8fd7bb9
docs(28): create phase plan
SimplicityGuy May 15, 2026
fc2397e
docs(phase-28): begin phase execution
SimplicityGuy May 15, 2026
3ed23b6
test(28-01): add Wave 0 test scaffolding + failing fingerprint-locali…
SimplicityGuy May 15, 2026
814085f
feat(28-01): add fingerprint URL allow-list validator + sub_batch_ind…
SimplicityGuy May 15, 2026
c1b5918
docs(28-01): complete Wave 0 scaffolding + fingerprint locality plan
SimplicityGuy May 15, 2026
173bd89
chore: merge executor worktree (worktree-agent-a04084d9a0fd6ae03) — p…
SimplicityGuy May 15, 2026
6cffd5a
docs(phase-28): update tracking after wave 0
SimplicityGuy May 15, 2026
e17c74c
test(28-03): add failing tests for dispatch grouping + chunking (RED)
SimplicityGuy May 15, 2026
ac0052b
test(28-02): replace Wave 0 stubs with failing schema/router/client t…
SimplicityGuy May 15, 2026
0dd94e8
feat(28-03): add dispatch grouping + revoked filter + chunking helper…
SimplicityGuy May 15, 2026
3e012e0
feat(28-02): add exec-batch progress endpoint + schema + agent client…
SimplicityGuy May 15, 2026
c6d0438
docs(28-03): complete dispatch grouping + chunking plan
SimplicityGuy May 15, 2026
c4b48ef
docs(28-02): complete exec-batch progress endpoint plan
SimplicityGuy May 15, 2026
89b9c8e
chore: merge executor worktree (worktree-agent-a792158a502e8ae7b)
SimplicityGuy May 15, 2026
463b642
chore: merge executor worktree (worktree-agent-a41cdd3f0f79b379c)
SimplicityGuy May 15, 2026
b0e60e7
docs(phase-28): update tracking after wave 1
SimplicityGuy May 15, 2026
9cdc782
test(28-05): add failing tests for per-proposal progress POSTs + SAQ-…
SimplicityGuy May 15, 2026
2c07444
test(28-04): add failing tests for dispatch rewrite + SSE extension +…
SimplicityGuy May 15, 2026
a67b00a
feat(28-05): per-proposal progress POSTs + SAQ-meta retry-stable UUIDs
SimplicityGuy May 15, 2026
ec16dba
docs(28-05): summary -- agent-side per-proposal progress POSTs + SAQ-…
SimplicityGuy May 15, 2026
486f581
feat(28-04): rewrite start_execution for per-agent dispatch + extend …
SimplicityGuy May 15, 2026
cd3c106
docs(28-04): complete per-agent dispatch + SSE extension plan
SimplicityGuy May 15, 2026
4fcec39
chore: merge executor worktree (worktree-agent-a7a1d1b6992801813)
SimplicityGuy May 15, 2026
3e7e327
chore: merge executor worktree (worktree-agent-adfc88948163abb39)
SimplicityGuy May 15, 2026
df5b677
docs(phase-28): update tracking after wave 2
SimplicityGuy May 15, 2026
11a98f5
test(28-06): add failing tests for cross-FS fingerprint notice banner
SimplicityGuy May 15, 2026
ca97e30
feat(28-06): add cross-FS fingerprint notice banner + PROJECT.md cons…
SimplicityGuy May 15, 2026
a294e09
docs(28-06): add plan SUMMARY for cross-FS fingerprint notice + TASK-…
SimplicityGuy May 15, 2026
85b10bc
chore: merge executor worktree (worktree-agent-ab5754c9a18fc2d1f)
SimplicityGuy May 15, 2026
88d6a0f
docs(phase-28): mark phase 28 complete + accumulate decisions
SimplicityGuy May 15, 2026
347c462
docs(phase-28): add code review report
SimplicityGuy May 15, 2026
d51d76f
docs(phase-28): add verification report — passed
SimplicityGuy May 15, 2026
fb4515b
docs(28): ship phase 28 — PR #62
SimplicityGuy May 16, 2026
f0d48ec
test(phase-28): fill patch-coverage gaps flagged by Codecov on PR #62
SimplicityGuy May 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .planning/PROJECT.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,8 @@ Full pipeline operational: scan → analyze → propose → approve → execute.
- **Scale**: Must handle ~200K files efficiently — batch processing and parallelization required
- **Naming format**: Live sets: `{Artist} - Live @ {Venue|Event} {YYYY.MM.DD}.{ext}`, Album tracks: `{Artist} - {Track #} - {Track Title}.{ext}`

**Per-agent fingerprint indices (v4.0).** Each file server's `audfprint` and `panako` sidecars index ONLY that file server's local files. Duplicate audio content landing on different file servers will NOT cross-match. Cross-file-server fingerprint matching is XAGENT-01 (deferred to a post-v4.0 milestone). The Duplicate Resolution admin UI surfaces this constraint as an inline, per-session-dismissible banner on every page load so the operator interprets fingerprint-derived results with this scope in mind.

## Key Decisions

| Decision | Rationale | Outcome |
Expand Down
12 changes: 9 additions & 3 deletions .planning/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ Full details: `.planning/milestones/v3.0-ROADMAP.md`
- [x] **Phase 25: Internal Agent HTTP API & Bearer Auth** — `/api/internal/agent/*` endpoints, token-hash auth middleware deriving `agent_id` from token, idempotent upserts on natural keys, rotatable tokens (completed 2026-05-12)
- [x] **Phase 26: Task Code Reorg & HTTP-Backed Agent Worker** — split `phaze.tasks.controller` (fileless) from `phaze.tasks.agent_worker` (file-bound), `PHAZE_ROLE` env-driven startup, per-agent SAQ queue (`phaze-agent-<id>`), self-contained job payloads (completed 2026-05-12)
- [x] **Phase 27: Watcher Service & User-Initiated Scan** — new `phaze-agent-watcher` compose service, watchdog with mtime settle/debounce, sentinel `LIVE` ScanBatch per agent, admin-triggered scan form (completed 2026-05-13)
- [ ] **Phase 28: Distributed Execution Dispatch** — group-by-agent approval dispatch, per-operation ExecutionLog PATCH, unified SSE progress aggregating across agents, per-agent fingerprint sidecars in execution path
- [x] **Phase 28: Distributed Execution Dispatch** — group-by-agent approval dispatch, per-operation ExecutionLog PATCH, unified SSE progress aggregating across agents, per-agent fingerprint sidecars in execution path (completed 2026-05-15)
- [ ] **Phase 29: Deployment Hardening & Agents Admin** — strip `SCAN_PATH`/`MODELS_PATH` from application-server compose, self-signed HTTPS w/ internal CA, Redis `requirepass` + LAN binding, `docker-compose.agent.yml`, per-file-server model download, heartbeat + Agents admin page

## Phase Details
Expand Down Expand Up @@ -160,7 +160,13 @@ Full details: `.planning/milestones/v3.0-ROADMAP.md`
3. The application server owns the `exec:{batch_id}` Redis hash and serves SSE progress from a single aggregated key; the admin UI shows unified `total / completed / failed` counts that match the sum across all participating agents
4. The execution UI exposes a per-agent breakdown (which agent handled which sub-batch, with its own counts) for debugging without requiring database access
5. Each file server's audfprint and panako sidecars index only that file server's files; fingerprint queries during execution-adjacent flows resolve against the local sidecar and the limitation (no cross-file-server fingerprint matching) is documented in the admin UI / docs
**Plans**: TBD
**Plans**: 6 plans
- [x] 28-01-PLAN.md — Wave 0: test scaffolding + new dirs + audfprint/panako allow-list validator + sub_batch_index schema field
- [x] 28-02-PLAN.md — Wave 1: ExecBatchProgressPayload + agent_exec_batches router + main.py wiring + PhazeAgentClient.post_exec_batch_progress (contract tests)
- [x] 28-03-PLAN.md — Wave 1: execution_dispatch service (group-by-agent + revoked filter + chunking) + grouping unit tests
- [x] 28-04-PLAN.md — Wave 2: start_execution rewrite + SSE generator extension + agents_table.html + progress.html rewrite + revoked banner
- [x] 28-05-PLAN.md — Wave 2: tasks/execution.py — per-proposal terminal progress POST + SAQ-meta UUID lift (closes L6/L22) + _classify_failure_step + <step>: <reason> error_message
- [x] 28-06-PLAN.md — Wave 3: cross_fs_fingerprint_notice.html partial + duplicates/list.html inclusion + PROJECT.md Constraints paragraph + STATE.md accumulation
**UI hint**: yes

### Phase 29: Deployment Hardening & Agents Admin
Expand Down Expand Up @@ -208,5 +214,5 @@ Full details: `.planning/milestones/v3.0-ROADMAP.md`
| 25. Internal Agent HTTP API & Bearer Auth | v4.0 | 8/8 | Complete | 2026-05-12 |
| 26. Task Code Reorg & HTTP-Backed Agent Worker | v4.0 | 13/13 | Complete | 2026-05-12 |
| 27. Watcher Service & User-Initiated Scan | v4.0 | 7/7 | Complete | 2026-05-14 |
| 28. Distributed Execution Dispatch | v4.0 | 0/? | Not started | - |
| 28. Distributed Execution Dispatch | v4.0 | 6/6 | Complete | 2026-05-15 |
| 29. Deployment Hardening & Agents Admin | v4.0 | 0/? | Not started | - |
38 changes: 22 additions & 16 deletions .planning/STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,16 @@
gsd_state_version: 1.0
milestone: v4.0
milestone_name: Distributed Agents
status: ready_to_plan
stopped_at: Phase 27 merged (PR #59 → main)
last_updated: "2026-05-14T19:50:00.000Z"
last_activity: 2026-05-14 -- Phase 27 merged into main as commit 4efb4a4
status: "Phase 28 shipped — PR #62"
stopped_at: "Phase 28 shipped (PR #62)"
last_updated: "2026-05-16T03:22:43.584Z"
last_activity: 2026-05-15 -- Phase 28 shipped
progress:
total_phases: 6
completed_phases: 5
total_plans: 33
completed_plans: 33
percent: 83
completed_phases: 4
total_plans: 39
completed_plans: 34
percent: 87
---

# Project State
Expand All @@ -21,14 +21,14 @@ progress:
See: .planning/PROJECT.md (updated 2026-04-02)

**Core value:** Get 200K messy music and concert files properly named, organized, deduplicated, with rich metadata in Postgres -- human-in-the-loop approval so nothing moves without review.
**Current focus:** Phase 27watcher-service-user-initiated-scan
**Current focus:** Phase 28distributed-execution-dispatch

## Current Position

Phase: 28
Plan: Not started
Status: Ready to plan
Last activity: 2026-05-14
Phase: 28 (distributed-execution-dispatch) — EXECUTING
Plan: 1 of 6
Status: Phase 28 shipped — PR #62
Last activity: 2026-05-15 -- Phase 28 shipped

Progress: [██████████] 100%

Expand Down Expand Up @@ -112,6 +112,12 @@ Progress: [██████████] 100%
- [Phase 27-05]: Stuck-file cap = 3600s default (D-02 / T-27-05); evicted entries log WARNING but do NOT post; bounded in-memory cost. Watcher POSTs chunk-of-1 with batch_id OMITTED (not None) to trigger server-side LIVE-sentinel resolution (D-18)
- [Phase 27-06]: HTMX poll-partial halt: terminal-state markup OMITS hx-trigger AND hx-get; outerHTML swap replaces the polling element entirely (Pitfall 6); cadence = every 2s for scan progress, every 5s for stats bar. Recent Scans mini-table uses transient _agent_name / _elapsed_seconds attrs on ORM rows to avoid N+1
- [Phase 27-07]: Compose 'watcher' service lives in root docker-compose.yml; Phase 29 will move it + 'worker' to docker-compose.agent.yml; depends_on api: service_started (no healthcheck); restart: unless-stopped is the only liveness mechanism in Phase 27. Volume mount SCAN_PATH:/data/music:ro only (no MODELS_PATH/OUTPUT_PATH; watcher is fileless-write)
- [Phase 28-01]: Fingerprint URL allow-list validator (`_enforce_localhost_only` on BaseSettings) blocks non-localhost `audfprint_url`/`panako_url` at config load (D-12 / TASK-04); `ExecuteApprovedBatchPayload.sub_batch_index: int = 0` schema field (D-10) — agent worker reports which chunk of a per-agent dispatch it owns
- [Phase 28-02]: `POST /api/internal/agent/exec-batches/{batch_id}/progress` handler order is part of the spec: 401→403(cross-tenant)→404(missing hash)→403(agent not in dispatch)→Redis-SET-NX dedup→HINCRBY per D-07; sub_batch_terminal=true promotes status when `subjobs_completed == subjobs_expected`
- [Phase 28-03]: Dispatch grouping uses in-Python `defaultdict(list)` over SQL `GROUP BY ... jsonb_agg(...)` — v4.0 scale (1-5 agents × ≤10K proposals) makes the type-safe path cheaper than DB aggregation; `Agent.revoked_at.is_(None)` filter applied both in JOIN and in skipped-count query
- [Phase 28-04]: SSE payloads rendered as 3 separate Jinja partials (`dispatch_summary_inline.html`, `agents_table.html`, `progress_row_inline.html`) via `_render_partial()` helper through `Jinja2Templates.TemplateResponse(...).body.decode()` — Semgrep XSS-lint requires this over bare `Environment.get_template().render()`
- [Phase 28-05]: `_load_or_seed_uuids(job, proposals)` persists BOTH `execution_log_id` and `progress_request_id` per-proposal UUIDs in `ctx['job'].meta` via single `await job.update(meta=...)` so SAQ retries reuse them (closes L6/L22, delivers D-15); failure progress POSTs use D-16 fire-and-forget (WARNING-on-failure swallow because file ops already committed); error_message format is `"<step>: <reason>"` (D-01)
- [Phase 28-06]: `cross_fs_fingerprint_notice.html` banner is dismissible per session only (no localStorage); included on `duplicates/list.html` as first child of the `space-y-6` div above `<h1>`; PROJECT.md Constraints paragraph documents XAGENT-01 (deferred cross-file-server fingerprint matching)

### Pending Todos

Expand Down Expand Up @@ -142,6 +148,6 @@ None.

## Session Continuity

Last session: 2026-05-13T18:45:31.242Z
Stopped at: Phase 27 UI-SPEC approved
Resume file: .planning/phases/27-watcher-service-user-initiated-scan/27-UI-SPEC.md
Last session: 2026-05-15T00:12:04.513Z
Stopped at: Phase 28 shipped (PR #62)
Resume file: .planning/phases/28-distributed-execution-dispatch/28-UI-SPEC.md
Loading
Loading