feat: v0.9 formulas + polish

docs/plans/2026-04-06-v09-formulas-polish.md

@@ -0,0 +1,172 @@
+# v0.9 — Formulas + Polish
+
+## Context
+
+All four roles are wired (concierge v0.5, architect v0.4, expert v0.2, researcher v0.8). The architect currently sends tasks one at a time via `send_task`. Recurring multi-step patterns (feature implementation, bug triage, code review) require manually re-creating the same dependency graph each time. Formulas codify these patterns as reusable TOML templates that the architect instantiates in one call, bulk-creating tasks with correct dependency edges. The daemon's existing taskboard + `EvaluateDeps` handles the rest.
+
+Additionally: config changes currently require daemon restart, and there's no hardening around partial writes for config files.
+
+## Phase 1: Formula Parsing — `internal/formula/`
+
+New package. Minimal structs, TOML parsing, DAG validation.
+
+### Files
+
+- **`internal/formula/formula.go`** — `Formula`, `Step` structs + `Load(path)` + `LoadAll(dir)` + `Validate()`
+- **`internal/formula/formula_test.go`** — happy path, cycle detection, duplicate IDs, missing depends_on refs, empty steps
+
+### Design
+
+```go
+type Step struct {
+    ID          string   `toml:"id"`
+    Role        string   `toml:"role"`        // "concierge", "architect", or expert name
+    Title       string   `toml:"title"`
+    Description string   `toml:"description"`
+    DependsOn   []string `toml:"depends_on"`
+}
+
+type Formula struct {
+    Description string `toml:"description"`
+    Steps       []Step `toml:"steps"`
+}
+```
+
+- `Load(path string) (*Formula, error)` — parse single TOML file, call `Validate()`
+- `LoadAll(dir string) (map[string]*Formula, error)` — scan `*.toml`, key by filename sans extension
+- `Validate(f *Formula) error` — checks:
+  - At least one step
+  - No duplicate step IDs
+  - All `depends_on` refs point to valid step IDs within the formula
+  - No cycles (Kahn's algorithm, same approach as `taskboard.DetectCycles`)
+  - Non-empty `id`, `role`, `title` on each step
+
+Uses `github.com/BurntSushi/toml` (already a dependency via `config`).
+
+## Phase 2: Formula Instantiation — architect MCP tool
+
+New `instantiate_formula` tool registered in `RegisterArchitectTools`.
+
+### Files
+
+- **`internal/mcp/architect_tools.go`** — add `instantiate_formula` tool registration + handler
+- **`internal/mcp/architect_tools_test.go`** — test instantiation produces correct messages
+
+### Tool Schema
+
+```text
+instantiate_formula
+  formula:    string (required) — formula name (filename without .toml)
+  prefix:     string (required) — ID prefix for generated tasks (e.g., "feat-auth")
+  overrides:  string (optional) — JSON object: step ID → custom body text
+  experts:    string (optional) — JSON object: step ID → specific expert name
+```
+
+### Handler Logic
+
+1. Load formula from `{poolDir}/formulas/{formula}.toml` via `formula.Load()`
+2. Parse `overrides` and `experts` JSON if provided
+3. For each step, build a `mail.Message`:
+   - `ID` = `{prefix}-{step.id}`
+   - `From` = `"architect"`
+   - `To` = step's role, overridden by `experts[step.id]` if present
+   - `Type` = `mail.TypeTask`
+   - `Body` = `overrides[step.id]` if present, else step's description (prepend step title as heading)
+   - `DependsOn` = step's `depends_on` with prefix applied (e.g., `["gather"]` → `["feat-auth-gather"]`)
+   - `Priority` = `mail.PriorityNormal`
+   - `Timestamp` = `time.Now().UTC()`
+4. Post all messages via `postMessage(poolDir, msg)`
+5. Return summary: formula name, N tasks created, task IDs
+
+### Validation
+
+- Formula must exist
+- Steps with `role = "experts"` MUST have an entry in `experts` map (architect provides the name)
+- Prefix must be filename-safe (same check as message ID: `filepath.Base(prefix) == prefix`)
+
+## Phase 3: Config Hot-Reload
+
+Watch `pool.toml` for changes, re-parse, validate, swap config under lock.
+
+### Files
+
+- **`internal/daemon/daemon.go`** — add `reloadConfig()` method, handle pool.toml events in `Run()`
+- **`internal/daemon/watcher.go`** — extend `Run()` to also emit events for `Write` on `.toml` files (not just `Create` on `.md`)
+- **`internal/daemon/daemon_test.go`** — test: modify pool.toml while daemon runs, verify new expert becomes spawnable
+
+### Watcher Changes
+
+Current watcher only handles `Create` events on `.md` files. For config reload:
+
+- Add a new event type flag or use `WatcherEvent.Dir` to distinguish config events
+- In `Run()`, also match `fsnotify.Write` events on files ending in `.toml`
+- Apply `waitForStable` before emitting
+- Pool directory is already watched (for postoffice). But `pool.toml` lives at `{poolDir}/pool.toml` — the watcher watches `{poolDir}/postoffice/`, not `{poolDir}/` itself. **Must add `poolDir` itself to the watcher.**
+
+### Daemon Changes
+
+In `Run()`, detect pool.toml events (check `event.Path` ends with `pool.toml`):
+
+```go
+func (d *Daemon) reloadConfig() error {
+    newCfg, err := config.LoadPool(d.poolDir)
+    if err != nil {
+        d.logger.Warn("Config reload failed, keeping current config", "error", err)
+        return err  // keep old config
+    }
+    d.mu.Lock()
+    defer d.mu.Unlock()
+    oldCfg := d.cfg
+    d.cfg = newCfg
+    // Rebuild shared expert lookup set
+    d.sharedSet = make(map[string]bool, len(newCfg.Shared.Include))
+    for _, name := range newCfg.Shared.Include {
+        d.sharedSet[name] = true
+    }
+    // Diff and log changes
+    d.logConfigDiff(oldCfg, newCfg)
+    return nil
+}
+```
+
+For expert add/remove: after reload, call `ensureDirs()` and add new inbox dirs to the watcher. For removals, stop watching but preserve directories.
+
+New event type: `EventConfigReloaded`.
+
+## Phase 4: Hardening
+
+- **`internal/daemon/daemon.go`** — add `formulas/` to `ensureDirs()`
+- **Audit atomic writes** — verify all daemon-consumed file paths use `atomicfile.WriteFile` (taskboard already does, postoffice already does, verify nothing was missed)
+- **TOML validation in reload** — already handled by `config.LoadPool` which calls `Validate()`. Parse failure = log warning + keep old config (built into Phase 3)
+
+## Implementation Order
+
+1. Phase 1 (formula parsing) — standalone, no dependencies
+2. Phase 2 (instantiation tool) — depends on Phase 1
+3. Phase 3 (config hot-reload) — independent of 1+2
+4. Phase 4 (hardening) — after all above, quick sweep
+
+Phases 1 and 3 can be developed in parallel, but I'll do them sequentially since 1 is a prerequisite for 2 and I want to test the full flow.
+
+## Key Patterns to Follow
+
+- **Test file structure**: comment block at top documenting coverage matrix (see existing `*_test.go` files)
+- **Chicago-school tests**: real objects, fakes only at I/O boundaries
+- **Error wrapping**: `fmt.Errorf("context: %w", err)`
+- **Atomic writes**: `atomicfile.WriteFile` for all daemon-consumed files
+- **Reuse `postMessage`** (`internal/mcp/postoffice.go`) for formula instantiation
+- **Reuse `splitCSV`** (`internal/mcp/architect_tools.go`) for parsing parameters
+- **MCP tool registration** follows `RegisterArchitectTools` pattern
+- **Event bus** emits at slog log points — add events for new behaviors
+
+## Verification
+
+After each phase:
+1. `make test` — all tests pass
+2. `make check` — vet + lint + test
+
+End-to-end verification:
+1. Create a test formula TOML, load and validate it
+2. Instantiate via the MCP tool, verify N messages posted with correct dependency edges
+3. Modify pool.toml in a daemon integration test, verify config swap + new expert watcher
+4. `make test-cover` — check coverage for new packages

internal/daemon/daemon.go

@@ -147,6 +147,11 @@ func (d *Daemon) Run(ctx context.Context) error {
 	}
 	defer watcher.Close()
 
+	// Watch pool dir itself for pool.toml changes (config hot-reload)
+	if err := watcher.Add(d.poolDir); err != nil {
+		return fmt.Errorf("watching pool dir: %w", err)
+	}
+
 	// Watch postoffice
 	postofficeDir := filepath.Join(d.poolDir, "postoffice")
 	if err := watcher.Add(postofficeDir); err != nil {
@@ -254,6 +259,14 @@ func (d *Daemon) Run(ctx context.Context) error {
 				return nil
 			}
 
+			// Config reload events
+			if event.Kind == EventKindConfig {
+				if filepath.Base(event.Path) == "pool.toml" {
+					d.reloadConfig(watcher)
+				}
+				continue
+			}
+
 			if event.Dir == postofficeDir {
 				d.handlePostoffice(childCtx, event.Path)
 			} else if event.Dir == approvalsDir {
@@ -1245,6 +1258,7 @@ func (d *Daemon) ensureDirs() error {
 		filepath.Join(d.poolDir, "postoffice"),
 		filepath.Join(d.poolDir, "contracts"),
 		filepath.Join(d.poolDir, "approvals"),
+		filepath.Join(d.poolDir, "formulas"),
 		// Built-in roles get top-level inbox + logs directories
 		filepath.Join(d.poolDir, "architect", "inbox"),
 		filepath.Join(d.poolDir, "architect", "logs"),
@@ -1295,6 +1309,154 @@ func (d *Daemon) ensureDirs() error {
 	return nil
 }
 
+// reloadConfig re-reads pool.toml, validates, and swaps the config under lock.
+// On parse/validation failure, the old config is kept and a warning is logged.
+// The watcher parameter is used to add inbox watches for new experts.
+//
+// The lock is held only for the config swap and diff. Filesystem I/O (dir
+// creation, watcher registration) runs outside the lock to avoid blocking
+// concurrent goroutines that need d.mu (registerTask, handleInbox, etc.).
+func (d *Daemon) reloadConfig(watcher *Watcher) {
+	newCfg, err := config.LoadPool(d.poolDir)
+	if err != nil {
+		d.logger.Warn("Config reload failed, keeping current config",
+			"error", err,
+		)
+		return
+	}
+
+	// Phase 1: Swap config and compute diff under lock (fast, no I/O)
+	d.mu.Lock()
+	oldCfg := d.cfg
+	d.cfg = newCfg
+
+	// Rebuild shared expert lookup set
+	d.sharedSet = make(map[string]bool, len(newCfg.Shared.Include))
+	for _, name := range newCfg.Shared.Include {
+		d.sharedSet[name] = true
+	}
+
+	// Rebuild curation scheduler with new thresholds
+	d.curation = newCurationScheduler(&newCfg.Curation, d.poolDir, d.logger)
+
+	// Diff pool-scoped experts
+	oldExperts := make(map[string]bool, len(oldCfg.Experts))
+	for name := range oldCfg.Experts {
+		oldExperts[name] = true
+	}
+	var addedExperts, removedExperts []string
+	for name := range newCfg.Experts {
+		if !oldExperts[name] {
+			addedExperts = append(addedExperts, name)
+		}
+	}
+	for name := range oldExperts {
+		if _, ok := newCfg.Experts[name]; !ok {
+			removedExperts = append(removedExperts, name)
+		}
+	}
+
+	// Diff shared experts
+	oldShared := make(map[string]bool, len(oldCfg.Shared.Include))
+	for _, name := range oldCfg.Shared.Include {
+		oldShared[name] = true
+	}
+	var addedShared, removedShared []string
+	for _, name := range newCfg.Shared.Include {
+		if !oldShared[name] {
+			addedShared = append(addedShared, name)
+		}
+	}
+	for _, name := range oldCfg.Shared.Include {
+		if !d.sharedSet[name] {
+			removedShared = append(removedShared, name)
+		}
+	}
+
+	d.mu.Unlock()
+
+	added := append(addedExperts, addedShared...)
+	removed := append(removedExperts, removedShared...)
+
+	// Phase 2: Filesystem setup outside the lock (slow I/O)
+	var setupFailed bool
+
+	// Pool-scoped expert dirs + watchers
+	for _, name := range addedExperts {
+		expertBase := filepath.Join(d.poolDir, "experts", name)
+		for _, sub := range []string{"inbox", "logs"} {
+			if err := os.MkdirAll(filepath.Join(expertBase, sub), 0o755); err != nil {
+				d.logger.Error("Failed to create directory for new expert",
+					"expert", name,
+					"dir", sub,
+					"error", err,
+				)
+				setupFailed = true
+			}
+		}
+		inboxDir := mail.ResolveInbox(d.poolDir, name)
+		if err := watcher.Add(inboxDir); err != nil {
+			d.logger.Error("Failed to watch new expert inbox",
+				"expert", name,
+				"error", err,
+			)
+			setupFailed = true
+		}
+	}
+
+	// Shared expert dirs + watchers
+	for _, name := range addedShared {
+		inboxDir := mail.ResolveSharedInbox(d.poolDir, name)
+		if err := os.MkdirAll(inboxDir, 0o755); err != nil {
+			d.logger.Error("Failed to create shared inbox directory",
+				"expert", name,
+				"error", err,
+			)
+			setupFailed = true
+		}
+		logDir := mail.ResolveSharedLogDir(d.poolDir, name)
+		if err := os.MkdirAll(logDir, 0o755); err != nil {
+			d.logger.Error("Failed to create shared log directory",
+				"expert", name,
+				"error", err,
+			)
+			setupFailed = true
+		}
+		if err := watcher.Add(inboxDir); err != nil {
+			d.logger.Error("Failed to watch shared expert inbox",
+				"expert", name,
+				"error", err,
+			)
+			setupFailed = true
+		}
+	}
+
+	for _, name := range removed {
+		d.logger.Info("Expert removed from config (inbox preserved)",
+			"expert", name,
+		)
+	}
+
+	if setupFailed {
+		d.logger.Warn("Config reloaded with partial setup failures",
+			"experts_added", added,
+			"experts_removed", removed,
+		)
+		return
+	}
+
+	d.logger.Info("Successfully reloaded config",
+		"experts_added", added,
+		"experts_removed", removed,
+	)
+
+	d.events.emit(Event{
+		Type:      EventConfigReloaded,
+		Timestamp: time.Now(),
+		Data:      ConfigReloadedData{ExpertsAdded: added, ExpertsRemoved: removed},
+	})
+}
+
 // sharedNamesMap returns the cached shared expert lookup set. May be nil.
 func (d *Daemon) sharedNamesMap() map[string]bool {
 	return d.sharedSet