From 86a0d7ed4137493beee0e2be1dd21a680b859bac Mon Sep 17 00:00:00 2001 From: Saiteja Kura Date: Mon, 13 Jul 2026 17:28:14 +0000 Subject: [PATCH 1/3] RFD: predicate caching Add design for caching predicate results using file-based watch paths. Two mechanisms: syntactic (watch field on skill groups) and runtime (stdout JSON from shell predicates). Coordinates with PM interface for depends-on() caching via Cargo.lock. --- md/rfds/predicate-caching/README.md | 81 +++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 md/rfds/predicate-caching/README.md diff --git a/md/rfds/predicate-caching/README.md b/md/rfds/predicate-caching/README.md new file mode 100644 index 00000000..90032901 --- /dev/null +++ b/md/rfds/predicate-caching/README.md @@ -0,0 +1,81 @@ +# Predicate caching + +Predicates (especially `shell(...)`) spawn processes on every sync. This RFD helps them declare watch paths. Symposium caches results and skips re-evaluation while watched files are unchanged. No watch declaration = never cached. + +## Problem + +Auto-sync means predicates re-evaluate on every agent session start. A workspace with 10 plugins, each with a `shell(...)` gate, forks 10+ processes every time. + +## Design + +Two ways to declare watch paths: + +### 1. Syntactic + +For predicates whose inputs are known at authorship time: + +```toml +[[skills]] +predicates = ["shell(grep -q lambda CargoBrazil.toml)"] +watch = ["CargoBrazil.toml"] +source.path = "skills/lambda" +``` + +`watch` is a sibling field, not embedded in the predicate string. + +### 2. Runtime + +The shell command emits JSON to stdout: + +```json +{"result": true, "watch": ["CargoBrazil.toml"]} +``` + +Commands that print nothing or non-JSON fall back to exit-code semantics with no caching. Backward compatible. + +Runtime takes priority when both are present. Either is sufficient to enable caching. + +## How it works + +Fingerprint is `mtime + size` (same model as Cargo). A missing file is a valid fingerprint state — invalidates when the file appears. + +Cache lives at `predicate-cache.json`, keyed by the normalized predicate string. Two plugins with the same predicate share one entry. + +Algorithm: + +1. Look up predicate in cache +2. If found and all watched files match fingerprints → use cached result +3. Otherwise → evaluate, store result + watch paths if declared +4. On write, prune entries for predicates no longer in any manifest + +Cache is discarded on Symposium version upgrade. `watch = []` (empty) means static for the current invocation only — no persistent caching without at least one file to stat. + +## Built-in predicates + +- `workspace-member()` → cached, watch = [] (static per run) +- `path_exists(path)` → cached, watches the path itself +- `depends-on(name)` → cached, watches PM manifest (e.g. `Cargo.lock`) +- `env(...)` → never cached (env isn't file-observable) +- `shell(...)` → cached only with explicit watch declaration + +## PM integration + +`list_deps` return type expands to include optional watch paths: + +```rust +struct DepsResult { + ids: Vec, + watch: Option>, +} +``` + +`CargoPm` returns `[Cargo.lock]`. This is a return-type change, not a new method. + +## Implementation steps + +1. Extend `PredicateResult` with `watch: Option>`. All evaluators return `None` initially — no behavior change. +2. Cache storage: read/write, fingerprint comparison, dead-entry pruning. +3. Built-in predicates return their natural watch sets. +4. Shell runtime output: parse stdout JSON, fall back to exit code for non-JSON. +5. Syntactic `watch` field on `[[skills]]` groups. +6. PM `list_deps` watch support (coordinate with pm-split). From d699619082f12d935a8848a069df749c2b018225 Mon Sep 17 00:00:00 2001 From: Saiteja Kura Date: Tue, 14 Jul 2026 17:41:27 +0000 Subject: [PATCH 2/3] refactor: predicate caching RFD v2 --- md/SUMMARY.md | 2 + md/rfds/predicate-caching/README.md | 75 ++++++++++++++++------------- 2 files changed, 44 insertions(+), 33 deletions(-) diff --git a/md/SUMMARY.md b/md/SUMMARY.md index 59d4a66e..c3ed94e3 100644 --- a/md/SUMMARY.md +++ b/md/SUMMARY.md @@ -86,6 +86,8 @@ - [Template](./rfds/TEMPLATE/README.md) - [Accepted](./rfds/accepted.md) - [Registry-centric plugin distribution](./rfds/registry-centric-plugins/README.md) + - [Predicate caching](./rfds/predicate-caching/README.md) - [Completed](./rfds/completed.md) - [Configuration parsing and normalization](./rfds/config-normalization/README.md) - [RFD Process](./rfds/rfd-process/README.md) + - diff --git a/md/rfds/predicate-caching/README.md b/md/rfds/predicate-caching/README.md index 90032901..23b853f7 100644 --- a/md/rfds/predicate-caching/README.md +++ b/md/rfds/predicate-caching/README.md @@ -1,62 +1,72 @@ # Predicate caching -Predicates (especially `shell(...)`) spawn processes on every sync. This RFD helps them declare watch paths. Symposium caches results and skips re-evaluation while watched files are unchanged. No watch declaration = never cached. +## TL;DR + +Predicates (especially custom predicates) spawn processes on every sync. This RFD lets them emit watch declarations via JSONL events. Symposium caches results and skips re-evaluation while watched files/env vars are unchanged. No watch declaration = never cached. ## Problem -Auto-sync means predicates re-evaluate on every agent session start. A workspace with 10 plugins, each with a `shell(...)` gate, forks 10+ processes every time. +Auto-sync means predicates re-evaluate on every agent session start. A workspace with 10 plugins, each with a custom predicate, forks 10+ processes every time. ## Design -Two ways to declare watch paths: +Custom predicates already emit JSONL events to stdout. We add a `Watch` event variant: -### 1. Syntactic +```rust +#[derive(Serialize, Deserialize)] +enum CustomPredicateEvent { + // ... existing variants (e.g. SelectedCrates) ... + Watch { + files: Vec, + env: HashMap, + } +} +``` -For predicates whose inputs are known at authorship time: +Emitted as a JSONL line: -```toml -[[skills]] -predicates = ["shell(grep -q lambda CargoBrazil.toml)"] -watch = ["CargoBrazil.toml"] -source.path = "skills/lambda" +```jsonl +{"watch": {"files": ["CargoBrazil.toml"], "env": {"LAMBDA_ENV": "prod"}}} ``` -`watch` is a sibling field, not embedded in the predicate string. +The predicate result still comes from exit status — the watch event is just a cache hint. Predicates that don't emit a `Watch` event are never cached (current behavior preserved). -### 2. Runtime +Both `files` and `env` are optional (either alone is sufficient to enable caching). Files are relative to workspace root. -The shell command emits JSON to stdout: +## SDK helper -```json -{"result": true, "watch": ["CargoBrazil.toml"]} -``` +The `symposium-sdk` crate provides a helper that reads env vars and automatically emits the cache line: -Commands that print nothing or non-JSON fall back to exit-code semantics with no caching. Backward compatible. +```rust +// In a custom predicate binary: +let val = symposium_sdk::env::var("LAMBDA_ENV")?; +// ^ reads the var AND emits {"watch": {"env": {"LAMBDA_ENV": "prod"}}} +``` -Runtime takes priority when both are present. Either is sufficient to enable caching. +This way predicate authors get caching for free when they use the SDK. ## How it works -Fingerprint is `mtime + size` (same model as Cargo). A missing file is a valid fingerprint state — invalidates when the file appears. +Fingerprint for files is `mtime + size` (same model as Cargo). A missing file is a valid fingerprint state — invalidates when the file appears. Env fingerprint is the literal string value (or absent). -Cache lives at `predicate-cache.json`, keyed by the normalized predicate string. Two plugins with the same predicate share one entry. +Cache lives at `~/.symposium/cache/predicates.json`, keyed by the normalized predicate string. No workspace-root keying needed — watch paths are resolved relative to the current workspace root at stat time, so different workspaces naturally produce different fingerprints. Algorithm: 1. Look up predicate in cache -2. If found and all watched files match fingerprints → use cached result -3. Otherwise → evaluate, store result + watch paths if declared +2. If found and all watched files + env vars match fingerprints → use cached result +3. Otherwise → evaluate, store result + watch if declared 4. On write, prune entries for predicates no longer in any manifest -Cache is discarded on Symposium version upgrade. `watch = []` (empty) means static for the current invocation only — no persistent caching without at least one file to stat. +Cache is discarded on Symposium version upgrade. ## Built-in predicates -- `workspace-member()` → cached, watch = [] (static per run) +- `workspace-member()` → no cache needed (already cheap, in-memory) - `path_exists(path)` → cached, watches the path itself - `depends-on(name)` → cached, watches PM manifest (e.g. `Cargo.lock`) -- `env(...)` → never cached (env isn't file-observable) -- `shell(...)` → cached only with explicit watch declaration +- `env(FOO=BAR)` → cached, watches env value of `FOO` +- `shell(cmd)` / custom predicates → cached only when they emit a `Watch` event ## PM integration @@ -65,7 +75,7 @@ Cache is discarded on Symposium version upgrade. `watch = []` (empty) means stat ```rust struct DepsResult { ids: Vec, - watch: Option>, + watch: Option>, } ``` @@ -73,9 +83,8 @@ struct DepsResult { ## Implementation steps -1. Extend `PredicateResult` with `watch: Option>`. All evaluators return `None` initially — no behavior change. -2. Cache storage: read/write, fingerprint comparison, dead-entry pruning. -3. Built-in predicates return their natural watch sets. -4. Shell runtime output: parse stdout JSON, fall back to exit code for non-JSON. -5. Syntactic `watch` field on `[[skills]]` groups. -6. PM `list_deps` watch support (coordinate with pm-split). +1. Add `Watch` variant to `CustomPredicateEvent`. Parse it from predicate stdout. No caching yet — just capture the data. +2. Cache storage in `~/.symposium/cache/predicates.json`: read/write, fingerprint comparison, dead-entry pruning. +3. Wire built-in predicates: `path_exists` emits file watch, `env` emits env watch, `depends-on` uses PM watch paths. +4. Add `symposium_sdk::env::var()` helper that auto-emits the cache event. +5. PM `list_deps` watch support (coordinate with pm-split). From 21fbff9b92cd892326b91a000e06d98f53318341 Mon Sep 17 00:00:00 2001 From: Saiteja Kura Date: Fri, 17 Jul 2026 21:27:15 +0000 Subject: [PATCH 3/3] refactor: predicate caching RFD v3 --- md/SUMMARY.md | 1 - md/rfds/predicate-caching/README.md | 75 ++++++++++++++--------------- 2 files changed, 35 insertions(+), 41 deletions(-) diff --git a/md/SUMMARY.md b/md/SUMMARY.md index c3ed94e3..80e45bc8 100644 --- a/md/SUMMARY.md +++ b/md/SUMMARY.md @@ -90,4 +90,3 @@ - [Completed](./rfds/completed.md) - [Configuration parsing and normalization](./rfds/config-normalization/README.md) - [RFD Process](./rfds/rfd-process/README.md) - - diff --git a/md/rfds/predicate-caching/README.md b/md/rfds/predicate-caching/README.md index 23b853f7..57b85e5f 100644 --- a/md/rfds/predicate-caching/README.md +++ b/md/rfds/predicate-caching/README.md @@ -2,7 +2,7 @@ ## TL;DR -Predicates (especially custom predicates) spawn processes on every sync. This RFD lets them emit watch declarations via JSONL events. Symposium caches results and skips re-evaluation while watched files/env vars are unchanged. No watch declaration = never cached. +Predicates, especially custom predicates, spawn processes on every sync. This RFD lets custom predicates emit `Watch` JSONL events for files and environment variables. Symposium caches their results and skips reevaluation while the union of watched inputs is unchanged. No watch hints means cached indefinitely; `Volatile` means never cached. ## Problem @@ -10,81 +10,76 @@ Auto-sync means predicates re-evaluate on every agent session start. A workspace ## Design -Custom predicates already emit JSONL events to stdout. We add a `Watch` event variant: +Custom predicates already emit JSONL events to stdout. We add two variants: ```rust #[derive(Serialize, Deserialize)] enum CustomPredicateEvent { - // ... existing variants (e.g. SelectedCrates) ... + // ... existing variants ... Watch { files: Vec, env: HashMap, - } + }, + Volatile {}, } ``` -Emitted as a JSONL line: +A predicate can emit multiple watch hints: ```jsonl -{"watch": {"files": ["CargoBrazil.toml"], "env": {"LAMBDA_ENV": "prod"}}} +{"watch": {"files": ["CargoBrazil.toml"]}} +{"watch": {"files": ["Config"]}} +{"watch": {"env": {"LAMBDA_ENV": "prod"}}} ``` -The predicate result still comes from exit status — the watch event is just a cache hint. Predicates that don't emit a `Watch` event are never cached (current behavior preserved). +Symposium unions these into one watch set. A change to any watched input causes one reevaluation. Files are relative to the workspace root. -Both `files` and `env` are optional (either alone is sufficient to enable caching). Files are relative to workspace root. +The process exit status determines the predicate result. `Watch` and `Volatile` events only control caching. No `Watch` events means the result is cached indefinitely and predicates must report every changing input or Symposium may reuse a stale result. `Volatile` disables caching and takes precedence over watch hints. + +```jsonl +{"volatile": {}} +``` ## SDK helper -The `symposium-sdk` crate provides a helper that reads env vars and automatically emits the cache line: +The `symposium-sdk` crate provides a helper that reads an environment variable and emits its watch event: ```rust -// In a custom predicate binary: let val = symposium_sdk::env::var("LAMBDA_ENV")?; -// ^ reads the var AND emits {"watch": {"env": {"LAMBDA_ENV": "prod"}}} +// Emits {"watch": {"env": {"LAMBDA_ENV": "prod"}}}. ``` -This way predicate authors get caching for free when they use the SDK. +Multiple helper calls emit multiple events, which Symposium unions. ## How it works -Fingerprint for files is `mtime + size` (same model as Cargo). A missing file is a valid fingerprint state — invalidates when the file appears. Env fingerprint is the literal string value (or absent). +File fingerprints use `mtime + size`; missing is a valid state. Environment fingerprints use the current value or absent state. -Cache lives at `~/.symposium/cache/predicates.json`, keyed by the normalized predicate string. No workspace-root keying needed — watch paths are resolved relative to the current workspace root at stat time, so different workspaces naturally produce different fingerprints. +Cache lives at `~/.symposium/cache/predicates.json`. -Algorithm: - -1. Look up predicate in cache -2. If found and all watched files + env vars match fingerprints → use cached result -3. Otherwise → evaluate, store result + watch if declared -4. On write, prune entries for predicates no longer in any manifest +1. Look up the predicate in the cache. +2. If all watched inputs match, use the cached result. An empty watch set always matches. +3. Otherwise, evaluate the predicate and obtain its result from the exit status. +4. If it emitted `Volatile`, do not cache the result. Otherwise, store the result with the union of its watch hints. Cache is discarded on Symposium version upgrade. ## Built-in predicates -- `workspace-member()` → no cache needed (already cheap, in-memory) -- `path_exists(path)` → cached, watches the path itself -- `depends-on(name)` → cached, watches PM manifest (e.g. `Cargo.lock`) -- `env(FOO=BAR)` → cached, watches env value of `FOO` -- `shell(cmd)` / custom predicates → cached only when they emit a `Watch` event +- `workspace-member()` requires no cache because it is already cheap and evaluated in memory. +- `path_exists(path)` watches the path itself. +- `env(FOO=BAR)` watches the value of `FOO`. +- `shell(cmd)` is volatile because its inputs are unknown. +- Caching `depends-on(name)` is deferred to the PM interface work. ## PM integration -`list_deps` return type expands to include optional watch paths: - -```rust -struct DepsResult { - ids: Vec, - watch: Option>, -} -``` - -`CargoPm` returns `[Cargo.lock]`. This is a return-type change, not a new method. +Changes to `list_deps` and PM-derived caching are deferred to the PM interface work. ## Implementation steps -1. Add `Watch` variant to `CustomPredicateEvent`. Parse it from predicate stdout. No caching yet — just capture the data. -2. Cache storage in `~/.symposium/cache/predicates.json`: read/write, fingerprint comparison, dead-entry pruning. -3. Wire built-in predicates: `path_exists` emits file watch, `env` emits env watch, `depends-on` uses PM watch paths. -4. Add `symposium_sdk::env::var()` helper that auto-emits the cache event. -5. PM `list_deps` watch support (coordinate with pm-split). +1. Add and parse `Watch` and `Volatile` events while keeping the exit status as the predicate result. +2. Union watch hints, cache an empty watch set indefinitely, and give `Volatile` precedence. +3. Add cache storage and fingerprint comparison. +4. Wire `path_exists`, `env`, and volatile `shell` behavior. +5. Add `symposium_sdk::env::var()` to emit environment watch events.