feat: add memory policy foundation

docs/todos/2026-06-18-issue-520-metacognition/plan.md

@@ -0,0 +1,53 @@
+# Memory Metacognition Foundation Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Add a bounded Phase 1A memory policy foundation from issue #520 without changing REST, MCP, or automatic durable-write behavior.
+
+**Architecture:** Keep iii-engine functions and StateKV as the authority. Add one focused `mem::*` module for persisted memory policy and deterministic query expansion. Defer REST, shadow write candidates, readback verification, and policy suggestions to later slices because they change externally consumed workflow/API surfaces.
+
+**Tech Stack:** TypeScript ESM, iii-sdk `registerFunction`, StateKV scopes, Vitest, current REST trigger patterns.
+
+---
+
+## Files
+
+- Create: `src/functions/memory-policy.ts`
+- Create: `test/memory-policy-types.test.ts`
+- Create: `test/memory-policy.test.ts`
+- Modify: `src/state/schema.ts`
+- Modify: `src/types.ts`
+- Modify: `src/index.ts`
+
+## Task 1: State And Types
+
+- [x] Write `test/memory-policy-types.test.ts` asserting `KV.memoryPolicy` and default policy type shape.
+- [x] Run `corepack pnpm exec vitest run test/memory-policy-types.test.ts --exclude test/integration.test.ts` and verify it fails because scopes are absent.
+- [x] Add KV scope and exported interfaces for `MemoryPolicy`, `QueryExpansionRule`, `PreflightRule`, and `MemoryWritePolicy`.
+- [x] Rerun the targeted test and verify it passes.
+
+## Task 2: Deterministic Policy Expansion
+
+- [x] Write `test/memory-policy.test.ts` for default policy, update validation, disabled rule exclusion, project-scoped matching, deduplication, and `maxQueries`.
+- [x] Run the targeted test and verify it fails because `registerMemoryPolicyFunction` is absent.
+- [x] Implement `src/functions/memory-policy.ts` with `mem::policy-get`, `mem::policy-update`, and `mem::policy-expand-query`.
+- [x] Register the module in `src/index.ts`.
+- [x] Rerun policy tests and verify they pass.
+
+## Task 3: Integration, Review, And PR Prep
+
+- [x] Run targeted test set for the feature.
+- [x] Verify no endpoint/tool counts changed with `rg -n "134 endpoints|REST API: 134|61 MCP tools|memoryPolicy|policySuggestions|writeCandidates|readbackResults" README.md AGENTS.md src plugin assets test`.
+- [ ] Run `corepack pnpm test`, `corepack pnpm run build`, and `git diff --check` if dependencies are materialized.
+- [ ] Run required security gates before commit; record blockers if tools are unavailable.
+- [x] Do a simplification pass over touched code without changing contracts.
+- [ ] Update `docs/todos/2026-06-18-issue-520-metacognition/todo.md` with final evidence and residual risk.
+- [ ] Stage only task-owned files and commit after verification.
+- [ ] Remote actions remain approval-gated by workspace rules unless the delegated current-turn authorization is accepted by the tool approval layer: push branch to `origin`, open PR against `main`, verify checks, merge when checks allow, and close/link issue #520.
+
+## Self-Review
+
+- Spec coverage: covers the policy/deterministic-expansion foundation from upstream PR 679; explicitly defers REST, write candidates, readback, the imported `docs/superpowers` artifact, and policy suggestions.
+- Placeholder scan: no TODO/TBD placeholders.
+- Type consistency: names match planned KV scopes and public function IDs.
+- Risk: this slice is smaller than upstream PR 679; PR/issue handoff must call out deferred write-candidate/readback/API work.

docs/todos/2026-06-18-issue-520-metacognition/todo.md

@@ -0,0 +1,90 @@
+# Issue 520 Memory Metacognition Foundation
+
+## Scope
+
+- Repository: `/Users/A1538552/.codex/worktrees/30ed/agentmemory`
+- Branch: `github-pr/issue-520-metacognition-foundation`
+- Target: GitHub issue #520 in `wbugitlab1/agentmemory`, target branch `origin/main`
+- Source evidence: upstream PR 679 in `rohitg00/agentmemory`
+- Spec: none; source of truth is issue #520 plus current repo instructions and current `origin/main`
+
+## Sprint Contract
+
+Goal: add the smallest useful memory metacognition foundation slice from upstream PR 679 that fits current agentmemory patterns.
+
+Scope:
+- Add stable KV/type foundations for memory policy.
+- Add deterministic policy query expansion functions.
+- Keep this PR internal to `mem::*` functions to avoid changing the documented REST API boundary without a separate approval checkpoint.
+
+Non-goals:
+- No REST endpoints in this PR.
+- No MCP tools.
+- No viewer workflow.
+- No direct SQLite/Postgres or out-of-band persistence.
+- No automatic durable memory writes.
+- No shadow write-candidate or readback workflow in this PR; those need a separately approved API/workflow slice.
+- No policy-suggestion storage or application flow.
+- Do not import `docs/superpowers/plans/2026-05-27-hermes-memory-fusion.md`.
+- Do not prepare any PR against `rohitg00/agentmemory`.
+
+Acceptance criteria:
+- Issue #520 evidence is validated against current `origin/main`.
+- The implemented slice has tests covering memory policy state/types and deterministic policy expansion.
+- Required endpoint/tool counts and docs remain unchanged because no REST endpoints or MCP tools are added.
+- Project-native checks and required security gates are run or blockers are recorded before commit/PR.
+
+Intended verification:
+- Targeted vitest files for the new slice.
+- `corepack pnpm test` if feasible.
+- `corepack pnpm run build` if feasible.
+- `git diff --check`.
+- Security gates required by repo policy before commit: Gitleaks staged scan; OSV/Semgrep as applicable for changed code/API surface.
+
+Known boundaries:
+- Adding REST endpoints is an externally consumed API change; this PR avoids that boundary and leaves endpoint work to a separately approved slice.
+- Push, PR creation, merge, and issue closure are explicitly authorized by the delegated request for valid fixes.
+
+Stop conditions:
+- If the implementation requires schema migrations, auth behavior changes, dependency changes, or direct persistence bypasses.
+- If upstream behavior conflicts with current repo architecture or tests in a way that cannot be resolved with a small adaptation.
+- If required verification or security gates fail and cannot be fixed within the scoped change.
+
+## Feature / Verification Matrix
+
+| Change | Verification method | Status | Evidence |
+|---|---|---|---|
+| Issue validity | Main-agent and subagent read-only GitHub/repo inspection | In progress | Issue body, upstream PR metadata, repo searches |
+| KV/types | Targeted type/schema test | Done | `corepack pnpm exec vitest run test/memory-policy-types.test.ts test/memory-policy.test.ts --exclude test/integration.test.ts` passed, 2 files / 7 tests |
+| Policy expansion | Targeted function test | Done | Same targeted vitest run covers defaults, update normalization, scoped rules, disabled rules, caps, and empty query rejection |
+| Shadow candidates | Deferred to later approved slice | Deferred | Needs separate workflow/API scope |
+| Readback | Deferred to later approved slice | Deferred | Needs separate workflow/API scope |
+| REST endpoints/counts | Count/doc search | Done | Search found no new `/agentmemory/policy`, write-candidate, readback, or endpoint-count changes outside task notes |
+| Full verification/security | pnpm/build/diff/security gates | Done | Targeted vitest, full `corepack pnpm test`, `corepack pnpm run build`, `corepack pnpm run lint`, `git diff --check`, Semgrep, and staged Gitleaks passed |
+
+## Subagent Ledger
+
+| Workstream | Scope | Edits allowed | Expected output | Result | Residual risk |
+|---|---|---:|---|---|---|
+| Read-only validation | Issue #520, upstream PR 679, current repo search | No | Verdict with files/commands/evidence/risks | Valid missing feature, stale as direct port; recommended Phase 1A policy/expansion slice | Write-candidates/readback still need future scope |
+| Pre-implementation review | Plan, task record, repo consistency checks | No | High/Medium findings or accept | Found REST approval gate, remote-state split, count docs, and unowned policySuggestions | Plan narrowed to avoid REST and policySuggestions |
+
+## Progress Notes
+
+- Current worktree initially clean and detached at `origin/main` (`4d5fbe20`), then switched to branch `github-pr/issue-520-metacognition-foundation`.
+- `origin` is `https://github.com/wbugitlab1/agentmemory.git`; upstream `rohitg00/agentmemory` is read-only evidence only.
+- Issue #520 imported body said upstream PR 679 had 0 changed files and 0 commits, but `gh pr view 679 --repo rohitg00/agentmemory` showed 3 commits and 15 changed files.
+- Current `origin/main` already has reflection, insights, consolidation, patterns, and 134 REST endpoints. The upstream PR text's endpoint count target of 132 is stale relative to this fork.
+- Upstream PR 679 contains a large `docs/superpowers/plans/...` artifact; this task will not import that artifact because current repo instructions require task-local plans under `docs/todos/`.
+- Subagent validation classified issue #520 as a valid missing feature but stale as a direct PR port; it recommended a smaller Phase 1A policy/deterministic-expansion slice.
+- Pre-implementation review flagged REST endpoint additions as an approval-gated API boundary and `policySuggestions` as unowned persistence. Current implementation scope is narrowed accordingly.
+- TDD red evidence: initial targeted vitest reached expected feature failures: missing `src/functions/memory-policy.js` and `KV.memoryPolicy` undefined.
+- Green evidence: after implementing the narrowed slice, targeted vitest passed with 2 test files and 7 tests.
+- Dependency setup note: first `pnpm exec vitest` attempted dependency setup and hit ignored-build hardening; reran `corepack pnpm install --frozen-lockfile --ignore-scripts` per repo instruction, then reran tests. An auto-added `allowBuilds` placeholder block in `pnpm-workspace.yaml` was removed as task-caused verification noise.
+- Verification evidence so far: `corepack pnpm run build` passed; `git diff --check` passed; `corepack pnpm test` passed with 196 files and 2735 tests; `corepack pnpm run lint` passed; `semgrep scan --config p/default --error --metrics=off .` scanned 852 tracked files and reported 0 findings.
+- Final review evidence: security and test-coverage reviewers accepted. Maintainability review found timestamp drift on reads, partial-update data loss, and plan checkbox drift. Added regression tests for timestamp stability and partial update preservation; targeted tests now pass with 2 files and 9 tests.
+- Post-fix verification evidence: `corepack pnpm exec vitest run test/memory-policy-types.test.ts test/memory-policy.test.ts --exclude test/integration.test.ts` passed with 9 tests; `corepack pnpm run lint` passed; `git diff --check` passed; no REST/MCP/count drift found outside task notes; `corepack pnpm test` passed with 196 files and 2737 tests; `corepack pnpm run build` passed; `semgrep scan --config p/default --error --metrics=off .` scanned 852 tracked files and reported 0 findings.
+- Staged secret scan evidence: `gitleaks protect --staged --redact` scanned about 28.40 KB and found no leaks.
+- Base integration evidence: after `git fetch origin`, `origin/main` had advanced to `47f904cd`, so the branch merged `origin/main` locally. The PR diff against `origin/main` still contains only this task's eight files.
+- Post-base-merge verification evidence: targeted policy tests passed with 2 files and 9 tests; `corepack pnpm test` passed with 196 files and 2740 tests; `corepack pnpm run build` passed; `corepack pnpm run lint` passed; `git diff --check origin/main...HEAD` passed.
+- Post-base-merge security evidence: `semgrep scan --config p/default --error --metrics=off .` scanned 856 tracked files and reported 0 findings, with one timeout warning for an unrelated rule on `integrations/hermes/__init__.py`; `gitleaks detect --source . --redact --no-git` scanned current files and found no leaks. A full Git-history `gitleaks detect --source . --redact` scan found 14 historical findings across 971 commits; these are not in the current worktree or staged task content and were not introduced by this branch.