WIP: arnold-merge chain snapshot (Sprint 0 spike)

chain_state.json

@@ -0,0 +1,12 @@
+{
+  "current_milestone_index": 1,
+  "current_plan_name": "sprint-1-foundation-schema-20260503-0201",
+  "last_state": "stalled",
+  "completed": [
+    {
+      "label": "sprint-0-spike",
+      "plan": "sprint-0-spike-validate-store-20260503-0106",
+      "status": "done"
+    }
+  ]
+}

docs/sprint-0-spike-report.md

@@ -0,0 +1,21 @@
+# Sprint 0 Spike Report
+
+## What the spike validated
+
+The throwaway-branch spike validated the refined storage direction against real Arnold call sites without touching production megaplan consumers. The unchanged Arnold `create_epic()` and `edit_epic()` flows ran through a compatibility adapter backed by the spike `FileStore`, including the seeded `inbound_message_id` message row, the turn update branch, checklist seeding and mutation, sprint replace/upsert, and queue transitions through `_lock_in`, `_queue`, `_pend`, `_reorder`, and gapless normalization. The spike journal also recovered correctly in both relevant crash windows: a process death after `prepare.json` but before `.commit` left no visible writes, and a process death after the `.commit` marker replayed the committed transaction on reopen.
+
+For the file-mode reader proof, the spike round-tripped copied real fixture trees from `arnold-source/.megaplan/plans/sprint-6-images-second-opinion` and `arnold-source/.megaplan/plans/sprint-3-multi-epic`, and the rework pass reran the same disposable-copy check on `arnold-source/.megaplan/plans/sprint-1b-discord-resident` specifically because that fixture carries both `review.json` and `execution_batch_10.json` (plus `execution_batch_11.json` through `execution_batch_14.json`). Those copies preserved filenames, layout, and file bytes after a byte-preserving read/write pass. The unchanged narrow `auto.py` helper surface still worked on the disposable copies: `_resolve_plan_dir(...)`, `_last_history_step_result(...)` at line 197, `_read_execute_blocking_deviations(...)` at line 223, and `_get_review_marker(...)` in the same tight loop used near line 685 all returned stable values without touching the source fixture tree. This is narrow compatibility evidence only; it is not proof that the broader plan-tree reader surface is already safe to migrate.
+
+## Protocol friction found
+
+The refined protocol is close, but the live Arnold/editorial surface is not yet a direct drop-in match. The real edit path still performs body writes through `update_epic(body=...)` rather than a dedicated `update_body(...)` call. The create path still depends on `seed_checklist(...)` as a first-class store operation. Queue behavior is still expressed as per-sprint mutations plus normalization, not as a single live `set_sprint_queue(...)` write. The store also has to support `transaction(epic_id=None)` plus message and turn updates before an epic exists, because Arnold binds the inbound message and turn rows only after `create_epic()` succeeds.
+
+The high-batch `auto.py` recheck also reinforces that this spike only proved the current narrow reader loop, not idealized plan-tree semantics. The copied `sprint-1b-discord-resident` fixture includes `execution_batch_10.json` through `execution_batch_14.json`, and `_read_execute_blocking_deviations(...)` still completed safely on that copy, but the helper currently sorts batch filenames lexicographically rather than numerically. Sprint 1 should keep that current behavior in mind when deciding whether the eventual repository abstraction needs to preserve filename ordering exactly or should normalize batch selection behind a compatibility layer.
+
+The journal spike also showed that transaction framing cannot be treated as file renames alone. The event log needs transaction-scoped framing with a shared `tx_id`, `_tx_begin`, `_tx_commit`, and tolerant scanning that ignores incomplete tails. Without that, recovery can restore files while leaving `events.jsonl` in a logically torn state.
+
+## Changes to absorb in Sprint 1
+
+Sprint 1 should keep the refined protocol, but land it behind a compatibility layer instead of forcing Arnold to change call shapes immediately. The production `Store` surface should preserve `transaction(epic_id=None)`, bridge `update_epic(body=...)` to `update_body(...)`, and either retain `seed_checklist(...)` or provide an explicit adapter-owned shim for it. Queue support should include the refined `set_sprint_queue(...)` primitive, but Sprint 1 still needs compatibility helpers for the live Arnold per-sprint queue workflow until the caller surface is migrated deliberately.
+
+On the file side, Sprint 1 should carry forward the transaction journal contract from the spike: `prepare.json`, `.commit`, recover-on-open, and framed `events.jsonl` writes that ignore incomplete transactions during replay. On the plan-tree side, Sprint 1 should treat the copied-fixture proof as evidence only for the narrow `auto.py` helper reads exercised here. Broader readers and writers across the rest of the `.megaplan/plans/*` surface remain explicitly deferred and need separate validation before they should be counted as compatible.

megaplan/_core/__init__.py

@@ -9,36 +9,63 @@
 
 # -- io.py: pure utilities, atomic I/O, paths, config -----------------------
 from .io import (
+    append_framed_json_records,
+    append_framed_json_transaction,
     artifact_path,
+    atomic_write_bytes,
     atomic_write_json,
     atomic_write_text,
     batch_artifact_path,
+    canonical_megaplan_root,
+    commit_journal_transaction,
+    committed_framed_json_transactions,
     collect_git_diff_patch,
     collect_git_diff_summary,
     compute_global_batches,
     compute_task_batches,
     config_dir,
     current_iteration_artifact,
     current_iteration_raw_artifact,
+    discard_uncommitted_journal_transaction,
     detect_available_agents,
     ensure_runtime_layout,
+    find_plan_dir,
     find_command,
+    framed_json_record_bytes,
+    fsync_dir,
+    fsync_file,
     get_effective,
+    has_any_plan_root,
+    journal_blob_promotion,
+    journal_bytes_write,
+    journal_commit_path,
+    journal_event_log,
+    journal_prepare_path,
+    journal_root,
+    journal_text_write,
     json_dump,
     list_batch_artifacts,
     load_config,
     load_finalize_snapshot,
     megaplan_root,
     normalize_text,
     now_utc,
+    orphan_plans_root,
+    plan_search_roots,
     plans_root,
+    prepare_journal_transaction,
     read_json,
+    read_committed_framed_json_records,
+    recover_journal,
+    repo_storage_id,
     render_final_md,
     save_config,
     schemas_root,
+    scrub_stale_staging_files,
     sha256_file,
     sha256_text,
     slugify,
+    write_journal_commit_marker,
 )
 
 # -- phase_runtime.py: centralized runtime policy ----------------------------
@@ -125,36 +152,63 @@
 
 __all__ = [
     # io
+    "append_framed_json_records",
+    "append_framed_json_transaction",
     "artifact_path",
+    "atomic_write_bytes",
     "atomic_write_json",
     "atomic_write_text",
     "batch_artifact_path",
+    "canonical_megaplan_root",
+    "commit_journal_transaction",
+    "committed_framed_json_transactions",
     "collect_git_diff_patch",
     "collect_git_diff_summary",
     "compute_global_batches",
     "compute_task_batches",
     "config_dir",
     "current_iteration_artifact",
     "current_iteration_raw_artifact",
+    "discard_uncommitted_journal_transaction",
     "detect_available_agents",
     "ensure_runtime_layout",
+    "find_plan_dir",
     "find_command",
+    "framed_json_record_bytes",
+    "fsync_dir",
+    "fsync_file",
     "get_effective",
+    "has_any_plan_root",
+    "journal_blob_promotion",
+    "journal_bytes_write",
+    "journal_commit_path",
+    "journal_event_log",
+    "journal_prepare_path",
+    "journal_root",
+    "journal_text_write",
     "json_dump",
     "list_batch_artifacts",
     "load_config",
     "load_finalize_snapshot",
     "megaplan_root",
     "normalize_text",
     "now_utc",
+    "orphan_plans_root",
+    "plan_search_roots",
     "plans_root",
+    "prepare_journal_transaction",
     "read_json",
+    "read_committed_framed_json_records",
+    "recover_journal",
+    "repo_storage_id",
     "render_final_md",
     "save_config",
     "schemas_root",
+    "scrub_stale_staging_files",
     "sha256_file",
     "sha256_text",
     "slugify",
+    "write_journal_commit_marker",
     # phase_runtime
     "DEFAULT_NON_EXECUTE_TIMEOUT_CAP_SECONDS",
     "PHASE_RUNTIME_POLICY",