From c3dca15e7bce3e19cfe54c492ab55316dc28bc3e Mon Sep 17 00:00:00 2001 From: PDD Bot Date: Wed, 8 Jul 2026 19:56:30 +0000 Subject: [PATCH 1/5] feat: transactional fingerprint finalization via FingerprintTransaction (#1926) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduces a single `FingerprintTransaction` context manager that enforces the 'artifact write ⇒ fingerprint write' invariant across all mutating PDD commands (sync, generate, update, auto-deps, fix, CI heal). Replaces scattered `save_fingerprint` call sites with one commit-or-fail code path featuring null-hash guards, atomic temp-file+rename persistence, and `FingerprintFinalizeError` propagation to Click (non-zero exit on failure). Co-Authored-By: Claude Sonnet 4.6 --- README.md | 8 +- architecture.json | 205 +++++++++++++++++- pdd/prompts/auto_deps_main_python.prompt | 10 +- pdd/prompts/ci_drift_heal_python.prompt | 3 +- .../fingerprint_transaction_python.prompt | 100 +++++++++ pdd/prompts/metadata_sync_python.prompt | 8 +- pdd/prompts/operation_log_python.prompt | 12 +- pdd/prompts/sync_main_python.prompt | 3 +- pdd/prompts/sync_orchestration_python.prompt | 12 +- pdd/prompts/update_main_python.prompt | 18 +- 10 files changed, 339 insertions(+), 40 deletions(-) create mode 100644 pdd/prompts/fingerprint_transaction_python.prompt diff --git a/README.md b/README.md index 538e885f97..0ba9b64326 100644 --- a/README.md +++ b/README.md @@ -1142,7 +1142,7 @@ pdd sync --no-one-session https://github.com/myorg/myrepo/issues/100 - **Compressed Context Telemetry**: Sync results and logs record whether compressed context was requested, the effective value after CLI and `.pddrc` resolution, whether it was actually applied for each phase, the source inputs used to build it, and whether the run fell back to agentic repair. This makes replay and benchmark comparisons distinguish normal sync from compressed-context sync. **Robust State Management**: -- **Fingerprint Files**: Maintains `.pdd/meta/{basename}_{language}.json` with operation history +- **Fingerprint Files**: Maintains `.pdd/meta/{basename}_{language}.json` with operation history. All fingerprint writes across every mutating command (sync, generate, example, update, fix, auto-deps, ci-heal) route through a single `FingerprintTransaction` context manager; writes are atomic (temp-file + `os.replace`) and enforced — a finalization failure is a command failure, not a silent warning. - **Run Reports**: Tracks test results, coverage, and execution status - **Lock Management**: Prevents race conditions with file-descriptor based locking - **Git Integration**: Leverages version control for change detection and rollback safety @@ -1888,7 +1888,7 @@ Where used: - Dependency references: Examples serve as lightweight (token efficient) interface references for other prompts and can be included as dependencies of a generate target. - Sanity checks: The example program is typically used as the runnable program for `crash` and `verify`, providing a quick end-to-end sanity check that the generated code runs and behaves as intended. - Auto-deps integration: The `auto-deps` command can scan example files (e.g., `examples/**/*.py`) and insert relevant references into prompts. Based on each example’s content (imports, API usage, filenames), it identifies useful development units to include as dependencies. -- Metadata finalization: A successful `pdd example` updates the affected module's fingerprint and clears its stale `.pdd/meta/__run.json` runtime-verification report, so a regenerated example never leaves runtime state describing the pre-mutation output. +- Metadata finalization: A successful `pdd example` updates the affected module's fingerprint and clears its stale `.pdd/meta/__run.json` runtime-verification report, so a regenerated example never leaves runtime state describing the pre-mutation output. The fingerprint write is atomic (temp-file + rename via `FingerprintTransaction`); a finalization failure exits non-zero rather than being surfaced as a warning. **When to use**: Choose this command when creating reusable references that other prompts can efficiently import. This produces token-efficient examples that are easier to reuse across multiple prompts compared to including full implementations. @@ -2988,13 +2988,13 @@ The command uses a two-stage retrieval pipeline when candidates exceed 50: After inserting `` directives, the command performs a **deduplication pass** that identifies and removes inline content in the prompt that semantically duplicates what the included documents already provide. -**Metadata finalization (on success):** After a successful `auto-deps` run, the command writes/updates a fingerprint in `.pdd/meta/` for the file that was actually written (with `operation="auto-deps"` and the current include-dependency hashes) and clears any stale per-module `_run.json` report keyed by the same identity, so downstream commands see a consistent view. The fingerprint write and the include-dependency metadata update are committed together (atomic from the caller's perspective) and use hash-based, language-agnostic helpers from `pdd.operation_log`. Finalization errors are surfaced as warnings and do not mask a successful `auto-deps` result. Invalid `` tags stripped by the prompt sanitizer and architecture-merge failures (e.g. `architecture.json` could not be patched in place) are also surfaced as yellow warnings before the success summary. +**Metadata finalization (on success):** After a successful `auto-deps` run, the command writes/updates a fingerprint in `.pdd/meta/` for the file that was actually written (with `operation="auto-deps"` and the current include-dependency hashes) and clears any stale per-module `_run.json` report keyed by the same identity, so downstream commands see a consistent view. The fingerprint write is atomic (temp-file + `os.replace` via `FingerprintTransaction`) and enforced — a finalization failure exits non-zero. Invalid `` tags stripped by the prompt sanitizer and architecture-merge failures (e.g. `architecture.json` could not be patched in place) are surfaced as yellow warnings before the success summary. Identity is derived from the **output path**: - **In-place mode** (`--output `, or `pdd sync`'s post-move flow): the output path is the canonical prompt, so the fingerprint lands at `.pdd/meta/_.json`. - **Default mode** (output is the separate `__with_deps.prompt` derivative): the fingerprint lands at `.pdd/meta/__with_deps.json`. Canonical metadata is intentionally untouched in this case — the derivative fingerprint records that an `auto-deps` run produced that file without overwriting the canonical module's record. -**Note on `pdd sync`:** `pdd sync` invokes `auto-deps` with a `*_with_deps.prompt` temp output and then moves it onto the canonical prompt. To avoid leaving orphan `.pdd/meta/*_with_deps.json` files for the moved temp prompt (and to avoid clearing the wrong run report), sync passes the internal `_skip_finalization=True` flag to `auto_deps_main` and owns canonical metadata itself: sync writes the canonical fingerprint via its atomic state machinery (`_save_fingerprint_atomic`) and clears the canonical `_run.json` via `clear_run_report` immediately after the move. +**Note on `pdd sync`:** `pdd sync` invokes `auto-deps` with a `*_with_deps.prompt` temp output and then moves it onto the canonical prompt. To avoid leaving orphan `.pdd/meta/*_with_deps.json` files for the moved temp prompt (and to avoid clearing the wrong run report), sync passes the internal `_skip_finalization=True` flag to `auto_deps_main` and owns canonical metadata itself: sync writes the canonical fingerprint via `FingerprintTransaction` (atomic temp-file + rename) and clears the canonical `_run.json` via `clear_run_report` immediately after the move. The command maintains a CSV file with the following columns: - `full_path`: The full path to the dependency file diff --git a/architecture.json b/architecture.json index 41de1ed75f..5b4a3da7d1 100644 --- a/architecture.json +++ b/architecture.json @@ -1543,7 +1543,8 @@ "insert_includes_python.prompt", "validate_prompt_includes_python.prompt", "operation_log_python.prompt", - "auto_deps_architecture_python.prompt" + "auto_deps_architecture_python.prompt", + "fingerprint_transaction_python.prompt" ], "priority": 35, "filename": "auto_deps_main_python.prompt", @@ -1558,7 +1559,7 @@ "functions": [ { "name": "auto_deps_main", - "signature": "(ctx: click.Context, prompt_file: str, directory_path: str, auto_deps_csv_path: Optional[str], output: Optional[str], force_scan: Optional[bool] = False, progress_callback: Optional[Callable[[int, int], None]] = None, include_docs: bool = False, no_dedup: bool = False, concurrency: int = 1, _skip_finalization: bool = False)", + "signature": "(ctx: click.Context, prompt_file: str, directory_path: str, auto_deps_csv_path: Optional[str], output: Optional[str], force_scan: Optional[bool] = False, progress_callback: Optional[Callable[[int, int], None]] = None, include_docs: bool = False, no_dedup: bool = False, concurrency: int = 1, compress: bool = False, _skip_finalization: bool = False)", "returns": "Tuple[str, float, str]" } ] @@ -2234,7 +2235,7 @@ }, { "reason": "Primary orchestration engine for transitioning .prompt files to source code, handling path resolution, incremental generation, and architectural conformance.", - "description": "Command-line interface for code generator. Parses arguments, orchestrates the workflow, and formats output. Raises structured ArchitectureConformanceError failures with output/expected/found/missing fields plus failed-attempt total_cost/model_name (constructor accepts an optional repair_directive override for the signature check whose source of truth is the prompt, not architecture.json), and injects a non-empty PDD_REPAIR_DIRECTIVE into the generation prompt inside an block so sync retry attempts receive concrete missing-export instructions. Architecture conformance now also enforces the prompt's signature across module/cli/command interface types in three categories: (1) missing function/method \u2014 declared callables (including dotted methods like ContentSelector.select) absent from the generated code surface as bare names in missing_symbols; (2) missing parameter \u2014 declared parameter names absent from a matching function/method signature surface as dotted funcname.paramname entries; (3) signature drift \u2014 annotation drift is conservative (raises only when both sides specify and differ) while default drift is strict (raises when the prompt declares a default and the generated code drops or changes it, since callers omitting an optional kwarg would otherwise break with TypeError). Each category emits a distinct error sentence so the agentic_sync_runner repair loop can build a targeted directive (add missing function/method, add missing parameter, or update parameter to match the prompt's annotation/default). Issue #1012 adds two additional repair-loop gates: (a) PublicSurfaceRegressionError \u2014 for mature modules (pre-existing non-empty code file), snapshots top-level functions, classes, recursively-walked nested classes/methods (so a method on `Outer.Inner` is captured as `Outer.Inner.method`), and module-level public `ast.Assign` / bound `ast.AnnAssign` (only when `node.value is not None`; bare annotations like `PUBLIC_NAME: int` don't bind a runtime attribute and are ignored) targets plus `ast.Import`/`ast.ImportFrom` aliases (so removing a re-export like `import git` or a public constant like `PUBLIC_SETTING = ...` surfaces as a regression). When the module declares `__all__` as a clean list/tuple of string constants, that list is AUTHORITATIVE per Python semantics: a name is public iff it appears in `__all__`, even if underscore-prefixed; names NOT in `__all__` are NOT public. Classes listed in `__all__` ALSO contribute their recursively-walked members (methods, nested classes, methods on nested classes at every depth) regardless of underscore prefix \u2014 so `__all__ = [\"Service\"]` protects `Service.run`, `Service.Inner`, and `Service.Inner.method`. Computed `__all__` (`sorted(...)`, `X + Y`) or AugAssign extension (`__all__ += [...]`) falls back to the heuristic. Without `__all__`, underscore-prefixed names (including dunders) are excluded at every level; `from X import *` contributes no fixed name and is ignored. The same `__all__` precedence rule (including the class-member recursion) is applied to the parallel `_snapshot_public_signatures` so signature-drift detection sees the same public set. `_snapshot_public_signatures` entries carry a leading kind prefix (`[function]`/`[async_function]`/`[class]` at top level; `[instance]`/`[staticmethod]`/`[classmethod]`/`[property:]` inside classes; `[assignment]` for module-level rebindings; and `[import]`/`[import:]`/`[import:from ]`/`[import:from :]` for re-exports) so a binding-kind flip (e.g. a re-exported `pathlib.Path` replaced by a same-named `def`) diffs even when the parameter list happens to match. For `@dataclass` classes without an explicit `__init__`, `_snapshot_public_signatures` synthesises the constructor signature from class-body `AnnAssign` fields in source order: `@dataclass(kw_only=True)` emits a leading `*` so every field is kw-only; an in-body `_: KW_ONLY` sentinel (`KW_ONLY` Name or `dataclasses.KW_ONLY` Attribute) splits positional vs kw-only fields and is recognised before the underscore-prefix skip; when both decorator and sentinel are present only ONE `*` is emitted (decorator wins); `InitVar[...]` fields ARE included (they are runtime constructor params even though not stored as instance attributes); `ClassVar[...]` fields and `field(init=False, ...)` defaults are excluded; an explicit `__init__` still wins over synthesis. Inherited dataclass fields are collected in REVERSE-MRO order (matching Python's `@dataclass`: `__dataclass_fields__` is populated by walking bases right-to-left so later bases overwrite earlier ones in dict-insertion order) \u2014 `class C(A, B)` synthesises `(b, a, c)`. A base decorated with `@dataclass(init=False)` STILL contributes its annotated fields to a derived `@dataclass`'s synth; `init=False` only suppresses the base's OWN `__init__` synthesis. Cross-module bases that resolve to imports rather than same-module `ClassDef`s are marked `[inherited_unresolved]`. When a derived class redeclares a base's field name, the derived annotation/default wins but the field's POSITION matches the base's original slot (insertion-order dict semantics). The gate raises when public symbols are removed/renamed or their callable signature changes unless the prompt body contains an anchored `BREAKING-CHANGE:` directive (regex `^\\s*BREAKING-CHANGE:\\s*` with `re.MULTILINE`) whose tail starts with a `remove`/`delete`/`drop`/`rename` action verb (followed by a comma-separated symbol list) for removals or a `change signature`/`change api`/`change contract` verb pair for signature drift. Buried mid-line marker mentions and bare `BREAKING-CHANGE:` markers do NOT disable the gate. First-time generation is exempt. (b) TestChurnError \u2014 when overwriting an existing non-empty test file through code_generator_main or cmd_test_main, computes the stdlib `difflib.unified_diff` churn ratio between pre and post test bodies and raises when the ratio exceeds `PDD_TEST_CHURN_THRESHOLD` (default 0.40, accepts both decimal `0.40` and percent `40%` strings; defensive fallback to default on unparseable values) without an anchored `BREAKING-CHANGE:` directive that pairs an opt-out verb (`rewrite`/`replace`/`regenerate`/`overwrite`/`churn`/`remove`/`drop`) with `tests` as the verb's direct object. PublicSurfaceRegressionError and TestChurnError share the existing PDD_REPAIR_DIRECTIVE plumbing so sync_main, sync_orchestration, and agentic_sync_runner route them through the same repair loop. Issue #1649 adds ProseOutputError before the architecture/public-surface/test-churn gates: when extracted generated content is empty or whitespace-only for a non-prompt target, code_generator_main raises a typed generation-output extraction failure containing provider/model, prompt, output path, extractor result, and raw-output excerpt, so sync retries once with an output-shape directive instead of misdiagnosing empty extraction as missing exports/churn. `PDD_ALLOW_EMPTY_GENERATION=1` is the explicit escape hatch for the rare intentional empty-output case.", + "description": "Command-line interface for code generator. Parses arguments, orchestrates the workflow, and formats output. Raises structured ArchitectureConformanceError failures with output/expected/found/missing fields plus failed-attempt total_cost/model_name (constructor accepts an optional repair_directive override for the signature check whose source of truth is the prompt, not architecture.json), and injects a non-empty PDD_REPAIR_DIRECTIVE into the generation prompt inside an block so sync retry attempts receive concrete missing-export instructions. Architecture conformance now also enforces the prompt's signature across module/cli/command interface types in three categories: (1) missing function/method — declared callables (including dotted methods like ContentSelector.select) absent from the generated code surface as bare names in missing_symbols; (2) missing parameter — declared parameter names absent from a matching function/method signature surface as dotted funcname.paramname entries; (3) signature drift — annotation drift is conservative (raises only when both sides specify and differ) while default drift is strict (raises when the prompt declares a default and the generated code drops or changes it, since callers omitting an optional kwarg would otherwise break with TypeError). Each category emits a distinct error sentence so the agentic_sync_runner repair loop can build a targeted directive (add missing function/method, add missing parameter, or update parameter to match the prompt's annotation/default). Issue #1012 adds two additional repair-loop gates: (a) PublicSurfaceRegressionError — for mature modules (pre-existing non-empty code file), snapshots top-level functions, classes, recursively-walked nested classes/methods (so a method on `Outer.Inner` is captured as `Outer.Inner.method`), and module-level public `ast.Assign` / bound `ast.AnnAssign` (only when `node.value is not None`; bare annotations like `PUBLIC_NAME: int` don't bind a runtime attribute and are ignored) targets plus `ast.Import`/`ast.ImportFrom` aliases (so removing a re-export like `import git` or a public constant like `PUBLIC_SETTING = ...` surfaces as a regression). When the module declares `__all__` as a clean list/tuple of string constants, that list is AUTHORITATIVE per Python semantics: a name is public iff it appears in `__all__`, even if underscore-prefixed; names NOT in `__all__` are NOT public. Classes listed in `__all__` ALSO contribute their recursively-walked members (methods, nested classes, methods on nested classes at every depth) regardless of underscore prefix — so `__all__ = [\"Service\"]` protects `Service.run`, `Service.Inner`, and `Service.Inner.method`. Computed `__all__` (`sorted(...)`, `X + Y`) or AugAssign extension (`__all__ += [...]`) falls back to the heuristic. Without `__all__`, underscore-prefixed names (including dunders) are excluded at every level; `from X import *` contributes no fixed name and is ignored. The same `__all__` precedence rule (including the class-member recursion) is applied to the parallel `_snapshot_public_signatures` so signature-drift detection sees the same public set. `_snapshot_public_signatures` entries carry a leading kind prefix (`[function]`/`[async_function]`/`[class]` at top level; `[instance]`/`[staticmethod]`/`[classmethod]`/`[property:]` inside classes; `[assignment]` for module-level rebindings; and `[import]`/`[import:]`/`[import:from ]`/`[import:from :]` for re-exports) so a binding-kind flip (e.g. a re-exported `pathlib.Path` replaced by a same-named `def`) diffs even when the parameter list happens to match. For `@dataclass` classes without an explicit `__init__`, `_snapshot_public_signatures` synthesises the constructor signature from class-body `AnnAssign` fields in source order: `@dataclass(kw_only=True)` emits a leading `*` so every field is kw-only; an in-body `_: KW_ONLY` sentinel (`KW_ONLY` Name or `dataclasses.KW_ONLY` Attribute) splits positional vs kw-only fields and is recognised before the underscore-prefix skip; when both decorator and sentinel are present only ONE `*` is emitted (decorator wins); `InitVar[...]` fields ARE included (they are runtime constructor params even though not stored as instance attributes); `ClassVar[...]` fields and `field(init=False, ...)` defaults are excluded; an explicit `__init__` still wins over synthesis. Inherited dataclass fields are collected in REVERSE-MRO order (matching Python's `@dataclass`: `__dataclass_fields__` is populated by walking bases right-to-left so later bases overwrite earlier ones in dict-insertion order) — `class C(A, B)` synthesises `(b, a, c)`. A base decorated with `@dataclass(init=False)` STILL contributes its annotated fields to a derived `@dataclass`'s synth; `init=False` only suppresses the base's OWN `__init__` synthesis. Cross-module bases that resolve to imports rather than same-module `ClassDef`s are marked `[inherited_unresolved]`. When a derived class redeclares a base's field name, the derived annotation/default wins but the field's POSITION matches the base's original slot (insertion-order dict semantics). The gate raises when public symbols are removed/renamed or their callable signature changes unless the prompt body contains an anchored `BREAKING-CHANGE:` directive (regex `^\\s*BREAKING-CHANGE:\\s*` with `re.MULTILINE`) whose tail starts with a `remove`/`delete`/`drop`/`rename` action verb (followed by a comma-separated symbol list) for removals or a `change signature`/`change api`/`change contract` verb pair for signature drift. Buried mid-line marker mentions and bare `BREAKING-CHANGE:` markers do NOT disable the gate. First-time generation is exempt. (b) TestChurnError — when overwriting an existing non-empty test file through code_generator_main or cmd_test_main, computes the stdlib `difflib.unified_diff` churn ratio between pre and post test bodies and raises when the ratio exceeds `PDD_TEST_CHURN_THRESHOLD` (default 0.40, accepts both decimal `0.40` and percent `40%` strings; defensive fallback to default on unparseable values) without an anchored `BREAKING-CHANGE:` directive that pairs an opt-out verb (`rewrite`/`replace`/`regenerate`/`overwrite`/`churn`/`remove`/`drop`) with `tests` as the verb's direct object. PublicSurfaceRegressionError and TestChurnError share the existing PDD_REPAIR_DIRECTIVE plumbing so sync_main, sync_orchestration, and agentic_sync_runner route them through the same repair loop. Issue #1649 adds ProseOutputError before the architecture/public-surface/test-churn gates: when extracted generated content is empty or whitespace-only for a non-prompt target, code_generator_main raises a typed generation-output extraction failure containing provider/model, prompt, output path, extractor result, and raw-output excerpt, so sync retries once with an output-shape directive instead of misdiagnosing empty extraction as missing exports/churn. `PDD_ALLOW_EMPTY_GENERATION=1` is the explicit escape hatch for the rare intentional empty-output case.", "dependencies": [ "construct_paths_python.prompt", "preprocess_python.prompt", @@ -2533,7 +2534,8 @@ "agentic_common_python.prompt", "agentic_update_python.prompt", "sync_determine_operation_python.prompt", - "operation_log_python.prompt" + "operation_log_python.prompt", + "fingerprint_transaction_python.prompt" ], "priority": 62, "filename": "update_main_python.prompt", @@ -3070,7 +3072,8 @@ "dependencies": [ "code_generator_main_python.prompt", "agentic_sync_runner_python.prompt", - "compressed_sync_context_python.prompt" + "compressed_sync_context_python.prompt", + "fingerprint_transaction_python.prompt" ], "priority": 70, "filename": "sync_orchestration_python.prompt", @@ -3086,7 +3089,7 @@ "functions": [ { "name": "sync_orchestration", - "signature": "(basename: str, target_coverage: float = 90.0, language: str = \"python\", prompts_dir: str = \"prompts\", code_dir: str = \"src\", examples_dir: str = \"examples\", tests_dir: str = \"tests\", max_attempts: int = 3, budget: float = 10.0, skip_verify: bool = False, skip_tests: bool = False, dry_run: bool = False, force: bool = False, strength: float = DEFAULT_STRENGTH, temperature: float = 0.0, time_param: float = 0.25, verbose: bool = False, quiet: bool = False, output_cost: Optional[str] = None, review_examples: bool = False, local: bool = False, context_config: Optional[Dict[str, str]] = None, context_override: Optional[str] = None, confirm_callback: Optional[Callable[[str, str], bool]] = None, no_steer: bool = False, steer_timeout: float = DEFAULT_STEER_TIMEOUT_S, agentic_mode: bool = False, snapshot_context: bool = False, compressed_context: bool = False) -> Dict[str, Any]", + "signature": "(basename: str, target_coverage: float = 90.0, language: str = \"python\", prompts_dir: str = \"prompts\", code_dir: str = \"src\", examples_dir: str = \"examples\", tests_dir: str = \"tests\", max_attempts: int = 3, budget: float = 10.0, skip_verify: bool = False, skip_tests: bool = False, dry_run: bool = False, force: bool = False, strength: float = DEFAULT_STRENGTH, temperature: float = 0.0, time_param: float = 0.25, verbose: bool = False, quiet: bool = False, output_cost: Optional[str] = None, review_examples: bool = False, local: bool = False, context_config: Optional[Dict[str, str]] = None, context_override: Optional[str] = None, confirm_callback: Optional[Callable[[str, str], bool]] = None, no_steer: bool = False, steer_timeout: float = DEFAULT_STEER_TIMEOUT_S, agentic_mode: bool = False, one_session: bool = False, compress: bool = False, evidence: bool = False, snapshot_context: bool = False, compressed_context: bool = False) -> Dict[str, Any]", "returns": "Dict[str, Any]" } ] @@ -5443,7 +5446,8 @@ "core/cloud_python.prompt", "code_generator_main_python.prompt", "agentic_sync_runner_python.prompt", - "compressed_sync_context_python.prompt" + "compressed_sync_context_python.prompt", + "fingerprint_transaction_python.prompt" ], "priority": 148, "filename": "sync_main_python.prompt", @@ -6000,7 +6004,7 @@ }, { "reason": "Orchestrates the 13-step agentic change workflow for implementing GitHub issues, with optional clean restart.", - "description": "Orchestrates the 13-step agentic change workflow. Supports `clean_restart=True` to ignore any persisted solving state for the issue and start a fresh full pdd-issue flow from the target base branch (issue #1149). Hardens JSON/Markdown artifact handling by clearing artifacts before attempts and internal provider retries, persisting only accepted decision JSON, deriving Step 9 manual-review lines exactly from accepted Step 9 JSON, and requiring Step 13 PR success/recovery to validate an OPEN PR in the exact repo/base/head branch with non-empty PR head SHA plus remote/local head SHA alignment even when the provider result is classified as failed; Step 13 clears unaccepted Markdown before posting a trusted success comment after prose synthesis or exact-head recovery. Includes Step 8.5 (pre-flight drift heal) \u2014 detects prompts whose code has drifted and runs `pdd update` per module inside the worktree before Step 9 rewrites the prompts. Includes Step 10.5 (doc-sync contract verifier) \u2014 before Step 10, calls pdd.sync_order.discover_associated_documents to populate the LLM's associated_documents context, using the authoritative changed-file set so Step 9's worktree fallback path cannot bypass discovery when FILES_* markers are missing; after Step 10, enforces that every discovered doc appears in exactly one of ASSOCIATED_DOCS_MODIFIED / ASSOCIATED_DOCS_CONFLICTS / ASSOCIATED_DOCS_UNCHANGED. Silent drops and bucket overlaps are appended as ORCHESTRATOR_POSTCHECK_WARNINGS and routed to Step 11 via step10_output; PDD_STRICT_DOC_SYNC=1 turns violations into hard workflow aborts (issue #739).", + "description": "Orchestrates the 13-step agentic change workflow. Supports `clean_restart=True` to ignore any persisted solving state for the issue and start a fresh full pdd-issue flow from the target base branch (issue #1149). Hardens JSON/Markdown artifact handling by clearing artifacts before attempts and internal provider retries, persisting only accepted decision JSON, deriving Step 9 manual-review lines exactly from accepted Step 9 JSON, and requiring Step 13 PR success/recovery to validate an OPEN PR in the exact repo/base/head branch with non-empty PR head SHA plus remote/local head SHA alignment even when the provider result is classified as failed; Step 13 clears unaccepted Markdown before posting a trusted success comment after prose synthesis or exact-head recovery. Includes Step 8.5 (pre-flight drift heal) — detects prompts whose code has drifted and runs `pdd update` per module inside the worktree before Step 9 rewrites the prompts. Includes Step 10.5 (doc-sync contract verifier) — before Step 10, calls pdd.sync_order.discover_associated_documents to populate the LLM's associated_documents context, using the authoritative changed-file set so Step 9's worktree fallback path cannot bypass discovery when FILES_* markers are missing; after Step 10, enforces that every discovered doc appears in exactly one of ASSOCIATED_DOCS_MODIFIED / ASSOCIATED_DOCS_CONFLICTS / ASSOCIATED_DOCS_UNCHANGED. Silent drops and bucket overlaps are appended as ORCHESTRATOR_POSTCHECK_WARNINGS and routed to Step 11 via step10_output; PDD_STRICT_DOC_SYNC=1 turns violations into hard workflow aborts (issue #739).", "dependencies": [ "architecture_sync_python.prompt", "agentic_common_python.prompt", @@ -7370,7 +7374,9 @@ { "reason": "Provides sync operation logging, fingerprints, run reports, and compression metadata storage.", "description": "Decorator-based operation logging that tracks PDD command executions. Manages fingerprints, run reports, and event logs. Works with Click commands to extract module identity from prompt_file parameter.", - "dependencies": [], + "dependencies": [ + "fingerprint_transaction_python.prompt" + ], "priority": 210, "filename": "operation_log_python.prompt", "filepath": "pdd/operation_log.py", @@ -7395,7 +7401,7 @@ }, { "name": "log_event", - "signature": "(basename: str, language: str, event: str, details: Mapping[str, Any] | None = None, project_root=None, paths=None) -> None", + "signature": "(basename: str, language: str, event: str, details: Mapping[str, Any] | None = None, project_root=None, paths=None, compression=None, agentic_fallback=None) -> None", "returns": "None" } ] @@ -8767,7 +8773,8 @@ "auto_include_python.prompt", "agentic_langtest_python.prompt", "architecture_sync_python.prompt", - "metadata_sync_python.prompt" + "metadata_sync_python.prompt", + "fingerprint_transaction_python.prompt" ], "priority": 80, "filename": "ci_drift_heal_python.prompt", @@ -9610,7 +9617,8 @@ "dependencies": [ "architecture_sync_python.prompt", "sync_determine_operation_python.prompt", - "operation_log_python.prompt" + "operation_log_python.prompt", + "fingerprint_transaction_python.prompt" ], "priority": 5, "filename": "metadata_sync_python.prompt", @@ -10984,5 +10992,178 @@ ] } } + }, + { + "reason": "Single enforced code path for all PDD fingerprint writes — atomic temp-file + os.replace with commit-or-fail semantics.", + "description": "Context manager that owns: eager identity and path resolution at entry, atomic file write (temp-file + os.replace) at exit, null-hash validation, commit-or-fail exit semantics, and dry-run/redirect skip control. Used by every PDD mutating command (sync, generate, example, update, fix, auto-deps, ci-heal).", + "dependencies": [ + "sync_determine_operation_python.prompt" + ], + "priority": 5, + "filename": "fingerprint_transaction_python.prompt", + "filepath": "pdd/fingerprint_transaction.py", + "tags": [ + "module", + "python" + ], + "interface": { + "type": "module", + "module": { + "functions": [ + { + "name": "FingerprintTransaction.__init__", + "signature": "(basename: str, language: str, operation: str, paths: Optional[Dict[str, Path]] = None, cost: float = 0.0, model: str = '') -> None", + "returns": "None" + }, + { + "name": "FingerprintTransaction.__enter__", + "signature": "() -> FingerprintTransaction", + "returns": "FingerprintTransaction" + }, + { + "name": "FingerprintTransaction.__exit__", + "signature": "(exc_type, exc_val, exc_tb) -> bool", + "returns": "bool" + }, + { + "name": "FingerprintTransaction.skip", + "signature": "(reason: str) -> None", + "returns": "None" + }, + { + "name": "FingerprintTransaction.set_include_deps_override", + "signature": "(deps: Dict[str, str]) -> None", + "returns": "None" + } + ] + } + }, + "contract_summary": { + "rules": [ + "R1", + "R2", + "R3", + "R4", + "R5", + "R6", + "R7", + "R8" + ], + "critical": [], + "stories": [], + "capabilities": [], + "coverage_status": "partial", + "evidence_status": "missing", + "waived": [], + "unchecked": [], + "rules_detail": { + "R1": { + "rule_id": "R1", + "status": "test-only", + "stories": [], + "tests": [ + "test_R1_rejects_zero_refund", + "test_R1_rejects_negative_refund", + "test_cross_module_known_id_no_issue", + "test_r1_is_checked" + ], + "waiver": null, + "waiver_status": null, + "waiver_expires": null, + "failures": [] + }, + "R2": { + "rule_id": "R2", + "status": "test-only", + "stories": [], + "tests": [ + "test_multiple_rules_from_one_function", + "test_r2_is_story_only" + ], + "waiver": null, + "waiver_status": null, + "waiver_expires": null, + "failures": [] + }, + "R3": { + "rule_id": "R3", + "status": "test-only", + "stories": [], + "tests": [ + "test_R3_no_provider_call_invalid", + "test_cross_module_refs_matching_prompt", + "test_cross_module_refs_other_prompt_excluded", + "test_comment_heuristic", + "test_r3_is_test_only" + ], + "waiver": null, + "waiver_status": null, + "waiver_expires": null, + "failures": [] + }, + "R4": { + "rule_id": "R4", + "status": "test-only", + "stories": [], + "tests": [ + "test_r4_is_story_only" + ], + "waiver": null, + "waiver_status": null, + "waiver_expires": null, + "failures": [] + }, + "R5": { + "rule_id": "R5", + "status": "test-only", + "stories": [], + "tests": [ + "test_r5_is_unchecked" + ], + "waiver": null, + "waiver_status": null, + "waiver_expires": null, + "failures": [] + }, + "R6": { + "rule_id": "R6", + "status": "test-only", + "stories": [], + "tests": [ + "test_r6_is_waived" + ], + "waiver": null, + "waiver_status": null, + "waiver_expires": null, + "failures": [] + }, + "R7": { + "rule_id": "R7", + "status": "failed", + "stories": [], + "tests": [ + "test_R7_broken_syntax" + ], + "waiver": null, + "waiver_status": null, + "waiver_expires": null, + "failures": [ + "test_receipt_failed.py: syntax error on line 4" + ] + }, + "R8": { + "rule_id": "R8", + "status": "test-only", + "stories": [], + "tests": [ + "test_covers_comment_style" + ], + "waiver": null, + "waiver_status": null, + "waiver_expires": null, + "failures": [] + } + } + } } ] diff --git a/pdd/prompts/auto_deps_main_python.prompt b/pdd/prompts/auto_deps_main_python.prompt index 422b480809..9621b2468b 100644 --- a/pdd/prompts/auto_deps_main_python.prompt +++ b/pdd/prompts/auto_deps_main_python.prompt @@ -16,6 +16,7 @@ validate_prompt_includes_python.prompt operation_log_python.prompt auto_deps_architecture_python.prompt +fingerprint_transaction_python.prompt % You are an expert Python engineer. Your goal is to write the `auto_deps_main` function that will be part of the `pdd` command-line program. This function will handle the core logic for the `auto-deps` command. @@ -86,17 +87,16 @@ • On any other exception: print a red error message (unless `quiet`) and return an error tuple `("", 0.0, f"Error: {exc}")` instead of calling `sys.exit(1)`. This allows the orchestrator to handle failures gracefully. - Metadata finalization (on successful runs only — skip on the error path above): • After the modified prompt and any CSV output have been written successfully, finalize fingerprint and include-dependency state for the file that was actually written (`output_file_paths["output"]`) so downstream commands see a consistent view. This must run for *every* successful auto-deps mutation — both the default CLI flow (where the output is `__with_deps.prompt`) and the in-place flow (where the output equals the input `prompt_file`). Issue #989 requires this for every mutated prompt file; do not gate the finalizer on `output_path == prompt_file`. - • Use the existing helpers from `pdd.operation_log` (`infer_module_identity`, `save_fingerprint`, `clear_run_report`, `get_run_report_path`). Behavior must be language-agnostic (hash-based). + • Use the existing helpers from `pdd.operation_log` (`infer_module_identity`, `clear_run_report`, `get_run_report_path`) and `FingerprintTransaction` from `pdd.fingerprint_transaction`. Behavior must be language-agnostic (hash-based). • Derive identity from the *output* path, not the input prompt: - Resolve `(basename, language)` via `infer_module_identity(Path(output_path))`. The output path is the file whose hash and dependency state the fingerprint records; tying the identity to that file keeps `.pdd/meta` self-consistent even when the canonical prompt is untouched (default mode) or when the file is the canonical prompt itself (in-place mode). `infer_module_identity` returns `(None, None)` for unrecognized prompt names — check `basename is None or language is None` explicitly and skip finalization (logged as a yellow warning) rather than guessing. - - Clear any stale per-module run report via `clear_run_report(basename, language, paths={"prompt": Path(output_path)})` because the prompt content changed. If `clear_run_report` raises, log a warning and continue — it must not abort the subsequent fingerprint save. + - Clear any stale per-module run report via `clear_run_report(basename, language, paths={"prompt": Path(output_path)})` because the prompt content changed. If `clear_run_report` raises, log a warning and continue — it must not abort the subsequent fingerprint transaction. - After invoking `clear_run_report`, verify the report path is actually gone via `get_run_report_path(basename, language, paths={"prompt": Path(output_path)})`; if it still exists, log a yellow warning (unless `quiet`). This is defensive because the helper silently swallows unlink failures. - - Write/update the fingerprint via `save_fingerprint(...)` with `operation="auto-deps"`, `paths={"prompt": Path(output_path)}`, `cost=total_cost`, and `model=model_name`, recording the new prompt hash and the current include-dependency hashes so the fingerprint write and the include-dependency metadata update are committed together (atomic from the caller's perspective). + - Write/update the fingerprint via `FingerprintTransaction(basename, language, operation="auto-deps", paths={"prompt": Path(output_path)}, cost=total_cost, model=model_name)` used as a context manager. When a pre-captured include graph is available, call `transaction.set_include_deps_override(include_deps)` before exiting the context. `FingerprintFinalizeError` propagates — a finalization failure exits non-zero. When `_skip_finalization=True`, call `transaction.skip(reason="_skip_finalization set by sync")` instead of omitting the transaction entirely. • Architecture file updates (architecture.json) are merged by `merge_auto_deps_includes_from_cwd` and are not finalized here — that file is not a `(basename, language)`-keyed module and has its own atomicity guarantees inside the merge helper. • Surface non-exception failures from the architecture merge (return shape `{"updated": bool, "messages": list[str], …}`): if the helper returned `updated=False`, emit each `messages` entry as a yellow `Warning: architecture.json not updated: …` line so the success message does not mask a silent no-op. • Surface invalid `` tags stripped by `sanitize_prompt_output` (it returns `(cleaned_prompt, invalid_includes: list[str])`): if `invalid_includes` is non-empty, emit a yellow warning naming the count, the output path, and the stripped entries so the user knows generated dependencies were dropped. - • Finalization errors must not mask a successful `auto-deps` result: log a yellow warning (unless `quiet`) and continue; do not raise. - • NOTE: `pdd sync` invokes `auto_deps_main` with a temp `__with_deps.prompt` output and then `shutil.move`s it onto the canonical prompt. To avoid leaving an orphan `.pdd/meta/*_with_deps.json` for that temp identity after the move (and to avoid clearing the wrong run report), sync passes the internal flag `_skip_finalization=True` and owns canonical metadata itself: it writes the canonical fingerprint via `_save_fingerprint_atomic` (`pdd/sync_orchestration.py` around line 2719) and clears the canonical `_run.json` via `clear_run_report` immediately after the move. When `_skip_finalization=True`, this finalizer must return immediately after the success console output without writing a fingerprint or clearing any run report. + • NOTE: `pdd sync` invokes `auto_deps_main` with a temp `__with_deps.prompt` output and then `shutil.move`s it onto the canonical prompt. To avoid leaving an orphan `.pdd/meta/*_with_deps.json` for that temp identity after the move (and to avoid clearing the wrong run report), sync passes the internal flag `_skip_finalization=True`; the `FingerprintTransaction` for that identity calls `transaction.skip(reason="_skip_finalization set by sync")` and sync owns canonical metadata itself via `FingerprintTransaction` (routed through `_save_fingerprint_atomic`) and clears the canonical `_run.json` via `clear_run_report` immediately after the move. When `_skip_finalization=True`, this finalizer must enter the `FingerprintTransaction` context but call `transaction.skip(...)` before returning. pdd/operation_log.py diff --git a/pdd/prompts/ci_drift_heal_python.prompt b/pdd/prompts/ci_drift_heal_python.prompt index cc4d3b5237..af2ed669bb 100644 --- a/pdd/prompts/ci_drift_heal_python.prompt +++ b/pdd/prompts/ci_drift_heal_python.prompt @@ -26,6 +26,7 @@ agentic_langtest_python.prompt architecture_sync_python.prompt metadata_sync_python.prompt +fingerprint_transaction_python.prompt % You are an expert Python engineer. Your goal is to write a standalone CI script that detects prompt/example drift across all PDD modules and auto-heals drifted modules. @@ -52,7 +53,7 @@ A standalone CI script (`pdd/ci_drift_heal.py`) that orchestrates drift detectio 5. **Scope control:** `--modules` limits detection to specific basenames. Without it, scan all discovered modules. Accept both space-separated and comma-separated module lists. 6. **Three-way heal dispatch:** Dispatch by `drift.operation`: - - `update`: run `pdd --force --strength 0.5 update ` then invoke the shared `run_metadata_sync` orchestrator (from `pdd.metadata_sync`) for the (prompt, code) pair before the follow-up `pdd --force --strength 0.5 example ` step. The orchestrator finalizes prompt tags, architecture entries, run-report cleanup, and fingerprint state in fixed order; after a successful orchestrator result, refresh the operation-log fingerprint using the authoritative resolved module paths (not only the prompt/code pair handed to the orchestrator) so existing example/test hashes are preserved in the final fingerprint. Pass the `paths` dict from `get_pdd_file_paths` to every `read_fingerprint` and `save_fingerprint` call so reads and writes all anchor at the same subproject `.pdd/meta` directory (issue #1211). If code_path is unresolved, fail closed (no repo-wide fallback). After update, run churn gate and structural invariants gate before the orchestrator call. **Metadata finalization is mandatory and never silent**: `_run_metadata_sync_safe` returning False MUST be treated as a hard heal failure in every invocation mode (PR auto-heal, push-to-main auto-heal, and preflight drift-heal). The module's `heal_module` call MUST return False, the module MUST be recorded as failed (not merely skipped), and `main` MUST return a non-zero exit code with an explicit log line `metadata finalization failed for : ` so the workflow step fails loudly rather than committing a half-synced state. This finalization-failure escalation overrides the push-to-main "advisory" clause in Requirement 15. **Fingerprint refresh paths (issue #1211)**: after a successful `run_metadata_sync` result, build `paths = {"prompt": p, "code": code_p}` directly from the known subproject paths — do NOT call `get_pdd_file_paths(basename, language)`, which resolves from CWD and returns parent-repo paths in parent-CWD mode. Pass this dict to every `read_fingerprint` and `save_fingerprint` call. If `code_p` is None when `result.ok` is True, raise `ValueError("authoritative prompt/code paths unavailable: code_path not provided")`. Example/test-file hash validation (checking `example_hash`/`test_files` in the fingerprint against paths keys `"example"`/`"test_files"`) is skipped since those keys are absent from this minimal dict; that gap is tracked at issue #870. **Module-scoped snapshot/restore (issue #1211)**: implement `_subproject_root_for(drift)` — walk up from `drift.prompt_path` then `drift.code_path` looking for `.pddrc`, fall back to `_repo_root()`. Use it inside `_snapshot_metadata_state_for(drift)` (replacing `_repo_root()`) so `.pdd/meta` paths resolve under the subproject root. Change `_restore_metadata_state_for(snapshot, root: Path)` to accept an explicit `root` parameter; at both call sites pass `_subproject_root_for(drift)`. Before calling `_run_metadata_sync_safe`, capture a per-module snapshot of `architecture.json`, the operation-log fingerprint path `.pdd/meta/_.json`, and the operation-log run-report path `.pdd/meta/__run.json` via `_snapshot_metadata_state_for(drift)`. The safe basename must match `pdd.operation_log` path semantics, so subdirectory basenames such as `commands/foo` snapshot `.pdd/meta/commands_foo_python.json` and `.pdd/meta/commands_foo_python_run.json`, not `.pdd/meta/commands/foo_python.json`. On any orchestrator-stage failure OR on follow-up `pdd example` failure, call `_revert_prompt_file(drift)` AND `_restore_metadata_state_for(snapshot, _subproject_root_for(drift))` to revert this module's writes only. Do NOT use a repo-scoped `git restore .pdd` / `git restore architecture.json` here — that would wipe earlier successful modules' state from the same run on a multi-module push-to-main heal. The legacy `_cleanup_metadata_artifacts` name is preserved as a no-op shim solely for backward import compatibility; new code uses the snapshot/restore primitives. **Preflight drift-heal uses the same orchestrator call** so CLI auto-heal and preflight share one metadata-finalization path, and the same finalization-failure escalation applies there. + - `update`: run `pdd --force --strength 0.5 update ` then invoke the shared `run_metadata_sync` orchestrator (from `pdd.metadata_sync`) for the (prompt, code) pair before the follow-up `pdd --force --strength 0.5 example ` step. The orchestrator finalizes prompt tags, architecture entries, run-report cleanup, and fingerprint state in fixed order; after a successful orchestrator result, refresh the operation-log fingerprint using the authoritative resolved module paths (not only the prompt/code pair handed to the orchestrator) so existing example/test hashes are preserved in the final fingerprint. **Fingerprint refresh paths (issue #1211)**: after a successful `run_metadata_sync` result, build `paths = {"prompt": p, "code": code_p}` directly from the known subproject paths — do NOT call `get_pdd_file_paths(basename, language)`, which resolves from CWD and returns parent-repo paths in parent-CWD mode. Use `FingerprintTransaction(basename, language, operation="update", paths=paths, cost=0.0, model="")` for this post-orchestrator refresh; `FingerprintFinalizeError` is treated as a hard heal failure consistent with "Metadata finalization is mandatory and never silent." Pass the same `paths` dict to every `read_fingerprint` call so reads and writes all anchor at the same subproject `.pdd/meta` directory. If `code_p` is None when `result.ok` is True, raise `ValueError("authoritative prompt/code paths unavailable: code_path not provided")`. Example/test-file hash validation (checking `example_hash`/`test_files` in the fingerprint against paths keys `"example"`/`"test_files"`) is skipped since those keys are absent from this minimal dict; that gap is tracked at issue #870. If code_path is unresolved, fail closed (no repo-wide fallback). After update, run churn gate and structural invariants gate before the orchestrator call. **Metadata finalization is mandatory and never silent**: `_run_metadata_sync_safe` returning False MUST be treated as a hard heal failure in every invocation mode (PR auto-heal, push-to-main auto-heal, and preflight drift-heal). The module's `heal_module` call MUST return False, the module MUST be recorded as failed (not merely skipped), and `main` MUST return a non-zero exit code with an explicit log line `metadata finalization failed for : ` so the workflow step fails loudly rather than committing a half-synced state. This finalization-failure escalation overrides the push-to-main "advisory" clause in Requirement 15. **Module-scoped snapshot/restore (issue #1211)**: implement `_subproject_root_for(drift)` — walk up from `drift.prompt_path` then `drift.code_path` looking for `.pddrc`, fall back to `_repo_root()`. Use it inside `_snapshot_metadata_state_for(drift)` (replacing `_repo_root()`) so `.pdd/meta` paths resolve under the subproject root. Change `_restore_metadata_state_for(snapshot, root: Path)` to accept an explicit `root` parameter; at both call sites pass `_subproject_root_for(drift)`. Before calling `_run_metadata_sync_safe`, capture a per-module snapshot of `architecture.json`, the operation-log fingerprint path `.pdd/meta/_.json`, and the operation-log run-report path `.pdd/meta/__run.json` via `_snapshot_metadata_state_for(drift)`. The safe basename must match `pdd.operation_log` path semantics, so subdirectory basenames such as `commands/foo` snapshot `.pdd/meta/commands_foo_python.json` and `.pdd/meta/commands_foo_python_run.json`, not `.pdd/meta/commands/foo_python.json`. On any orchestrator-stage failure OR on follow-up `pdd example` failure, call `_revert_prompt_file(drift)` AND `_restore_metadata_state_for(snapshot, _subproject_root_for(drift))` to revert this module's writes only. Do NOT use a repo-scoped `git restore .pdd` / `git restore architecture.json` here — that would wipe earlier successful modules' state from the same run on a multi-module push-to-main heal. The legacy `_cleanup_metadata_artifacts` name is preserved as a no-op shim solely for backward import compatibility; new code uses the snapshot/restore primitives. **Preflight drift-heal uses the same orchestrator call** so CLI auto-heal and preflight share one metadata-finalization path, and the same finalization-failure escalation applies there. - `example`: run `pdd --force --strength 0.5 example ` directly. Require both paths resolved; fail closed otherwise. In CI, when `PDD_HEAL_SKIP_REVIEW_ONLY_EXAMPLE_DRIFT=1`, the drift reason is git-reclassified as already-covered PR review work (`Code and prompt changed together...` or `Prompt changed without code changes...`), and the resolved example file already exists, skip the example rewrite and leave that reviewed artifact to the PR author; missing examples still run the heal. When `PDD_HEAL_SKIP_EXISTING_EXAMPLE_DRIFT=1` and any resolved example file already exists, skip the example rewrite as a broader fallback. - `auto-deps`: run `pdd --force --strength 0.5 auto-deps --output --csv project_dependencies.csv` directly. Do not route this through full `pdd sync`, because sync can continue into generate/crash flows after dependency insertion. - `verify`/`generate`/`test`/`crash`: run `pdd --force --strength 0.5 sync ` to let sync_determine_operation dispatch correctly. In PR auto-heal mode (no `--skip-ci`), the subprocess env MUST include `PDD_DISABLE_TEST_EXTEND=1` so this nested sync cannot escalate a narrow heal into coverage-driven `test_extend`. Do NOT use `pdd example` for these — it would overwrite user edits (verify), skip code regeneration (generate), or save a stale fingerprint. The only exceptions are the git-based clean-CI reclassifications above, where a reported `auto-deps`/`generate` operation is converted to `example` or skipped based on actual code/prompt changes in the PR. diff --git a/pdd/prompts/fingerprint_transaction_python.prompt b/pdd/prompts/fingerprint_transaction_python.prompt new file mode 100644 index 0000000000..f6012a13dd --- /dev/null +++ b/pdd/prompts/fingerprint_transaction_python.prompt @@ -0,0 +1,100 @@ +context/python_preamble.prompt + +Single enforced code path for all PDD fingerprint writes — atomic temp-file + os.replace with commit-or-fail semantics. + + +{ + "type": "module", + "module": { + "functions": [ + {"name": "FingerprintTransaction.__init__", "signature": "(basename: str, language: str, operation: str, paths: Optional[Dict[str, Path]] = None, cost: float = 0.0, model: str = '') -> None", "returns": "None"}, + {"name": "FingerprintTransaction.__enter__", "signature": "() -> FingerprintTransaction", "returns": "FingerprintTransaction"}, + {"name": "FingerprintTransaction.__exit__", "signature": "(exc_type, exc_val, exc_tb) -> bool", "returns": "bool"}, + {"name": "FingerprintTransaction.skip", "signature": "(reason: str) -> None", "returns": "None"}, + {"name": "FingerprintTransaction.set_include_deps_override", "signature": "(deps: Dict[str, str]) -> None", "returns": "None"} + ] + } +} + + +sync_determine_operation_python.prompt + +% Goal +Write `pdd/fingerprint_transaction.py` — the single authoritative context manager for all PDD fingerprint writes. Replaces scattered `save_fingerprint` call sites with one enforced, atomic code path. + +% Role & Scope +This module provides the `FingerprintTransaction` class used by every PDD mutating command (sync, generate, example, update, fix, auto-deps, ci-heal). It owns: eager identity and path resolution at entry, atomic file write (temp-file + `os.replace`) at exit, null-hash validation, commit-or-fail exit semantics, and dry-run/redirect skip control. It is a leaf import — it does NOT import from the seven caller modules, eliminating circular dependency risk. + + +- Does not invoke LLM calls. +- Does not read or write run reports (that remains in operation_log). +- Does not push commits or run git commands. +- Does not own the `@log_operation` decorator (that stays in operation_log). + + + +- Commit: the `__exit__` path that writes the fingerprint file atomically and raises `FingerprintFinalizeError` on failure. +- Skip: suppressing the commit so `__exit__` is a no-op; used for dry-run, redirected output, or run-report-not-cleared conditions. +- Null hash: a `prompt_hash`, `code_hash`, `example_hash`, or `test_hash` value that is `None` — an indication that `calculate_current_hashes` could not resolve the file, producing a fingerprint that can never converge with on-disk state. + + + +R1 (MUST): When `__exit__` is called with `exc_type is None` (clean body) and `skip()` has NOT been called, write the fingerprint atomically (temp-file in `fingerprint_path.parent` + `os.fsync` + `os.replace`); raise `FingerprintFinalizeError` on ANY failure during this write. +R2 (MUST): When `__exit__` is called with `exc_type is not None` (exception in body), propagate the exception without touching the fingerprint file. Return `False` (do not suppress the exception). +R3 (MUST): Before writing, validate that `prompt_hash` is not `None`. If it is `None`, raise `FingerprintFinalizeError` naming the null-hash field and the fingerprint path (prevents the #1305 null-hash loop class). +R4 (MUST): Resolve `fingerprint_path` eagerly in `__init__` via `get_fingerprint_path(basename, language, paths=paths)`. Never re-resolve it later (prevents the #1211/#1290 wrong-root class). +R5 (MUST): Use a temp file in the **same directory** as the fingerprint (`dir=fingerprint_path.parent`) so that `os.replace` is a single-filesystem rename and is POSIX-atomic. Never write to a cross-device temp directory. +R6 (MUST): `skip(reason)` suppresses the commit; log the reason at INFO level via stdlib `logging`. After `skip()` is called, `__exit__` on a clean body MUST be a no-op (no file write, no error). +R7 (MUST NOT): Raise `FingerprintFinalizeError` when `skip()` has been called, even if `prompt_hash` would otherwise be `None`. +R8 (MUST): `FingerprintFinalizeError` message MUST include the fingerprint path, the operation name, and the underlying cause (if any). + + +% Requirements + +1. **`FingerprintFinalizeError(RuntimeError)`** — raised when atomic fingerprint write fails or null-hash is detected. Message format: `"[{operation}] fingerprint finalization failed for {fingerprint_path}: {cause}"`. No additional base class is required (no PDD-specific exception hierarchy exists). + +2. **`FingerprintTransaction.__init__`**: Eagerly resolve everything needed for the commit: + - Accept `basename`, `language`, `operation`, `paths` (optional `Dict[str, Path]`), `cost`, and `model`. + - Call `get_fingerprint_path(basename, language, paths=paths)` and store as `self._fingerprint_path`. + - Store `basename`, `language`, `operation`, `cost`, `model`, and `paths` for use in `__exit__`. + - Initialize `self._skipped: bool = False`, `self._skip_reason: Optional[str] = None`, and `self._include_deps_override: Optional[Dict[str, str]] = None`. + +3. **`skip(reason: str)`**: Set `self._skipped = True` and `self._skip_reason = reason`. Log at `logging.INFO`: `"FingerprintTransaction.skip for %s/%s (%s): %s"`. Must be idempotent (calling twice is safe). + +4. **`set_include_deps_override(deps: Dict[str, str])`**: Store `deps` for use in `__exit__` when building the `Fingerprint` dataclass. Called by `auto_deps_main` after the prompt file is written but before `__exit__`. + +5. **`__enter__`**: Return `self`. No side effects. + +6. **`__exit__(exc_type, exc_val, exc_tb)`**: + - If `exc_type is not None`: return `False` (propagate the exception, do not write). + - If `self._skipped`: log at DEBUG that the commit was suppressed; return `False`. + - Otherwise (clean body, not skipped): execute the atomic commit: + a. Read the previous fingerprint via `read_fingerprint(self._basename, self._language, paths=self._paths)` to obtain stored `include_deps` (use `None` when no previous fingerprint exists). + b. Call `calculate_current_hashes(self._paths, stored_include_deps=stored_deps)` to get current file hashes, where `stored_deps` is the `include_deps` from step (a) (or `None` when no previous fingerprint exists). + c. If `current_hashes.get('prompt_hash') is None`: raise `FingerprintFinalizeError` (R3). + d. Build the `Fingerprint` dataclass: `command=self._operation`, `pdd_version=__version__` (imported from `pdd`), `timestamp=datetime.now(timezone.utc).isoformat()`, `prompt_hash/code_hash/example_hash/test_hash/test_files` from `current_hashes`, and `include_deps` from `current_hashes.get('include_deps')` — the freshly extracted deps from step (b) — or `self._include_deps_override` when set, which takes priority over the computed current include_deps. The corrected logic is: `include_deps = self._include_deps_override if self._include_deps_override is not None else current_hashes.get('include_deps')`. + e. Create a temp file: `fd, tmp_path = tempfile.mkstemp(dir=self._fingerprint_path.parent, suffix=".tmp")`. + f. Write JSON via `json.dump(asdict(fingerprint), f, indent=2)` then `os.fsync(fd)` then close. + g. Call `os.replace(tmp_path, self._fingerprint_path)` for atomic rename. + h. On ANY exception during steps (a)–(g): attempt `os.unlink(tmp_path)` (suppress `OSError`); then raise `FingerprintFinalizeError` with the original exception as cause. + - Return `False` (never suppress exceptions from the body). + +7. **Imports**: Use only stdlib and existing PDD primitives — `get_fingerprint_path` from `pdd.operation_log`; `read_fingerprint`, `calculate_current_hashes`, `Fingerprint` from `pdd.sync_determine_operation`; `__version__` from `pdd`; `datetime`, `timezone` from `datetime`. Use `dataclasses.asdict` for JSON serialization. Use `tempfile.mkstemp`, `os.replace`, `os.fsync`, `os.unlink`, `json`, `logging`. + +8. **Windows note**: Document in a module-level comment that `os.replace()` on Windows is not POSIX-atomic when the destination exists; this is the same behavior as the existing `_save_fingerprint_atomic` implementation and is a known limitation. + +% Dependencies + + pdd/operation_log.py + + + pdd/sync_determine_operation.py + + + + context/operation_log_example.py + + +% Deliverables +- Code: `pdd/fingerprint_transaction.py` +- Public API: `FingerprintTransaction`, `FingerprintFinalizeError` diff --git a/pdd/prompts/metadata_sync_python.prompt b/pdd/prompts/metadata_sync_python.prompt index 69ca4b216b..5ead5c943e 100644 --- a/pdd/prompts/metadata_sync_python.prompt +++ b/pdd/prompts/metadata_sync_python.prompt @@ -20,6 +20,7 @@ architecture_sync_python.prompt sync_determine_operation_python.prompt operation_log_python.prompt +fingerprint_transaction_python.prompt % Goal Write `pdd/metadata_sync.py` — a single-function orchestrator that runs prompt metadata finalization stages in a fixed order and returns a structured result. @@ -34,7 +35,7 @@ This module provides the shared `run_metadata_sync` entry point used by `pdd upd 2. `tags`: **preserve existing PDD metadata tags, or seed tags from the architecture entry when the prompt has none.** Uses `pdd.architecture_sync.generate_tags_from_architecture` (multi-language compatible, no language-specific AST dependency) to render the seed block. Writes the seeded prompt back to disk unless `dry_run=True`. When the prompt has zero PDD tags AND no architecture entry exists to seed from, report `skipped` with a reason naming #870 — never `ok` (claiming `ok with 0 tag(s)` would mislead operators). **LLM-first tag refresh (rewriting existing-but-stale tags) is out of scope for this orchestrator and tracked at #870**; downstream callers that need refresh should invoke that primitive directly. 3. `architecture`: call `update_architecture_from_prompt(prompt_filename, prompts_dir, architecture_path, dry_run, prompt_content_override=)` so architecture.json picks up the freshly-generated tags without a disk race. `prompt_filename` MUST be the prompts-dir-relative path (e.g. `commands/foo_python.prompt`), not just the basename — `update_architecture_from_prompt` matches architecture entries by their `filename` field which is path-aware per Issue #617. Fall back to the basename only when the prompt sits outside any `prompts/` ancestor. When the architecture lookup returns "No architecture entry found for: …", treat that as `skipped` (unregistered module is a normal state for tools/scripts), NOT as `failed`, so the fingerprint stage is not gated off. **Gated symmetrically with `run_report` and `fingerprint`**: if `prompt` or `tags` reported `failed`, mark `architecture` as `skipped` with the failing stage as the reason and do NOT invoke `update_architecture_from_prompt` — that call writes architecture.json on success, and running it after an earlier hard failure means the architecture entry can be modified on disk before the overall result reports `failed`, which on the push-to-main auto-heal path can land via the heal-commit's scoped `git add -u` even though downstream stages correctly stop. 4. `run_report`: clear or refresh stale run reports tied to this prompt/code pair (only the entries this sync invalidates — do not wipe unrelated entries). Build a `paths` hint from the known file paths — `{"prompt": prompt_path}` plus `"code": code_path` when `code_path` is set — and pass it as `paths=` to `clear_run_report` so that `.pdd/meta` is anchored at the subproject root rather than the run CWD (supports parent-directory workflows, issue #1211). **Gated symmetrically with `fingerprint`**: if `prompt`, `tags`, or `architecture` reported `failed`, mark `run_report` as `skipped` with the failing stage as the reason and do NOT invoke `clear_run_report` — the deletion is an on-disk side effect that, left to run after an earlier hard failure, would create partial `.pdd/meta` state that auto-heal's scoped staging (`git add -u` plus per-module fingerprint pathspecs) could still pick up on the push-to-main path. - 5. `fingerprint`: finalize fingerprint state LAST via the fingerprint primitives from `pdd.sync_determine_operation` / `pdd.operation_log`. Pass the same `paths` hint (built from `prompt_path`/`code_path`) to `read_fingerprint` and `save_fingerprint` so fingerprint reads and writes anchor at the subproject `.pdd/meta` directory (issue #1211). Gated on no prior stage reporting `failed` — `skipped` and `dry_run` are acceptable upstream statuses (matches `MetadataSyncResult.ok`). If any earlier stage failed, mark fingerprint as `skipped` with the failing stage as the reason. This prevents storing a fingerprint that records "in sync" when an earlier metadata layer is broken. + 5. `fingerprint`: finalize fingerprint state LAST via `FingerprintTransaction` directly (not the `save_fingerprint` wrapper). Pass the same `paths` hint (built from `prompt_path`/`code_path`) so fingerprint reads and writes anchor at the subproject `.pdd/meta` directory (issue #1211). Gated on no prior stage reporting `failed` — `skipped` and `dry_run` are acceptable upstream statuses (matches `MetadataSyncResult.ok`). If any earlier stage failed, mark fingerprint as `skipped` with the failing stage as the reason. This prevents storing a fingerprint that records "in sync" when an earlier metadata layer is broken. `FingerprintFinalizeError` is caught by the per-stage handler (Req 4) and reported as `{"status": "failed", "reason": str(exc)}` — this is the correct behavior for `metadata_sync` as an orchestrator that surfaces failures via `MetadataSyncResult.ok` rather than propagating them. 3. **Result type** `MetadataSyncResult` — a Pydantic v2 model with: - `prompt_path: Path` - `code_path: Optional[Path]` @@ -69,8 +70,11 @@ This module provides the shared `run_metadata_sync` entry point used by `pdd upd pdd/sync_determine_operation.py - pdd/operation_log.py + pdd/operation_log.py + + pdd/fingerprint_transaction.py + % Deliverables diff --git a/pdd/prompts/operation_log_python.prompt b/pdd/prompts/operation_log_python.prompt index 585fa146a1..6589412b98 100644 --- a/pdd/prompts/operation_log_python.prompt +++ b/pdd/prompts/operation_log_python.prompt @@ -1,5 +1,7 @@ Provides sync operation logging, fingerprints, run reports, and compression metadata storage. +fingerprint_transaction_python.prompt + { "type": "module", @@ -34,7 +36,8 @@ A shared logging infrastructure module for tracking all PDD operations. Provides 3. **Module Identity Inference** - Extract basename and language from prompt file paths. Reconstructs directory context relative to the configured prompts root, so `prompts/frontend/settings/page_typescriptreact.prompt` yields basename `frontend/settings/page` and language `typescriptreact`. Paths without a `prompts/` ancestor fall back to filename-only. Language is always normalized to lowercase. **Issue #1211**: when the nearest `.pddrc` defines a non-default `prompts_dir` (e.g. `"prompts/backend"`), `infer_module_identity` MUST anchor the subdir extraction on that configured root — not the literal `"prompts"` segment. Otherwise the decorator would emit basenames like `backend/services/foo` while `construct_paths`/`sync` emit `services/foo`, splitting fingerprint filenames into two (`backend_services_foo_*.json` vs `services_foo_*.json`) and silently hiding metadata from sync. Use a helper `_prompts_dir_for_prompt(prompt_path)` that finds the nearest `.pddrc` via `construct_paths._find_nearest_pddrc_for_file`, resolves the context (file-based first, then cwd-based), and returns its `prompts_dir`. Fall back to the literal `prompts/` anchor when no `.pddrc` or no `prompts_dir` is set. -4. **State File Management** - Save fingerprints (`.json`), run reports (`_run.json`), and sync logs (`_sync.log`) to `.pdd/meta/`. All paths use sanitized basenames and `language.lower()`. Stale run reports are cleared only after the invalidating operation succeeds, and must be cleared before `save_fingerprint` so a fresh fingerprint never coexists with a stale per-module run report (issue #1057). The decorator must verify that a pre-existing run report is actually gone before saving a fingerprint; if deletion cannot be verified, warn and skip `save_fingerprint`. A failed command must never erase existing runtime verification state. +4. **State File Management** - Save fingerprints (`.json`), run reports (`_run.json`), and sync logs (`_sync.log`) to `.pdd/meta/`. All paths use sanitized basenames and `language.lower()`. Stale run reports are cleared only after the invalidating operation succeeds, and must be cleared before `save_fingerprint` so a fresh fingerprint never coexists with a stale per-module run report (issue #1057). The decorator must verify that a pre-existing run report is actually gone before saving a fingerprint; if deletion cannot be verified, call `transaction.skip(reason="run report not cleared")` rather than warning and skipping ad-hoc. `FingerprintFinalizeError` propagates to the Click boundary (non-zero exit). A failed command must never erase existing runtime verification state. + - `save_fingerprint` is a thin public wrapper that constructs a `FingerprintTransaction` and commits it. **`FingerprintFinalizeError` propagates to callers — fingerprint write failures are command failures, not warnings.** Callers needing skip control (dry-run, redirected output) construct `FingerprintTransaction` directly and call `transaction.skip(reason=...)`. - `save_fingerprint` must write the full `Fingerprint` dataclass format (from `sync_determine_operation`) using `calculate_current_hashes` so that `read_fingerprint` can parse it. This ensures manual commands (generate, example) don't break sync's fingerprint tracking. Pass stored `include_deps` from previous fingerprint for prompt hash calculation. - Metadata path resolution (issue #1211): `ensure_meta_dir`, `get_log_path`, `get_fingerprint_path`, and `get_run_report_path` all accept an optional `project_root` argument that anchors `.pdd/meta` under that root, plus an optional `paths` dict (the same `paths` flowing through `save_fingerprint`). Resolution order when `project_root` is omitted: (1) walk up from each file in `paths` until a `.pddrc` is found — this is what handles the case where the subproject .pddrc lives BELOW the run CWD; (2) walk up from CWD; (3) fall back to CWD. `save_fingerprint`, `save_run_report`, `clear_run_report`, `_clear_run_report_before_fingerprint`, and `append_log_entry` all accept `paths` and thread it through. The `@log_operation` decorator captures the inferred `prompt_file` path as an anchoring hint and passes a `paths` dict to every metadata write it makes (append_log_entry, save_fingerprint, save_run_report, _clear_run_report_before_fingerprint) so a decorated command run from a parent CWD still anchors its sync log, fingerprint, and run report at the subproject's `.pdd/meta`. **Issue #1305 / #1211**: for the `save_fingerprint` call specifically, the decorator MUST pass the authoritative, complete file-path set — `Path` objects keyed by `prompt`/`code`/`example`/`test` — and NOT a bare `{"prompt": prompt_file}` hint. `calculate_current_hashes` only hashes values that are `Path` instances, so a raw string prompt value plus absent code/example/test keys yields all-null `prompt_hash/code_hash/example_hash/test_hash`; that null fingerprint never matches the files on disk, so it never converges and CI auto-heal re-runs `pdd example`/`generate` on every pass, rewriting the generated example and overwriting maintainer cleanup. Resolve that set via `get_pdd_file_paths`, but anchor it at the prompt file's subproject by passing an absolute `prompts_dir` pointing at the prompts-root directory that contains this `prompt_file`: `get_pdd_file_paths(basename, language, prompts_dir=)`. This anchoring is REQUIRED because `get_pdd_file_paths` otherwise re-resolves the prompts root from the run CWD (`_find_pddrc_file()` walks up from CWD): a command invoked from a PARENT CWD for a nested subproject (the #1211 case — subproject `.pddrc` BELOW the run CWD) would then resolve to the parent, fail to find the prompt, point code/example/test at non-existent parent files (all-null hashes again), AND write the fingerprint to the parent `.pdd/meta` — splitting it from the sync log/run report that the prompt-path hint still anchors at the subproject. The absolute prompts root must be the BASE prompts directory (e.g. `/prompts`) — the `prompts` component nearest the prompt file, the same root `get_pdd_file_paths` uses by default — NOT a deeper, context-configured `prompts_dir` such as `prompts/commands`. `get_pdd_file_paths` searches for the prompt recursively under this root and computes `architecture.json` lookup keys *relative to it*, so a deeper root changes that key (`checkup_python.prompt` instead of `commands/checkup_python.prompt`) and silently breaks filepath resolution for subdir-context modules (e.g. `pdd/commands/checkup.py`). Compute it by taking the path up to and including the `prompts` component nearest the prompt file (resolved absolute), falling back to the prompt file's own directory when the path has no `prompts` component. Once the prompt resolves to its real subproject path, `construct_paths` detects the subproject `.pddrc` lives below the CWD and anchors code/example/test inside the subproject too, so all four hashes are real and the fingerprint lands in the subproject's `.pdd/meta`. Passing the full, subproject-anchored `Path` dict produces real, non-null hashes for both the `example` and `generate` operations and lets a second auto-heal pass on an unchanged tree detect no drift. Existence-gate the result: if resolution silently produced a `prompt` path that is not on disk (a mis-resolution that did not raise), the other paths are wrong too, so fall back to `{"prompt": Path(prompt_file)}` so `prompt_hash` is still real and the write anchors at the subproject. Likewise, if prompts-root computation or `get_pdd_file_paths` resolution raises, fall back to `{"prompt": Path(prompt_file)}` (a coerced `Path`, never a raw string) so anchoring still works. The lighter prompt-path hint may still be used for the non-fingerprint writes (append_log_entry/save_run_report/clear) purely for anchoring. Issue #983 contract preserved: when the caller passes a non-empty `paths`, `save_fingerprint` never calls `get_pdd_file_paths` — the decorator resolves the full path set itself before the call. Expose `_detect_project_root` and `_detect_project_root_from_paths` helpers for cross-module reuse by `sync_determine_operation.get_meta_dir` / `read_fingerprint` / `read_run_report`. @@ -44,9 +47,9 @@ A shared logging infrastructure module for tracking all PDD operations. Provides % The @log_operation Decorator A decorator for CLI commands that automatically logs operations and manages state files. Parameters control which state files are updated: -- `updates_fingerprint` - Save fingerprint on success +- `updates_fingerprint` - Save fingerprint on success via `FingerprintTransaction`; `FingerprintFinalizeError` propagates to Click boundary (non-zero exit) - `updates_run_report` - Save run report on success (if result is a dict) -- `clears_run_report` - Clear stale run report only after the wrapped command succeeds, before `save_fingerprint`. Verify a pre-existing run report was removed; if it remains or cannot be inspected, warn and skip fingerprint saving. A failed command must not clear the existing run report (issue #1057). +- `clears_run_report` - Clear stale run report only after the wrapped command succeeds, before `FingerprintTransaction.__exit__`. Verify a pre-existing run report was removed; if it remains or cannot be inspected, call `transaction.skip(reason="run report not cleared")` rather than warning and skipping ad-hoc. `FingerprintFinalizeError` propagates to Click boundary. A failed command must not clear the existing run report (issue #1057). The decorator must: - Work with Click commands @@ -59,6 +62,9 @@ The decorator must: context/sync_determine_operation_example.py + + pdd/fingerprint_transaction.py + % Deliverables diff --git a/pdd/prompts/sync_main_python.prompt b/pdd/prompts/sync_main_python.prompt index 7343df20bd..8c1d5ae754 100644 --- a/pdd/prompts/sync_main_python.prompt +++ b/pdd/prompts/sync_main_python.prompt @@ -15,6 +15,7 @@ code_generator_main_python.prompt agentic_sync_runner_python.prompt compressed_sync_context_python.prompt +fingerprint_transaction_python.prompt # sync_main_python.prompt % You are an expert Python engineer. Your goal is to write a Python function, 'sync_main', that will be the CLI wrapper for the sync command. This function handles interface logic (environment variables, parameter validation, path construction) and calls sync_orchestration to perform the actual sync workflow. @@ -124,7 +125,7 @@ - Merge costs: add `pre_cost` (including failed conformance attempts and the final successful generate attempt) to the one-session result's `total_cost`; pass only `remaining_budget - pre_cost` into the one-session phase. - Include snapshot status and artifact paths from generation in the one-session summary/result when available, without displaying sensitive artifact contents or forbidden detail strings. - If code generation fails (code file still missing), return failure without proceeding to one-session - - **Post-sync**: Save fingerprint via `save_fingerprint()` so next sync sees files as up-to-date + - **Post-sync**: Save fingerprint via `FingerprintTransaction` (not the `save_fingerprint` wrapper) so next sync sees files as up-to-date; `FingerprintFinalizeError` propagates to Click (non-zero exit) — one-session sync MUST NOT catch it. - **Auto-submit**: On success and not local mode, call `_auto_submit_example()` to submit the example to PDD Cloud 9. **Auto-Submit Example** (`_auto_submit_example` helper): diff --git a/pdd/prompts/sync_orchestration_python.prompt b/pdd/prompts/sync_orchestration_python.prompt index 638faa76ad..d9a92d9190 100644 --- a/pdd/prompts/sync_orchestration_python.prompt +++ b/pdd/prompts/sync_orchestration_python.prompt @@ -14,6 +14,7 @@ code_generator_main_python.prompt agentic_sync_runner_python.prompt compressed_sync_context_python.prompt +fingerprint_transaction_python.prompt # sync_orchestration_python.prompt @@ -183,6 +184,7 @@ from .operation_log import ( save_run_report, clear_run_report, ) +from .fingerprint_transaction import FingerprintTransaction, FingerprintFinalizeError from .sync_determine_operation import ( sync_determine_operation, get_pdd_file_paths, @@ -289,9 +291,11 @@ Detect headless environments (no TTY, CI environment variable, or `quiet=True`) Use `AtomicStateUpdate` context manager for consistent state writes: - Ensures run_report and fingerprint are both written or neither is written - Uses temp file + atomic rename pattern for crash safety -- After each successful operation, save a `Fingerprint` to disk with current file hashes +- After each successful operation, save a `Fingerprint` to disk with current file hashes via `FingerprintTransaction`. `FingerprintFinalizeError` propagates through the operation dispatch to Click; it MUST NOT be caught by per-operation `except` handlers. +- **No-op fix handling**: for no-op fix operations (detected before finalization — see "Return Value Parsing" below), construct a `FingerprintTransaction` and call `transaction.skip(reason="no-op fix — no LLM invocation")` instead of omitting the transaction entirely, to preserve the single-code-path invariant. +- The fingerprint-write leg of `AtomicStateUpdate` and `_save_fingerprint_atomic` MUST route through `FingerprintTransaction`. `_save_fingerprint_atomic` becomes a thin wrapper that constructs and enters a `FingerprintTransaction`. - **Case-insensitive language in paths**: All metadata file paths (fingerprint `.json`, run report `_run.json`, sync log `_sync.log`) must use `language.lower()` for consistent paths on case-sensitive filesystems. This applies to any direct path construction in this module (e.g., `f"{_safe_basename(basename)}_{language.lower()}_run.json"`). -- **Subproject anchoring (issue #1211)**: never compose `META_DIR / ` directly for runtime writes — the module-level `META_DIR` is CWD-resolved at import time and won't follow a subproject `.pddrc` that lives below the run CWD. Instead, route fingerprint/run-report/log paths through `get_fingerprint_path(...)`, `get_run_report_path(...)`, and `get_log_path(...)` (from `operation_log`) and pass `paths=pdd_files` so the helper resolves the meta dir via upward `.pddrc` detection from those paths. `_save_run_report_atomic` and `_save_fingerprint_atomic` both accept `paths` and forward them into the atomic-state file path AND into `read_fingerprint` for stored-deps lookup. Every `read_run_report` / `clear_run_report` / `log_event` / `append_log_entry` call inside `sync_orchestration` must pass `paths=pdd_files` once `pdd_files` is in scope — including the post-`pdd_files` branches in coverage retry, crash-detection, generate logic, lifecycle events (sync_start, lock_acquired, lock_released, budget_warning/exceeded, cycle_detected, steering_override, etc.), and per-operation log_entry appends. The dry-run / `_display_sync_log` branch executes BEFORE `pdd_files` is set, so it does a best-effort `get_pdd_file_paths(...)` lookup (wrapped in try/except) and forwards the result via `paths=` so log display still finds the subproject log when invoked from a parent CWD. Mismatched paths leave stale state behind and let stale parent metadata leak into decision logic. The only acceptable `META_DIR` reference outside of these helpers is the `if __name__ == '__main__':` demo block at the end of the file. +- **Subproject anchoring (issue #1211)**: never compose `META_DIR / ` directly for runtime writes — the module-level `META_DIR` is CWD-resolved at import time and won't follow a subproject `.pddrc` that lives below the run CWD. Instead, route fingerprint/run-report/log paths through `get_fingerprint_path(...)`, `get_run_report_path(...)`, and `get_log_path(...)` (from `operation_log`) and pass `paths=pdd_files` so the helper resolves the meta dir via upward `.pddrc` detection from those paths. `_save_run_report_atomic` both accepts `paths` and forwards it into the atomic-state file path AND into `read_fingerprint` for stored-deps lookup. Every `read_run_report` / `clear_run_report` / `log_event` / `append_log_entry` call inside `sync_orchestration` must pass `paths=pdd_files` once `pdd_files` is in scope — including the post-`pdd_files` branches in coverage retry, crash-detection, generate logic, lifecycle events (sync_start, lock_acquired, lock_released, budget_warning/exceeded, cycle_detected, steering_override, etc.), and per-operation log_entry appends. The dry-run / `_display_sync_log` branch executes BEFORE `pdd_files` is set, so it does a best-effort `get_pdd_file_paths(...)` lookup (wrapped in try/except) and forwards the result via `paths=` so log display still finds the subproject log when invoked from a parent CWD. Mismatched paths leave stale state behind and let stale parent metadata leak into decision logic. The only acceptable `META_DIR` reference outside of these helpers is the `if __name__ == '__main__':` demo block at the end of the file. ### Skip Handling @@ -371,7 +375,7 @@ Use these functions from `operation_log` module (already imported above): - `update_log_entry(entry, success, cost, model, duration, error)` - Add execution results to log entry - `append_log_entry(basename, language, entry, paths=None)` - Append entry to log file; pass `paths=pdd_files` so the log anchors at the subproject `.pdd/meta` (issue #1211) - `log_event(basename, language, event_type, details, invocation_mode, paths=None)` - Log system events (lock_acquired, budget_warning, etc.); pass `paths=pdd_files` for subproject anchoring -- `save_fingerprint(basename, language, operation, paths=None, cost, model)` - Save fingerprint after success; pass `paths=pdd_files` for subproject anchoring +- `save_fingerprint(basename, language, operation, paths=None, cost, model)` - Thin wrapper around `FingerprintTransaction`; `FingerprintFinalizeError` propagates. Pass `paths=pdd_files` for subproject anchoring. Use `FingerprintTransaction` directly when skip control is needed. - `save_run_report(basename, language, report_data, paths=None)` - Save RunReport dict; pass `paths=pdd_files` for subproject anchoring (issue #1211) - `clear_run_report(basename, language, paths=None)` - Remove stale run report; pass `paths=pdd_files` for subproject anchoring @@ -441,7 +445,7 @@ Handle command results in multiple formats: - For test operation failure: if `result[4]` (error_message) is non-empty, append it to the errors list so it reaches the sync log - For crash/fix operations returning tuples: extract error message from `result[1]` on failure and append to errors list - For `ArchitectureConformanceError` exception paths, extract `total_cost` and `model_name` from the exception into a result-shaped dict before logging so failed conformance attempts do not report `$0.00`. -- Save fingerprint after successful operations regardless of format, **except** for no-op fix operations. A no-op fix is detected when `operation == 'fix'` and `actual_cost == 0.0` and `model_name` is empty/`'none'`/`'unknown'`/`'n/a'`. This check **must run before** `_save_fingerprint_atomic` so that a logical failure (consecutive no-op breaker) does not persist stale "fix completed" state. Do not call `_save_fingerprint_atomic` for no-op fixes. +- For fingerprint finalization after successful operations: construct a `FingerprintTransaction` for every operation regardless of format. **For no-op fix operations**, call `transaction.skip(reason="no-op fix — no LLM invocation")` instead of skipping the transaction entirely. A no-op fix is detected when `operation == 'fix'` and `actual_cost == 0.0` and `model_name` is empty/`'none'`/`'unknown'`/`'n/a'`. This check **must run before** `FingerprintTransaction.__exit__` so that a logical failure (consecutive no-op breaker) does not persist stale "fix completed" state. ### Non-Python Language Handling and Agentic Mode diff --git a/pdd/prompts/update_main_python.prompt b/pdd/prompts/update_main_python.prompt index 7e53ef339b..0c2ba06d4e 100644 --- a/pdd/prompts/update_main_python.prompt +++ b/pdd/prompts/update_main_python.prompt @@ -38,6 +38,7 @@ agentic_update_python.prompt sync_determine_operation_python.prompt operation_log_python.prompt +fingerprint_transaction_python.prompt % Goal Write `pdd/update_main.py` — the CLI wrapper for updating prompts based on modified code, featuring repository-wide scanning, agentic routing, and post-update synchronization. @@ -66,15 +67,16 @@ Supports three modes: true update, regeneration, and repository-wide updates. Ro - **Failure reporting**: When any stage in `MetadataSyncResult` is not `ok`, print a Rich error line per failed stage with the stage name and reason. The `metadata` column makes it obvious which layer is incomplete. - **Backward compatibility**: When `sync_metadata=False` (default), behavior is unchanged for repo mode — it keeps its current post-update fingerprint and architecture-sync calls. Single-file and regeneration modes still run the default fingerprint finalization defined in Requirement 15 (this is the only metadata write they perform by default). 15. Default fingerprint finalization for single-file and regeneration modes: - - After a successful single-file or regeneration update, after the prompt has been written and before returning a non-None `(prompt, cost, model)` tuple, write a current fingerprint for the affected `(prompt, code)` pair. - - Reuse `infer_module_identity(prompt_path)` and `save_fingerprint(...)` from `pdd.operation_log`; do not introduce a parallel fingerprint implementation. Pass `paths={"prompt": Path(prompt_path), "code": Path(code_path)}` (omitting `"code"` when code path is not available) to `save_fingerprint` so `.pdd/meta` anchors at the subproject root when running from a parent directory (issue #1211). + - After a successful single-file or regeneration update, after the prompt has been written and before returning a non-None `(prompt, cost, model)` tuple, write a current fingerprint for the affected `(prompt, code)` pair using an explicit `FingerprintTransaction`. + - Reuse `infer_module_identity(prompt_path)` from `pdd.operation_log`; construct `FingerprintTransaction(basename, language, operation="update", paths={"prompt": Path(prompt_path), "code": Path(code_path)}, cost=cost, model=model)` (omitting `"code"` when code path is not available) so `.pdd/meta` anchors at the subproject root when running from a parent directory (issue #1211). - Skip this default finalization in repo mode because repo mode already performs post-update fingerprinting. - - If `sync_metadata=True`, do not double-write and print `[info][metadata] Skipping fingerprint finalization: orchestrator owns fingerprint stage[/info]` unless `quiet`. - - If `dry_run=True`, do not finalize metadata and print `[info][metadata] Skipping fingerprint finalization: dry-run mode[/info]` unless `quiet`. - - If the written prompt path differs from the canonical source prompt (i.e. `--output` redirected the write away from the input prompt in true-update mode), do not finalize metadata and print `[info][metadata] Skipping fingerprint finalization: output redirected[/info]` unless `quiet`. This guard applies in addition to (and independently of) the `sync_metadata=True` skip — when both are set, the orchestrator call itself is also skipped per Requirement 14 so neither layer records a fingerprint against the redirected path. - - If `infer_module_identity(prompt_path)` returns `(None, None)`, do not finalize metadata and print `[info][metadata] Skipping fingerprint finalization: unable to infer module identity for [/info]` unless `quiet`. - - If `save_fingerprint` raises, keep the successful return tuple and print `[warning][metadata] Fingerprint save failed: [/warning]` unless `quiet`. - - Stale `_run.json` cleanup MUST reuse `pdd.operation_log._clear_run_report_before_fingerprint(basename, language, paths={"prompt": Path(prompt_path), "code": Path(code_path)})` (omitting `"code"` when unavailable), which clears the run report then re-checks existence and returns `False` (with a yellow console warning) when a silent `os.remove` left it on disk. When the helper returns `False`, skip `save_fingerprint` so a fresh fingerprint never coexists with stale runtime state (issue #1106; mirrors repo-mode and the `log_operation` decorator). The helper's warning surfaces unconditionally — it does NOT honour the caller's `quiet` flag, intentionally, because it describes a real metadata problem. Wrap the helper call in `try/except`: on an unexpected exception, emit `[warning][metadata] Run report clear failed: [/warning]` unless `quiet` and skip `save_fingerprint` (best-effort cleanup must never break the successful update tuple). The function-local `from .operation_log import (_clear_run_report_before_fingerprint, infer_module_identity, save_fingerprint)` MUST also be wrapped in `try/except ImportError`: on ImportError, emit `[warning][metadata] Could not import finalization helpers: [/warning]` unless `quiet` and return without calling `save_fingerprint`. This is required because `_clear_run_report_before_fingerprint` is a private underscore-prefixed symbol and therefore more fragile to internal `operation_log` renames than the public names alongside it — without the import guard, an ImportError would propagate to `update_main`'s outer `except Exception: return None` and silently convert a successful `(prompt, cost, model)` tuple to `None`, violating the "best-effort metadata cleanup must never break the successful update tuple" contract. + - Skip conditions — call `transaction.skip(reason=...)` rather than returning early from the transaction: + - If `sync_metadata=True`: `transaction.skip(reason="orchestrator owns fingerprint stage")`; print `[info][metadata] Skipping fingerprint finalization: orchestrator owns fingerprint stage[/info]` unless `quiet`. + - If `dry_run=True`: `transaction.skip(reason="dry-run mode")`; print `[info][metadata] Skipping fingerprint finalization: dry-run mode[/info]` unless `quiet`. + - If the written prompt path differs from the canonical source prompt (output redirected): `transaction.skip(reason="output redirected")`; print `[info][metadata] Skipping fingerprint finalization: output redirected[/info]` unless `quiet`. This guard applies in addition to (and independently of) the `sync_metadata=True` skip. + - If `infer_module_identity(prompt_path)` returns `(None, None)`: `transaction.skip(reason="unable to infer module identity")`; print `[info][metadata] Skipping fingerprint finalization: unable to infer module identity for [/info]` unless `quiet`. + - `FingerprintFinalizeError` propagates — the successful return tuple is NOT preserved on fingerprint write failure; Click exits non-zero. This is the enforced commit-or-fail contract (issue #1926). + - Stale `_run.json` cleanup MUST reuse `pdd.operation_log._clear_run_report_before_fingerprint(basename, language, paths={"prompt": Path(prompt_path), "code": Path(code_path)})` (omitting `"code"` when unavailable), which clears the run report then re-checks existence and returns `False` (with a yellow console warning) when a silent `os.remove` left it on disk. When the helper returns `False`, call `transaction.skip(reason="run report not cleared")` so a fresh fingerprint never coexists with stale runtime state (issue #1106). The helper's warning surfaces unconditionally — it does NOT honour the caller's `quiet` flag, intentionally, because it describes a real metadata problem. Wrap the helper call in `try/except`: on an unexpected exception, emit `[warning][metadata] Run report clear failed: [/warning]` unless `quiet` and call `transaction.skip(reason="run report clear failed")` (best-effort cleanup must never block the transaction context). The function-local `from .operation_log import (_clear_run_report_before_fingerprint, infer_module_identity)` and `from .fingerprint_transaction import FingerprintTransaction, FingerprintFinalizeError` MUST be wrapped in `try/except ImportError`: on ImportError, emit `[warning][metadata] Could not import finalization helpers: [/warning]` unless `quiet` and return without entering the transaction. % Modes 1. **True update**: `input_prompt_file` + `modified_code_file` + (use_git OR `input_code_file`). From 3295ac99a8adbe85e00e1d4f214bddc82fc9785c Mon Sep 17 00:00:00 2001 From: Serhan Date: Fri, 10 Jul 2026 17:59:56 -0700 Subject: [PATCH 2/5] fix: enforce transactional fingerprint finalization --- README.md | 2 +- architecture.json | 28 ++- context/fingerprint_transaction_example.py | 38 +++ pdd/auto_deps_main.py | 113 ++++----- pdd/ci_drift_heal.py | 23 +- pdd/commands/fix.py | 2 +- pdd/commands/maintenance.py | 4 + pdd/commands/modify.py | 4 + pdd/fingerprint_transaction.py | 191 ++++++++++++++ pdd/metadata_sync.py | 16 +- pdd/operation_log.py | 87 +++---- pdd/pin_example_hack.py | 42 ++-- pdd/prompts/auto_deps_main_python.prompt | 8 +- pdd/prompts/commands/fix_python.prompt | 2 +- .../commands/maintenance_python.prompt | 2 + pdd/prompts/commands/modify_python.prompt | 3 +- pdd/prompts/metadata_sync_python.prompt | 4 +- pdd/prompts/update_main_python.prompt | 4 +- pdd/sync_main.py | 20 +- pdd/sync_orchestration.py | 98 ++++---- pdd/update_main.py | 235 ++++++------------ tests/test_auto_deps_main.py | 66 ++--- tests/test_ci_drift_heal.py | 2 +- tests/test_fingerprint_invariant.py | 118 +++++++++ tests/test_fingerprint_transaction.py | 164 ++++++++++++ tests/test_metadata_sync.py | 25 +- tests/test_operation_log.py | 40 ++- tests/test_update_main.py | 201 ++++++++------- 28 files changed, 1000 insertions(+), 542 deletions(-) create mode 100644 context/fingerprint_transaction_example.py create mode 100644 pdd/fingerprint_transaction.py create mode 100644 tests/test_fingerprint_invariant.py create mode 100644 tests/test_fingerprint_transaction.py diff --git a/README.md b/README.md index fb820fea3d..fa4e87423e 100644 --- a/README.md +++ b/README.md @@ -2756,7 +2756,7 @@ Options: - `--git`: Use git history to find the original code file, eliminating the need for the `INPUT_CODE_FILE` argument. - `--extensions EXTENSIONS`: In repository-wide mode, filter the update to only include files with the specified comma-separated extensions (e.g., `py,js,ts`). - `--simple`: Use the legacy 2-stage LLM update process instead of the default agentic mode. Useful when agentic CLIs are not available or for faster updates. -- `--sync-metadata`: After the prompt update, run the shared metadata-sync orchestrator so prompt PDD tags, `architecture.json` entries, run reports, and fingerprint state are reconciled in one step. Works in single-file, regeneration, and repo modes. **Fingerprint note:** default single-file/regeneration `pdd update ` already finalizes the per-target fingerprint (`.pdd/meta/_.json`) on success, and logs a skip reason when finalization is intentionally bypassed; `--sync-metadata` does not gate that behavior. Before writing the single-file/regeneration fingerprint, the command clears the affected module's stale `.pdd/meta/__run.json` runtime-verification report and re-checks it is gone; if the stale report survives the clear attempt (for example, silent `os.remove` failure on permissions/locks), the fingerprint write is skipped with a yellow warning so a fresh fingerprint cannot coexist with stale runtime state — best-effort, the successful `(prompt, cost, model)` update tuple is preserved either way (closing issue [#1106](https://github.com/promptdriven/pdd/issues/1106)). The stale-report warning intentionally surfaces even under `--quiet`, because it describes a real metadata-consistency problem the operator should learn about regardless of other log suppression. Default repo-mode `pdd update --repo` likewise finalizes per-pair fingerprints and, before writing each fingerprint, clears the affected module's stale `.pdd/meta/__run.json` runtime-verification report so metadata and runtime state stay in lock-step (clear failures are surfaced as non-fatal warnings, and if the stale `_run.json` still exists after the clear attempt the fingerprint write is skipped so a fresh fingerprint cannot coexist with stale runtime state — closing issue [#1057](https://github.com/promptdriven/pdd/issues/1057)). Without this flag, the broader prompt-tag/architecture/run-report orchestrator is not run and those layers must be reconciled with separate commands. **Scope note:** the `tags` stage currently *preserves* existing PDD tags and only *seeds* tags from the matching `architecture.json` entry when a prompt has none — LLM-first **refresh** of stale-but-present tags is tracked at issue [#870](https://github.com/promptdriven/pdd/issues/870) and is not invoked by this orchestrator. When a prompt has zero PDD tags AND no architecture entry, the `tags` stage reports `skipped` (never `ok`) so operators see honest status. On any stage `failed`, `pdd update --sync-metadata` exits non-zero so CI auto-heal does not treat a half-finalized update as healed. +- `--sync-metadata`: After the prompt update, run the shared metadata-sync orchestrator so prompt PDD tags, `architecture.json` entries, run reports, and fingerprint state are reconciled in one step. Works in single-file, regeneration, and repo modes. **Fingerprint note:** default single-file/regeneration and repo updates already finalize affected fingerprints through `FingerprintTransaction`. Before commit, PDD clears and verifies removal of stale `_run.json` state. A cleanup, null-hash, or atomic-persist failure is a command failure with a non-zero exit; PDD never reports a successful mutation whose required fingerprint was skipped (#1926). Intentional non-mutating paths such as dry-run, redirected output, or orchestrator-owned finalization use an explicit transaction skip. Without `--sync-metadata`, the broader prompt-tag/architecture orchestrator is not run. **Scope note:** the tags stage preserves existing PDD tags and only seeds tags from a matching architecture entry when none exist; LLM-first refresh of stale tags remains tracked by #870. Any failed metadata stage exits non-zero. Example (Metadata Sync): ```bash diff --git a/architecture.json b/architecture.json index 382fa53c5b..50e2710dd0 100644 --- a/architecture.json +++ b/architecture.json @@ -1589,7 +1589,8 @@ "insert_includes_python.prompt", "validate_prompt_includes_python.prompt", "operation_log_python.prompt", - "auto_deps_architecture_python.prompt" + "auto_deps_architecture_python.prompt", + "fingerprint_transaction_python.prompt" ], "priority": 35, "filename": "auto_deps_main_python.prompt", @@ -2579,7 +2580,8 @@ "agentic_common_python.prompt", "agentic_update_python.prompt", "sync_determine_operation_python.prompt", - "operation_log_python.prompt" + "operation_log_python.prompt", + "fingerprint_transaction_python.prompt" ], "priority": 62, "filename": "update_main_python.prompt", @@ -3116,7 +3118,8 @@ "dependencies": [ "code_generator_main_python.prompt", "agentic_sync_runner_python.prompt", - "compressed_sync_context_python.prompt" + "compressed_sync_context_python.prompt", + "fingerprint_transaction_python.prompt" ], "priority": 70, "filename": "sync_orchestration_python.prompt", @@ -5489,7 +5492,8 @@ "core/cloud_python.prompt", "code_generator_main_python.prompt", "agentic_sync_runner_python.prompt", - "compressed_sync_context_python.prompt" + "compressed_sync_context_python.prompt", + "fingerprint_transaction_python.prompt" ], "priority": 148, "filename": "sync_main_python.prompt", @@ -5552,7 +5556,9 @@ { "reason": "CLI entry point for the change command.", "description": "Command-line interface for change. Parses arguments, orchestrates the workflow, and formats output.", - "dependencies": [], + "dependencies": [ + "fingerprint_transaction_python.prompt" + ], "priority": 150, "filename": "change_main_python.prompt", "filepath": "pdd/change_main.py", @@ -8586,7 +8592,8 @@ "description": "Contains maintenance and setup commands for the PDD CLI, including single-module sync, no-argument project-wide sync, GitHub issue sync dispatch, prompt-to-architecture sync, auto-deps, and setup.", "dependencies": [ "auto_deps_main_python.prompt", - "architecture_sync_python.prompt" + "architecture_sync_python.prompt", + "fingerprint_transaction_python.prompt" ], "priority": 70, "filename": "commands/maintenance_python.prompt", @@ -8631,7 +8638,8 @@ "agentic_split_python.prompt", "change_main_python.prompt", "agentic_change_python.prompt", - "update_main_python.prompt" + "update_main_python.prompt", + "fingerprint_transaction_python.prompt" ], "priority": 70, "filename": "commands/modify_python.prompt", @@ -8813,7 +8821,8 @@ "auto_include_python.prompt", "agentic_langtest_python.prompt", "architecture_sync_python.prompt", - "metadata_sync_python.prompt" + "metadata_sync_python.prompt", + "fingerprint_transaction_python.prompt" ], "priority": 80, "filename": "ci_drift_heal_python.prompt", @@ -9656,7 +9665,8 @@ "dependencies": [ "architecture_sync_python.prompt", "sync_determine_operation_python.prompt", - "operation_log_python.prompt" + "operation_log_python.prompt", + "fingerprint_transaction_python.prompt" ], "priority": 5, "filename": "metadata_sync_python.prompt", diff --git a/context/fingerprint_transaction_example.py b/context/fingerprint_transaction_example.py new file mode 100644 index 0000000000..7e28dc1bac --- /dev/null +++ b/context/fingerprint_transaction_example.py @@ -0,0 +1,38 @@ +"""Minimal local example for :mod:`pdd.fingerprint_transaction`.""" + +from __future__ import annotations + +import json +import tempfile +from pathlib import Path + +from pdd.fingerprint_transaction import FingerprintTransaction + + +def run_fingerprint_transaction_example() -> dict[str, object]: + """Finalize a tiny unit and return its persisted fingerprint payload.""" + + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / ".pddrc").write_text("{}", encoding="utf-8") + prompt = root / "prompts" / "hello_python.prompt" + code = root / "src" / "hello.py" + prompt.parent.mkdir(parents=True) + code.parent.mkdir(parents=True) + prompt.write_text("Write a hello function.\n", encoding="utf-8") + code.write_text("def hello(): return 'hello'\n", encoding="utf-8") + + with FingerprintTransaction( + "hello", + "python", + "generate", + paths={"prompt": prompt, "code": code}, + ) as transaction: + destination = transaction.fingerprint_path + + return json.loads(destination.read_text(encoding="utf-8")) + + +if __name__ == "__main__": + payload = run_fingerprint_transaction_example() + print(payload["command"], payload["prompt_hash"], payload["code_hash"]) diff --git a/pdd/auto_deps_main.py b/pdd/auto_deps_main.py index 101697c4cb..19e8c1da68 100644 --- a/pdd/auto_deps_main.py +++ b/pdd/auto_deps_main.py @@ -12,9 +12,12 @@ from .insert_includes import insert_includes from .validate_prompt_includes import sanitize_prompt_output from .auto_deps_architecture import merge_auto_deps_includes_from_cwd +from .fingerprint_transaction import ( + FingerprintFinalizeError, + FingerprintTransaction, +) from .operation_log import ( infer_module_identity, - save_fingerprint, clear_run_report, get_run_report_path, ) @@ -188,8 +191,8 @@ def auto_deps_main( # written (``output_path``); in in-place mode this resolves to the # canonical ``_`` module, in the default CLI # flow it resolves to the ``__with_deps`` - # derivative. Errors are surfaced as warnings and never mask a - # successful auto-deps result. + # derivative. Finalization is part of command success: any + # failure is reported as a non-zero command result (#1926). # # ``_skip_finalization=True`` is set by ``pdd sync`` because it # invokes auto-deps with a temp ``__with_deps`` @@ -199,78 +202,56 @@ def auto_deps_main( # derivative's identity is the wrong target for run-report clears # (sync owns the canonical fingerprint write and the canonical # run-report clear on its side). - if _skip_finalization: - return cleaned_prompt, total_cost, model_name - try: - basename, language = infer_module_identity(Path(output_path)) - if basename is None or language is None: - if not quiet: - console.print( - f"[yellow]Warning: Could not infer module identity for " - f"{output_path}; skipping fingerprint finalization.[/yellow]" - ) - else: - # Issue #1211: route clear/verify/save through the same - # `paths` hint (the prompt path we just wrote) so all - # three target the subproject's .pdd/meta — not a parent - # CWD orphan — when auto-deps is run from above the - # subproject root. - _autodeps_paths = {"prompt": Path(output_path)} - # Clear stale run report; do not let its failure block fingerprint save - try: - clear_run_report(basename, language, paths=_autodeps_paths) - except Exception as cr_exc: - if not quiet: - console.print( - f"[yellow]Warning: Failed to clear run report for " - f"{basename}_{language}: {cr_exc}[/yellow]" - ) - # Defensive: clear_run_report() in pdd.operation_log silently swallows - # OSError on the actual unlink, so verify the report is really gone. - try: - _stale_report_path = get_run_report_path( - basename, language, paths=_autodeps_paths - ) - if _stale_report_path.exists(): - if not quiet: - console.print( - f"[yellow]Warning: clear_run_report did not remove " - f"{_stale_report_path}; downstream sync may still see " - f"stale results.[/yellow]" - ) - except Exception as _vrf_exc: - if not quiet: - console.print( - f"[yellow]Warning: could not verify run-report removal: " - f"{_vrf_exc}[/yellow]" - ) - try: - save_fingerprint( - basename=basename, - language=language, - operation="auto-deps", - paths=_autodeps_paths, - cost=total_cost, - model=model_name, - ) - except Exception as fp_exc: - if not quiet: - console.print( - f"[yellow]Warning: Failed to save fingerprint for " - f"{basename}_{language}: {fp_exc}[/yellow]" - ) - except Exception as meta_exc: - # Never mask a successful auto-deps result on metadata errors + basename, language = infer_module_identity(Path(output_path)) + if basename is None or language is None: if not quiet: console.print( - f"[yellow]Warning: Metadata finalization encountered an error: {meta_exc}[/yellow]" + f"[yellow]Warning: Could not infer module identity for " + f"{output_path}; skipping fingerprint finalization.[/yellow]" ) + else: + _autodeps_paths = {"prompt": Path(output_path)} + with FingerprintTransaction( + basename=basename, + language=language, + operation="auto-deps", + paths=_autodeps_paths, + cost=total_cost, + model=model_name, + ) as transaction: + if _skip_finalization: + transaction.skip("_skip_finalization set by sync") + else: + try: + clear_run_report( + basename, + language, + paths=_autodeps_paths, + ) + stale_report_path = get_run_report_path( + basename, + language, + paths=_autodeps_paths, + ) + except Exception as exc: + raise FingerprintFinalizeError( + f"[auto-deps] fingerprint finalization failed for " + f"{transaction.fingerprint_path}: run report cleanup: {exc}" + ) from exc + if stale_report_path.exists(): + raise FingerprintFinalizeError( + f"[auto-deps] fingerprint finalization failed for " + f"{transaction.fingerprint_path}: stale run report remains " + f"at {stale_report_path}" + ) return cleaned_prompt, total_cost, model_name except click.Abort: # Re-raise to allow orchestrators (e.g. pdd sync) to stop the loop raise + except FingerprintFinalizeError: + raise except Exception as exc: if not quiet: console.print(f"[red]Error in auto-deps: {exc}[/red]") diff --git a/pdd/ci_drift_heal.py b/pdd/ci_drift_heal.py index a87dd695d0..16ef4c27f2 100644 --- a/pdd/ci_drift_heal.py +++ b/pdd/ci_drift_heal.py @@ -792,7 +792,8 @@ def _run_metadata_sync_safe( ok = bool(getattr(result, "ok", False)) if ok: try: - from pdd.operation_log import infer_module_identity, save_fingerprint + from pdd.fingerprint_transaction import FingerprintTransaction + from pdd.operation_log import infer_module_identity from pdd.sync_determine_operation import read_fingerprint basename, language = infer_module_identity(str(p)) @@ -807,25 +808,15 @@ def _run_metadata_sync_safe( if code_p is None: raise ValueError("authoritative prompt/code paths unavailable: code_path not provided") paths: Dict[str, Any] = {"prompt": p, "code": code_p} - # Preserve the previous user-facing command so the released - # `sync_determine_operation._is_workflow_complete` (which only - # accepts verify/test/fix/update as complete) keeps recognizing - # the workflow as synced after this internal refresh. - prev_fp_for_cmd = read_fingerprint(basename, language, paths=paths) - prev_cmd = getattr(prev_fp_for_cmd, "command", None) if prev_fp_for_cmd else None - preserved_command = ( - prev_cmd - if prev_cmd in ("verify", "test", "fix", "update") - else "fix" - ) - save_fingerprint( + with FingerprintTransaction( basename=basename, language=language, - operation=preserved_command, + operation="update", paths=paths, cost=0.0, - model="metadata_sync", - ) + model="", + ): + pass fingerprint = read_fingerprint(basename, language, paths=paths) if ( fingerprint is None diff --git a/pdd/commands/fix.py b/pdd/commands/fix.py index a49dbd765f..5fe5a47ade 100644 --- a/pdd/commands/fix.py +++ b/pdd/commands/fix.py @@ -122,7 +122,7 @@ def _is_user_story_file(value: str) -> bool: help="Behavior when context compression fails (default: full).", ) @click.pass_context -@log_operation(operation="fix", clears_run_report=True) +@log_operation(operation="fix", clears_run_report=True, updates_fingerprint=True) @track_cost def fix( ctx: click.Context, diff --git a/pdd/commands/maintenance.py b/pdd/commands/maintenance.py index b8ca343784..7b662492b4 100644 --- a/pdd/commands/maintenance.py +++ b/pdd/commands/maintenance.py @@ -9,6 +9,7 @@ from ..architecture_sync import sync_prompts_to_architecture from ..sync_main import sync_main from ..auto_deps_main import auto_deps_main +from ..fingerprint_transaction import FingerprintFinalizeError from ..agentic_sync import _is_github_issue_url, run_agentic_sync, run_global_sync from ..construct_paths import _find_pddrc_file, _load_pddrc_config from ..track_cost import track_cost @@ -992,6 +993,9 @@ def auto_deps( return result, total_cost, model_name except click.Abort: raise + except FingerprintFinalizeError as exception: + handle_error(exception, "auto-deps", ctx.obj.get("quiet", False)) + raise click.exceptions.Exit(1) from exception except Exception as exception: handle_error(exception, "auto-deps", ctx.obj.get("quiet", False)) return None diff --git a/pdd/commands/modify.py b/pdd/commands/modify.py index 45e3f5dac7..0145814343 100644 --- a/pdd/commands/modify.py +++ b/pdd/commands/modify.py @@ -16,6 +16,7 @@ from ..agentic_change import run_agentic_change from ..update_main import update_main from ..track_cost import track_cost +from ..fingerprint_transaction import FingerprintFinalizeError from ..core.errors import handle_error from ..core.utils import echo_model_line from ..operation_log import log_operation @@ -634,6 +635,9 @@ def update( # update_main). Letting `except Exception` swallow it would silently # convert it to exit 0 and re-introduce the bug fixed for #871. raise + except FingerprintFinalizeError as e: + handle_error(e, "update", ctx.obj.get("quiet", False)) + raise click.exceptions.Exit(1) from e except Exception as e: handle_error(e, "update", ctx.obj.get("quiet", False)) return None diff --git a/pdd/fingerprint_transaction.py b/pdd/fingerprint_transaction.py new file mode 100644 index 0000000000..07c608bfd3 --- /dev/null +++ b/pdd/fingerprint_transaction.py @@ -0,0 +1,191 @@ +"""Transactional fingerprint finalization. + +This module is the one implementation that computes and persists PDD +fingerprints. ``os.replace`` is atomic on POSIX when source and destination +share a filesystem. On Windows, replacing an existing destination does not +have the same POSIX atomicity guarantee; this is a known platform limitation. +""" + +from __future__ import annotations + +import json +import logging +import os +import tempfile +from dataclasses import asdict +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Dict, Optional + +from . import __version__ +from .operation_log import get_fingerprint_path +from .sync_determine_operation import ( + Fingerprint, + calculate_current_hashes, + read_fingerprint, +) + + +logger = logging.getLogger(__name__) + + +class FingerprintFinalizeError(RuntimeError): + """Raised when a required fingerprint cannot be finalized.""" + + +class FingerprintTransaction: # pylint: disable=too-many-instance-attributes + """Compute and atomically persist one command fingerprint on clean exit.""" + + def __init__( # pylint: disable=too-many-arguments,too-many-positional-arguments + self, + basename: str, + language: str, + operation: str, + paths: Optional[Dict[str, Path]] = None, + cost: float = 0.0, + model: str = "", + ) -> None: + self._basename = basename + self._language = language.lower() + self._operation = operation + self._paths: Dict[str, Path] = dict(paths or {}) + self._cost = cost + self._model = model + # Resolve once. This prevents a cwd change during a command from moving + # the final write to a different project (#1211/#1290). + self._fingerprint_path = get_fingerprint_path( + basename, + self._language, + paths=self._paths, + ) + self._skipped = False + self._skip_reason: Optional[str] = None + self._include_deps_override: Optional[Dict[str, str]] = None + + @property + def fingerprint_path(self) -> Path: + """Return the eagerly-resolved destination path.""" + + return self._fingerprint_path + + def __enter__(self) -> "FingerprintTransaction": + return self + + def skip(self, reason: str) -> None: + """Suppress persistence for an intentional non-mutating path.""" + + self._skipped = True + self._skip_reason = reason + logger.info( + "FingerprintTransaction.skip for %s/%s (%s): %s", + self._basename, + self._language, + self._operation, + reason, + ) + + def set_include_deps_override(self, deps: Dict[str, str]) -> None: + """Preserve a dependency graph captured before a prompt rewrite.""" + + self._include_deps_override = dict(deps) + + def _error(self, cause: Any) -> FingerprintFinalizeError: + return FingerprintFinalizeError( + f"[{self._operation}] fingerprint finalization failed for " + f"{self._fingerprint_path}: {cause}" + ) + + def _build_payload(self) -> Dict[str, Any]: + """Build and validate the canonical serialized fingerprint payload.""" + + previous = read_fingerprint( + self._basename, + self._language, + paths=self._paths, + ) + stored_deps = previous.include_deps if previous else None + # Sync captures includes before auto-deps can strip them. Use that + # graph for hashing as well as persistence when it is supplied. + hash_deps = ( + self._include_deps_override + if self._include_deps_override is not None + else stored_deps + ) + current_hashes = calculate_current_hashes( + self._paths, + stored_include_deps=hash_deps, + ) + if current_hashes.get("prompt_hash") is None: + raise self._error("prompt_hash is null") + + include_deps = ( + self._include_deps_override + if self._include_deps_override is not None + else current_hashes.get("include_deps") + ) + fingerprint = Fingerprint( + pdd_version=__version__, + timestamp=datetime.now(timezone.utc).isoformat(), + command=self._operation, + prompt_hash=current_hashes.get("prompt_hash"), + code_hash=current_hashes.get("code_hash"), + example_hash=current_hashes.get("example_hash"), + test_hash=current_hashes.get("test_hash"), + test_files=current_hashes.get("test_files"), + include_deps=include_deps, + ) + return asdict(fingerprint) + + def prepare(self) -> Dict[str, Any]: + """Return a validated payload for a surrounding atomic state commit.""" + + try: + return self._build_payload() + except FingerprintFinalizeError: + raise + except Exception as exc: + raise self._error(exc) from exc + + def _commit(self) -> None: + tmp_path: Optional[str] = None + try: + payload = self._build_payload() + self._fingerprint_path.parent.mkdir(parents=True, exist_ok=True) + fd, tmp_path = tempfile.mkstemp( + dir=self._fingerprint_path.parent, + prefix=f".{self._fingerprint_path.stem}_", + suffix=".tmp", + ) + with os.fdopen(fd, "w", encoding="utf-8") as handle: + json.dump(payload, handle, indent=2) + handle.flush() + os.fsync(handle.fileno()) + os.replace(tmp_path, self._fingerprint_path) + tmp_path = None + except FingerprintFinalizeError: + raise + except Exception as exc: + raise self._error(exc) from exc + finally: + if tmp_path is not None: + try: + os.unlink(tmp_path) + except OSError: + pass + + def __exit__(self, exc_type, exc_val, exc_tb) -> bool: + if exc_type is not None: + return False + if self._skipped: + logger.debug( + "Fingerprint commit suppressed for %s/%s: %s", + self._basename, + self._language, + self._skip_reason, + ) + return False + self._commit() + return False + + +__all__ = ["FingerprintFinalizeError", "FingerprintTransaction"] diff --git a/pdd/metadata_sync.py b/pdd/metadata_sync.py index 00c92e99d4..d5f8df7396 100644 --- a/pdd/metadata_sync.py +++ b/pdd/metadata_sync.py @@ -17,7 +17,8 @@ update_architecture_from_prompt, ) from .architecture_registry import find_architecture_for_project -from .operation_log import clear_run_report, infer_module_identity, save_fingerprint +from .fingerprint_transaction import FingerprintTransaction +from .operation_log import clear_run_report, get_run_report_path, infer_module_identity _THEME = Theme( @@ -454,6 +455,12 @@ def run_metadata_sync( if code_path is not None: _rr_paths["code"] = code_path clear_run_report(basename, language, paths=_rr_paths) + if get_run_report_path( + basename, + language, + paths=_rr_paths, + ).exists(): + raise RuntimeError("run report remains after clear") result.stages["run_report"] = StageStatus( status="ok", detail=f"cleared run report for {detail}" ) @@ -515,14 +522,15 @@ def run_metadata_sync( if _prev_cmd in ("verify", "test", "fix", "update") else "fix" ) - save_fingerprint( + with FingerprintTransaction( basename=basename, language=language, operation=_preserved_command, paths=paths, cost=0.0, model="metadata_sync", - ) + ): + pass result.stages["fingerprint"] = StageStatus( status="ok", detail=f"saved fingerprint for {detail}" ) @@ -534,4 +542,4 @@ def run_metadata_sync( result.stages["fingerprint"] = StageStatus(status="failed", reason=reason) _stage_log_exit("fingerprint", "failed", reason) - return result \ No newline at end of file + return result diff --git a/pdd/operation_log.py b/pdd/operation_log.py index 2bcf9f8415..239bd5834d 100644 --- a/pdd/operation_log.py +++ b/pdd/operation_log.py @@ -209,7 +209,7 @@ def get_log_path( upward .pddrc detection seeded by `paths` (Issue #1211). """ ensure_meta_dir(project_root=project_root, paths=paths) - return _resolve_meta_dir(project_root, paths=paths) / f"{_safe_basename(basename)}_{language}_sync.log" + return _resolve_meta_dir(project_root, paths=paths) / f"{_safe_basename(basename)}_{language.lower()}_sync.log" def get_fingerprint_path( @@ -226,7 +226,7 @@ def get_fingerprint_path( run CWD is found), then up from CWD, then falls back to CWD. """ ensure_meta_dir(project_root=project_root, paths=paths) - return _resolve_meta_dir(project_root, paths=paths) / f"{_safe_basename(basename)}_{language}.json" + return _resolve_meta_dir(project_root, paths=paths) / f"{_safe_basename(basename)}_{language.lower()}.json" def get_run_report_path( @@ -240,7 +240,7 @@ def get_run_report_path( Same project-root resolution as get_fingerprint_path (Issue #1211). """ ensure_meta_dir(project_root=project_root, paths=paths) - return _resolve_meta_dir(project_root, paths=paths) / f"{_safe_basename(basename)}_{language}_run.json" + return _resolve_meta_dir(project_root, paths=paths) / f"{_safe_basename(basename)}_{language.lower()}_run.json" def infer_module_identity(prompt_file_path: Union[str, Path]) -> Tuple[Optional[str], Optional[str]]: @@ -578,17 +578,11 @@ def save_fingerprint( cost: float = 0.0, model: str = "unknown" ) -> None: - """ - Save the current fingerprint/state to the state file. + """Save current state through the authoritative fingerprint transaction. - Writes the full Fingerprint dataclass format compatible with read_fingerprint() - in sync_determine_operation.py. This ensures manual commands (generate, example) - don't break sync's fingerprint tracking. + ``FingerprintFinalizeError`` intentionally propagates. A mutating command + cannot report success when its fingerprint was not persisted (#1926). """ - from dataclasses import asdict - from datetime import timezone - from .sync_determine_operation import calculate_current_hashes, Fingerprint, read_fingerprint - from . import __version__ # Issue #983: when the caller provides `paths`, use them directly — do # NOT call get_pdd_file_paths. Issue #1211: at the same time, use those @@ -602,32 +596,17 @@ def save_fingerprint( logger.warning("Could not resolve paths for %s/%s: %s", basename, language, e) paths = {} - path = get_fingerprint_path(basename, language, paths=paths) - - # Issue #522: Pass stored include deps for prompt hash calculation - prev_fp = read_fingerprint(basename, language, paths=paths) - stored_deps = prev_fp.include_deps if prev_fp else None - current_hashes = calculate_current_hashes(paths, stored_include_deps=stored_deps) if paths else {} - - # Create Fingerprint with same format as _save_fingerprint_atomic - fingerprint = Fingerprint( - pdd_version=__version__, - timestamp=datetime.now(timezone.utc).isoformat(), - command=operation, - prompt_hash=current_hashes.get('prompt_hash'), - code_hash=current_hashes.get('code_hash'), - example_hash=current_hashes.get('example_hash'), - test_hash=current_hashes.get('test_hash'), - test_files=current_hashes.get('test_files'), - include_deps=current_hashes.get('include_deps'), # Issue #522 - ) + from .fingerprint_transaction import FingerprintTransaction - try: - with open(path, 'w', encoding='utf-8') as f: - json.dump(asdict(fingerprint), f, indent=2) - except Exception as e: - console = Console() - console.print(f"[yellow]Warning: Failed to save fingerprint to {path}: {e}[/yellow]") + with FingerprintTransaction( + basename=basename, + language=language, + operation=operation, + paths=paths, + cost=cost, + model=model, + ): + pass def save_run_report( @@ -795,7 +774,7 @@ def wrapper(*args: Any, **kwargs: Any) -> Any: fingerprint_allowed = _clear_run_report_before_fingerprint( basename, language, paths=log_paths ) - if updates_fingerprint and fingerprint_allowed: + if updates_fingerprint: # Issue #1305 + #1211: save_fingerprint hashes only # Path values, so a bare {"prompt": } hint # yields all-null hashes (the prompt string is @@ -843,14 +822,30 @@ def wrapper(*args: Any, **kwargs: Any) -> Any: e, ) fingerprint_paths = {"prompt": Path(prompt_file)} - save_fingerprint( - basename, - language, - operation=operation, - paths=fingerprint_paths, - cost=cost, - model=model, - ) + if fingerprint_allowed: + save_fingerprint( + basename, + language, + operation=operation, + paths=fingerprint_paths, + cost=cost, + model=model, + ) + else: + # Even intentional suppression routes through + # the transaction so future mutating paths + # cannot grow a second finalization branch. + from .fingerprint_transaction import FingerprintTransaction + + with FingerprintTransaction( + basename, + language, + operation, + paths=fingerprint_paths, + cost=cost, + model=model, + ) as transaction: + transaction.skip("run report not cleared") if updates_run_report and isinstance(result, dict): save_run_report(basename, language, result, paths=log_paths) return wrapper diff --git a/pdd/pin_example_hack.py b/pdd/pin_example_hack.py index 15b4889cfd..1d5b205535 100644 --- a/pdd/pin_example_hack.py +++ b/pdd/pin_example_hack.py @@ -19,6 +19,7 @@ import click import logging +from .fingerprint_transaction import FingerprintTransaction # --- Constants --- MAX_CONSECUTIVE_TESTS = 3 # Allow up to 3 consecutive test attempts @@ -246,31 +247,20 @@ def _save_operation_fingerprint(basename: str, language: str, operation: str, model: The model used. atomic_state: Optional AtomicStateUpdate for atomic writes (Issue #159 fix). """ - from datetime import datetime, timezone - from .sync_determine_operation import calculate_current_hashes, Fingerprint - from . import __version__ - - current_hashes = calculate_current_hashes(paths) - fingerprint = Fingerprint( - pdd_version=__version__, - timestamp=datetime.now(timezone.utc).isoformat(), - command=operation, - prompt_hash=current_hashes.get('prompt_hash'), - code_hash=current_hashes.get('code_hash'), - example_hash=current_hashes.get('example_hash'), - test_hash=current_hashes.get('test_hash'), - test_files=current_hashes.get('test_files'), # Bug #156 - ) - - fingerprint_file = META_DIR / f"{_safe_basename(basename)}_{language.lower()}.json" - if atomic_state: - # Buffer for atomic write - atomic_state.set_fingerprint(asdict(fingerprint), fingerprint_file) - else: - # Legacy direct write - META_DIR.mkdir(parents=True, exist_ok=True) - with open(fingerprint_file, 'w') as f: - json.dump(asdict(fingerprint), f, indent=2, default=str) + with FingerprintTransaction( + basename, + language, + operation, + paths=paths, + cost=cost, + model=model, + ) as transaction: + if atomic_state is not None: + atomic_state.set_fingerprint( + transaction.prepare(), + transaction.fingerprint_path, + ) + transaction.skip("deferred to AtomicStateUpdate") def _python_cov_target_for_code_file(code_file: Path) -> str: """Return a `pytest-cov` `--cov` target for a Python code file. @@ -1761,4 +1751,4 @@ def __init__(self, rc, out, err): PDD_DIR.mkdir(exist_ok=True) META_DIR.mkdir(exist_ok=True) result = sync_orchestration(basename="my_calculator", language="python", quiet=True) - print(json.dumps(result, indent=2)) \ No newline at end of file + print(json.dumps(result, indent=2)) diff --git a/pdd/prompts/auto_deps_main_python.prompt b/pdd/prompts/auto_deps_main_python.prompt index 9621b2468b..fe9c5f2dee 100644 --- a/pdd/prompts/auto_deps_main_python.prompt +++ b/pdd/prompts/auto_deps_main_python.prompt @@ -36,7 +36,7 @@ - `no_dedup` (`bool`): Flag to disable redundant inline content removal. Default False. - `concurrency` (`int`): Number of parallel workers for file summarization. Default 1. - `compress` (`bool`): Flag to use AST-based compression for Python includes. Default False. - - `_skip_finalization` (`bool`): Internal flag — when True, return immediately after the success console output without writing a fingerprint or clearing any run report. Default False. `pdd sync` sets this to True because sync owns canonical metadata bookkeeping itself (see the sync NOTE in the Behavior section). Leading underscore signals this is not part of the public CLI surface. + - `_skip_finalization` (`bool`): Internal flag — when True, still enter `FingerprintTransaction` but call `transaction.skip(reason="_skip_finalization set by sync")` without writing a fingerprint or clearing any run report. Default False. `pdd sync` sets this to True because sync owns canonical metadata bookkeeping itself (see the sync NOTE in the Behavior section). Leading underscore signals this is not part of the public CLI surface. Outputs: - Returns a tuple containing (`str`, `float`, `str`): @@ -90,8 +90,8 @@ • Use the existing helpers from `pdd.operation_log` (`infer_module_identity`, `clear_run_report`, `get_run_report_path`) and `FingerprintTransaction` from `pdd.fingerprint_transaction`. Behavior must be language-agnostic (hash-based). • Derive identity from the *output* path, not the input prompt: - Resolve `(basename, language)` via `infer_module_identity(Path(output_path))`. The output path is the file whose hash and dependency state the fingerprint records; tying the identity to that file keeps `.pdd/meta` self-consistent even when the canonical prompt is untouched (default mode) or when the file is the canonical prompt itself (in-place mode). `infer_module_identity` returns `(None, None)` for unrecognized prompt names — check `basename is None or language is None` explicitly and skip finalization (logged as a yellow warning) rather than guessing. - - Clear any stale per-module run report via `clear_run_report(basename, language, paths={"prompt": Path(output_path)})` because the prompt content changed. If `clear_run_report` raises, log a warning and continue — it must not abort the subsequent fingerprint transaction. - - After invoking `clear_run_report`, verify the report path is actually gone via `get_run_report_path(basename, language, paths={"prompt": Path(output_path)})`; if it still exists, log a yellow warning (unless `quiet`). This is defensive because the helper silently swallows unlink failures. + - Clear any stale per-module run report via `clear_run_report(basename, language, paths={"prompt": Path(output_path)})` because the prompt content changed. If cleanup raises, raise `FingerprintFinalizeError` with the fingerprint path and underlying cause; metadata cleanup is part of required finalization. + - After invoking `clear_run_report`, verify the report path is actually gone via `get_run_report_path(basename, language, paths={"prompt": Path(output_path)})`; if it still exists, raise `FingerprintFinalizeError`. A successful artifact mutation may not coexist with stale runtime state or report success without its fingerprint (#1926). - Write/update the fingerprint via `FingerprintTransaction(basename, language, operation="auto-deps", paths={"prompt": Path(output_path)}, cost=total_cost, model=model_name)` used as a context manager. When a pre-captured include graph is available, call `transaction.set_include_deps_override(include_deps)` before exiting the context. `FingerprintFinalizeError` propagates — a finalization failure exits non-zero. When `_skip_finalization=True`, call `transaction.skip(reason="_skip_finalization set by sync")` instead of omitting the transaction entirely. • Architecture file updates (architecture.json) are merged by `merge_auto_deps_includes_from_cwd` and are not finalized here — that file is not a `(basename, language)`-keyed module and has its own atomicity guarantees inside the merge helper. • Surface non-exception failures from the architecture merge (return shape `{"updated": bool, "messages": list[str], …}`): if the helper returned `updated=False`, emit each `messages` entry as a yellow `Warning: architecture.json not updated: …` line so the success message does not mask a silent no-op. @@ -99,7 +99,7 @@ • NOTE: `pdd sync` invokes `auto_deps_main` with a temp `__with_deps.prompt` output and then `shutil.move`s it onto the canonical prompt. To avoid leaving an orphan `.pdd/meta/*_with_deps.json` for that temp identity after the move (and to avoid clearing the wrong run report), sync passes the internal flag `_skip_finalization=True`; the `FingerprintTransaction` for that identity calls `transaction.skip(reason="_skip_finalization set by sync")` and sync owns canonical metadata itself via `FingerprintTransaction` (routed through `_save_fingerprint_atomic`) and clears the canonical `_run.json` via `clear_run_report` immediately after the move. When `_skip_finalization=True`, this finalizer must enter the `FingerprintTransaction` context but call `transaction.skip(...)` before returning. -pdd/operation_log.py +pdd/operation_log.py diff --git a/pdd/prompts/commands/fix_python.prompt b/pdd/prompts/commands/fix_python.prompt index e33b7d755e..7008f31427 100644 --- a/pdd/prompts/commands/fix_python.prompt +++ b/pdd/prompts/commands/fix_python.prompt @@ -44,7 +44,7 @@ - Manual: `--output-test`, `--output-code`, `--output-results`, `--loop`, `--verification-program`, `--max-attempts` (3), `--budget` (5.0), `--auto-submit`, `--agentic-fallback/--no-agentic-fallback` (T), `--compress-test-context`, `--context-compression` (choice), `--compression-fallback` (choice). - Both: `--protect-tests/--no-protect-tests` (F). - **Technical Details**: - - Decorate with `@log_operation(operation="fix", clears_run_report=True)` and `@track_cost`. + - Decorate with `@log_operation(operation="fix", clears_run_report=True, updates_fingerprint=True)` and `@track_cost`, so successful direct fixes commit through `FingerprintTransaction` and finalization failures propagate to Click. - Use deferred imports inside the command for mode-specific functions. - Return `Optional[Tuple[result_dict, cost, model]]`. - Re-raise Click exceptions; use `handle_error(e, "fix", quiet)` for others. diff --git a/pdd/prompts/commands/maintenance_python.prompt b/pdd/prompts/commands/maintenance_python.prompt index 6326e5a8b0..f1e4b08b93 100644 --- a/pdd/prompts/commands/maintenance_python.prompt +++ b/pdd/prompts/commands/maintenance_python.prompt @@ -16,6 +16,7 @@ auto_deps_main_python.prompt architecture_sync_python.prompt +fingerprint_transaction_python.prompt % You are an expert Python engineer. Your goal is to write `pdd/commands/maintenance.py`. @@ -99,6 +100,7 @@ - `--include-docs` (bool flag, default False): Include documentation files (`.md`, `.txt`, `.rst`) in dependency discovery. Pass as `include_docs` to `auto_deps_main`. - `--no-dedup` (bool flag, default False): Disable redundant inline content removal after inserting includes. Pass as `no_dedup` to `auto_deps_main`. - `--concurrency` (int, default 1): Number of parallel workers for file summarization LLM calls. Pass as `concurrency` to `auto_deps_main`. + - Import `FingerprintFinalizeError`. Catch it before the broad exception handler, report it with `handle_error`, and raise `click.exceptions.Exit(1)` so required fingerprint failure cannot become exit code 0. 5. **setup**: - Interface: Match the `setup` command specification in `README.md`. - Logic: Install shell completion, then run setup utility. diff --git a/pdd/prompts/commands/modify_python.prompt b/pdd/prompts/commands/modify_python.prompt index 8a5855df99..5e07b283ab 100644 --- a/pdd/prompts/commands/modify_python.prompt +++ b/pdd/prompts/commands/modify_python.prompt @@ -20,6 +20,7 @@ change_main_python.prompt agentic_change_python.prompt update_main_python.prompt +fingerprint_transaction_python.prompt % Goal Write `pdd/commands/modify.py`. @@ -75,7 +76,7 @@ This module provides Click commands (`split`, `change`, `update`) for prompt man % Error Handling - `split` and `change`: re-raise `click.Abort`, `click.UsageError`, `click.exceptions.Exit`. -- `update`: re-raise `click.Abort`, `click.UsageError`, and `click.exceptions.Exit`. The `Exit` re-raise is required because `update_main` raises `click.exceptions.Exit(1)` when `sync_metadata=True` finalization fails (the preflight auto-heal subprocess keys off `returncode != 0` — without the re-raise, the bare `except Exception` would swallow it back to `exit_code 0`, reintroducing the #871 silent-success bug). +- `update`: re-raise `click.Abort`, `click.UsageError`, and `click.exceptions.Exit`. Catch `FingerprintFinalizeError` before the broad handler, report it, and convert it to `click.exceptions.Exit(1)` so required finalization cannot become exit code 0 (#1926/#871). - Otherwise call `handle_error(e, command_name, quiet)` and return `None`. % Dependencies diff --git a/pdd/prompts/metadata_sync_python.prompt b/pdd/prompts/metadata_sync_python.prompt index 5ead5c943e..10740ab65d 100644 --- a/pdd/prompts/metadata_sync_python.prompt +++ b/pdd/prompts/metadata_sync_python.prompt @@ -34,7 +34,7 @@ This module provides the shared `run_metadata_sync` entry point used by `pdd upd 1. `prompt`: confirm the prompt file exists and is non-empty; record current on-disk content for the next stage. 2. `tags`: **preserve existing PDD metadata tags, or seed tags from the architecture entry when the prompt has none.** Uses `pdd.architecture_sync.generate_tags_from_architecture` (multi-language compatible, no language-specific AST dependency) to render the seed block. Writes the seeded prompt back to disk unless `dry_run=True`. When the prompt has zero PDD tags AND no architecture entry exists to seed from, report `skipped` with a reason naming #870 — never `ok` (claiming `ok with 0 tag(s)` would mislead operators). **LLM-first tag refresh (rewriting existing-but-stale tags) is out of scope for this orchestrator and tracked at #870**; downstream callers that need refresh should invoke that primitive directly. 3. `architecture`: call `update_architecture_from_prompt(prompt_filename, prompts_dir, architecture_path, dry_run, prompt_content_override=)` so architecture.json picks up the freshly-generated tags without a disk race. `prompt_filename` MUST be the prompts-dir-relative path (e.g. `commands/foo_python.prompt`), not just the basename — `update_architecture_from_prompt` matches architecture entries by their `filename` field which is path-aware per Issue #617. Fall back to the basename only when the prompt sits outside any `prompts/` ancestor. When the architecture lookup returns "No architecture entry found for: …", treat that as `skipped` (unregistered module is a normal state for tools/scripts), NOT as `failed`, so the fingerprint stage is not gated off. **Gated symmetrically with `run_report` and `fingerprint`**: if `prompt` or `tags` reported `failed`, mark `architecture` as `skipped` with the failing stage as the reason and do NOT invoke `update_architecture_from_prompt` — that call writes architecture.json on success, and running it after an earlier hard failure means the architecture entry can be modified on disk before the overall result reports `failed`, which on the push-to-main auto-heal path can land via the heal-commit's scoped `git add -u` even though downstream stages correctly stop. - 4. `run_report`: clear or refresh stale run reports tied to this prompt/code pair (only the entries this sync invalidates — do not wipe unrelated entries). Build a `paths` hint from the known file paths — `{"prompt": prompt_path}` plus `"code": code_path` when `code_path` is set — and pass it as `paths=` to `clear_run_report` so that `.pdd/meta` is anchored at the subproject root rather than the run CWD (supports parent-directory workflows, issue #1211). **Gated symmetrically with `fingerprint`**: if `prompt`, `tags`, or `architecture` reported `failed`, mark `run_report` as `skipped` with the failing stage as the reason and do NOT invoke `clear_run_report` — the deletion is an on-disk side effect that, left to run after an earlier hard failure, would create partial `.pdd/meta` state that auto-heal's scoped staging (`git add -u` plus per-module fingerprint pathspecs) could still pick up on the push-to-main path. + 4. `run_report`: clear stale run reports tied to this prompt/code pair, then verify `get_run_report_path(...).exists()` is false. A surviving report or inspection error makes this stage `failed`, which gates fingerprint persistence. Build and pass the same prompt/code `paths` hint to both helpers for subproject anchoring (#1211). If an earlier prompt/tags/architecture stage failed, mark this stage `skipped` without deleting anything. 5. `fingerprint`: finalize fingerprint state LAST via `FingerprintTransaction` directly (not the `save_fingerprint` wrapper). Pass the same `paths` hint (built from `prompt_path`/`code_path`) so fingerprint reads and writes anchor at the subproject `.pdd/meta` directory (issue #1211). Gated on no prior stage reporting `failed` — `skipped` and `dry_run` are acceptable upstream statuses (matches `MetadataSyncResult.ok`). If any earlier stage failed, mark fingerprint as `skipped` with the failing stage as the reason. This prevents storing a fingerprint that records "in sync" when an earlier metadata layer is broken. `FingerprintFinalizeError` is caught by the per-stage handler (Req 4) and reported as `{"status": "failed", "reason": str(exc)}` — this is the correct behavior for `metadata_sync` as an orchestrator that surfaces failures via `MetadataSyncResult.ok` rather than propagating them. 3. **Result type** `MetadataSyncResult` — a Pydantic v2 model with: - `prompt_path: Path` @@ -70,7 +70,7 @@ This module provides the shared `run_metadata_sync` entry point used by `pdd upd pdd/sync_determine_operation.py - pdd/operation_log.py + pdd/operation_log.py pdd/fingerprint_transaction.py diff --git a/pdd/prompts/update_main_python.prompt b/pdd/prompts/update_main_python.prompt index 0c2ba06d4e..e08ed61aa0 100644 --- a/pdd/prompts/update_main_python.prompt +++ b/pdd/prompts/update_main_python.prompt @@ -76,7 +76,7 @@ Supports three modes: true update, regeneration, and repository-wide updates. Ro - If the written prompt path differs from the canonical source prompt (output redirected): `transaction.skip(reason="output redirected")`; print `[info][metadata] Skipping fingerprint finalization: output redirected[/info]` unless `quiet`. This guard applies in addition to (and independently of) the `sync_metadata=True` skip. - If `infer_module_identity(prompt_path)` returns `(None, None)`: `transaction.skip(reason="unable to infer module identity")`; print `[info][metadata] Skipping fingerprint finalization: unable to infer module identity for [/info]` unless `quiet`. - `FingerprintFinalizeError` propagates — the successful return tuple is NOT preserved on fingerprint write failure; Click exits non-zero. This is the enforced commit-or-fail contract (issue #1926). - - Stale `_run.json` cleanup MUST reuse `pdd.operation_log._clear_run_report_before_fingerprint(basename, language, paths={"prompt": Path(prompt_path), "code": Path(code_path)})` (omitting `"code"` when unavailable), which clears the run report then re-checks existence and returns `False` (with a yellow console warning) when a silent `os.remove` left it on disk. When the helper returns `False`, call `transaction.skip(reason="run report not cleared")` so a fresh fingerprint never coexists with stale runtime state (issue #1106). The helper's warning surfaces unconditionally — it does NOT honour the caller's `quiet` flag, intentionally, because it describes a real metadata problem. Wrap the helper call in `try/except`: on an unexpected exception, emit `[warning][metadata] Run report clear failed: [/warning]` unless `quiet` and call `transaction.skip(reason="run report clear failed")` (best-effort cleanup must never block the transaction context). The function-local `from .operation_log import (_clear_run_report_before_fingerprint, infer_module_identity)` and `from .fingerprint_transaction import FingerprintTransaction, FingerprintFinalizeError` MUST be wrapped in `try/except ImportError`: on ImportError, emit `[warning][metadata] Could not import finalization helpers: [/warning]` unless `quiet` and return without entering the transaction. + - Stale `_run.json` cleanup MUST reuse `pdd.operation_log._clear_run_report_before_fingerprint(basename, language, paths={"prompt": Path(prompt_path), "code": Path(code_path)})` (omitting `"code"` when unavailable), which clears the run report then re-checks existence and returns `False` (with a yellow console warning) when a silent `os.remove` left it on disk. When the helper returns `False`, call `transaction.skip(reason="run report not cleared")` and raise `FingerprintFinalizeError` so a fresh fingerprint never coexists with stale runtime state and the mutation cannot report success (issues #1106 and #1926). On an unexpected cleanup exception, call `transaction.skip(reason="run report clear failed")` and raise `FingerprintFinalizeError` with the destination and underlying cause. The helper's warning surfaces unconditionally — it does NOT honour the caller's `quiet` flag. The function-local `from .operation_log import (_clear_run_report_before_fingerprint, infer_module_identity)` MUST remain import-guarded for compatibility. % Modes 1. **True update**: `input_prompt_file` + `modified_code_file` + (use_git OR `input_code_file`). @@ -115,7 +115,7 @@ Supports three modes: true update, regeneration, and repository-wide updates. Ro - After resolving pairs, call `ensure_pddrc_for_scan` from `pddrc_initializer` to auto-create `.pddrc` if needed. - **Change detection**: Use `get_git_changed_files` + `is_code_changed` to filter pairs. Also include pairs with empty (0-byte) prompt files regardless of code changes. Include dependency hashes from fingerprints are checked so that changes to shared include files (preambles, examples) trigger updates. - Progress bar: Rich Progress with spinner, bar, percentage, time remaining, and running total cost. -- **Post-update fingerprinting** (legacy, `sync_metadata=False` only): After each successful pair update, infer module identity via `infer_module_identity`, then call `clear_run_report(basename, language, paths={"prompt": Path(prompt_path), "code": Path(code_path)})` BEFORE `save_fingerprint` so stale `.pdd/meta/__run.json` runtime-verification state from before the mutation is invalidated rather than left to coexist with the freshly written fingerprint. Pass the same `paths` hint to `get_run_report_path` and `save_fingerprint` so all metadata writes anchor at the subproject root (issue #1211). Wrap `clear_run_report` in its own try/except and surface failures as a non-fatal warning (mirroring the single-file finalize path in `_finalize_target_fingerprint`); after the clear attempt, re-check `get_run_report_path(basename, language, paths=...).exists()` and **skip `save_fingerprint` for that pair (with a yellow warning) when the stale `_run.json` still exists** — this is the safer behavior required by issue #1057 so a fresh fingerprint can never coexist with stale runtime-verification state. When `sync_metadata=True`, the per-pair `run_metadata_sync` call replaces this — do NOT also call `clear_run_report` or `save_fingerprint` here, or the fingerprint will be written before the orchestrator's gates run and a stale fingerprint can record "in sync" even when an earlier metadata stage failed. +- **Post-update fingerprinting** (legacy, `sync_metadata=False` only): After each successful pair update, infer module identity, call `_clear_run_report_before_fingerprint(...)`, and then call the transaction-backed `save_fingerprint` with the same prompt/code `paths` hint. Any cleanup exception, surviving stale `_run.json`, null prompt hash, or persist failure raises `FingerprintFinalizeError`; repo update stops and exits non-zero instead of reporting a partially finalized success (#1926). When `sync_metadata=True`, the per-pair `run_metadata_sync` call replaces this — do NOT double-write metadata. - **Post-update architecture sync** (legacy, `sync_metadata=False` only): Use `find_architecture_for_project` to locate architecture files; for each successfully updated prompt, call `update_architecture_from_prompt` if architecture file exists. When `sync_metadata=True`, this is handled inside `run_metadata_sync` (which also gates the write on prior-stage success), so do NOT call it again from `update_main`. - **Post-update PRD sync** (both branches): If architecture entries were updated, find PRD file and use `run_agentic_task` to review/update PRD content. Unpack the agent result as `(llm_success, llm_output, llm_cost, _llm_model)`, add `llm_cost` to the repository total when present, and treat `llm_success=False` as an error status without writing the PRD. Parse `...` tags from `llm_output` only when the call succeeded; otherwise leave the PRD unchanged. PRD sync stays in `update_main` even under `sync_metadata=True` — the orchestrator deliberately does NOT call it (see Requirement 11). Key off the count of pairs whose `MetadataSyncResult.stages['architecture']` reported `updated`/`ok` (or, in the legacy branch, off the count returned by the inline arch loop). - Summary table: `prompt file`, `status`, `cost`, `model`, `error` columns. When `sync_metadata=True`, append a `metadata` column (one of `synced`, `partial:`, `failed:`, `skipped`, `dry-run`) so partial heals surface visibly. Show architecture/PRD status if relevant. diff --git a/pdd/sync_main.py b/pdd/sync_main.py index 8ca29b5a12..0d4e25ea30 100644 --- a/pdd/sync_main.py +++ b/pdd/sync_main.py @@ -29,6 +29,7 @@ ) from .sync_determine_operation import get_pdd_file_paths, AmbiguousModuleError from .operation_log import get_run_report_path +from .fingerprint_transaction import FingerprintTransaction from .architecture_include_validation import print_architecture_include_validation_warnings from .compressed_sync_context import build_compressed_sync_context, metadata as compressed_context_metadata from .sync_orchestration import sync_orchestration @@ -1227,12 +1228,19 @@ def _one_session_compressed_context( # Post-sync: save fingerprint so next sync sees files as up-to-date if one_session_result.get("success"): - from .operation_log import save_fingerprint - save_fingerprint( - basename, resolved_language, "fix", - pdd_files, one_session_result.get("total_cost", 0.0), - one_session_result.get("model_name", "unknown") or pre_model or "unknown", - ) + with FingerprintTransaction( + basename, + resolved_language, + "fix", + paths=pdd_files, + cost=one_session_result.get("total_cost", 0.0), + model=( + one_session_result.get("model_name", "unknown") + or pre_model + or "unknown" + ), + ): + pass # Post-sync: auto-submit example to cloud on success if one_session_result.get("success") and not local: diff --git a/pdd/sync_orchestration.py b/pdd/sync_orchestration.py index 4e7e70c145..5317295680 100644 --- a/pdd/sync_orchestration.py +++ b/pdd/sync_orchestration.py @@ -38,13 +38,16 @@ update_log_entry, append_log_entry, log_event, - save_fingerprint, save_run_report, clear_run_report, get_log_path, get_run_report_path, get_fingerprint_path, ) +from .fingerprint_transaction import ( + FingerprintFinalizeError, + FingerprintTransaction, +) from .sync_determine_operation import ( sync_determine_operation, get_pdd_file_paths, @@ -520,6 +523,8 @@ def _atomic_write(self, data: Dict[str, Any], target_path: Path) -> None: try: with os.fdopen(fd, 'w') as f: json.dump(data, f, indent=2, default=str) + f.flush() + os.fsync(f.fileno()) # Atomic rename - guaranteed atomic on POSIX systems os.replace(temp_path, target_path) @@ -532,7 +537,14 @@ def _commit(self): """Commit all pending state updates atomically.""" # Write fingerprint first (checkpoint), then run_report if self.pending.fingerprint and self.pending.fingerprint_path: - self._atomic_write(self.pending.fingerprint, self.pending.fingerprint_path) + try: + self._atomic_write(self.pending.fingerprint, self.pending.fingerprint_path) + except Exception as exc: + operation = self.pending.fingerprint.get("command", "sync") + raise FingerprintFinalizeError( + f"[{operation}] fingerprint finalization failed for " + f"{self.pending.fingerprint_path}: {exc}" + ) from exc if self.pending.run_report and self.pending.run_report_path: self._atomic_write(self.pending.run_report, self.pending.run_report_path) @@ -682,43 +694,24 @@ def _save_fingerprint_atomic(basename: str, language: str, operation: str, include_deps_override: Pre-captured include deps (Issue #522). Used when auto-deps may have stripped tags before fingerprint save. """ - if atomic_state: - # Buffer for atomic write - from datetime import datetime, timezone - from .sync_determine_operation import calculate_current_hashes, Fingerprint, read_fingerprint - from . import __version__ - - # Issue #522: Use override deps if provided (captured before auto-deps), - # otherwise fall back to stored deps from previous fingerprint + with FingerprintTransaction( + basename, + language, + operation, + paths=paths, + cost=cost, + model=model, + ) as transaction: if include_deps_override is not None: - stored_deps = include_deps_override - else: - prev_fp = read_fingerprint(basename, language, paths=paths) - stored_deps = prev_fp.include_deps if prev_fp else None - current_hashes = calculate_current_hashes(paths, stored_include_deps=stored_deps) - # If override provided and current extraction found nothing, use the override - if include_deps_override and not current_hashes.get('include_deps'): - current_hashes['include_deps'] = include_deps_override - fingerprint = Fingerprint( - pdd_version=__version__, - timestamp=datetime.now(timezone.utc).isoformat(), - command=operation, - prompt_hash=current_hashes.get('prompt_hash'), - code_hash=current_hashes.get('code_hash'), - example_hash=current_hashes.get('example_hash'), - test_hash=current_hashes.get('test_hash'), - test_files=current_hashes.get('test_files'), # Bug #156 - include_deps=current_hashes.get('include_deps'), # Issue #522 - ) - - # Issue #1211: route the atomic fingerprint file through the - # paths-aware helper so subprojects whose .pddrc is below run CWD - # get the file under /.pdd/meta, not parent CWD. - fingerprint_file = get_fingerprint_path(basename, language, paths=paths) - atomic_state.set_fingerprint(asdict(fingerprint), fingerprint_file) - else: - # Direct write using operation_log - save_fingerprint(basename, language, operation, paths, cost, model) + transaction.set_include_deps_override(include_deps_override) + if atomic_state is not None: + # The shared transaction owns path resolution, hash calculation, + # null validation and serialization. AtomicStateUpdate only + # delays the already-validated payload so its run report and + # fingerprint can commit together. + payload = transaction.prepare() + atomic_state.set_fingerprint(payload, transaction.fingerprint_path) + transaction.skip("deferred to AtomicStateUpdate") def _python_cov_target_for_code_file(code_file: Path) -> str: """Return a `pytest-cov` `--cov` target for a Python code file. @@ -2691,12 +2684,17 @@ def sync_worker_logic(): # ``_run.json`` because the prompt # content just changed. Mirrors the # clear after generate (line 2272). - try: - clear_run_report(basename, language, paths=pdd_files) - except Exception: - # Never mask a successful auto-deps - # result on metadata cleanup errors. - pass + from .operation_log import _clear_run_report_before_fingerprint + + if not _clear_run_report_before_fingerprint( + basename, + language, + paths=pdd_files, + ): + raise FingerprintFinalizeError( + "[auto-deps] fingerprint finalization failed: " + "stale run report could not be cleared" + ) else: temp_output.unlink() result = (new_content, 0.0, 'no-changes') @@ -3530,7 +3528,17 @@ def __init__(self, rc, out, err): and actual_cost == 0.0 and _model_name_lower_check in ['', 'none', 'unknown', 'n/a'] ) - if not _is_noop_fix: + if _is_noop_fix: + with FingerprintTransaction( + basename, + language, + operation, + paths=pdd_files, + cost=actual_cost, + model=str(model_name), + ) as transaction: + transaction.skip("no-op fix — no LLM invocation") + else: _save_fingerprint_atomic(basename, language, operation, pdd_files, actual_cost, str(model_name), atomic_state=atomic_state, include_deps_override=include_deps_override) if operation_rollback is not None: operation_rollback.commit() diff --git a/pdd/update_main.py b/pdd/update_main.py index 0543cbb2f3..d549707786 100644 --- a/pdd/update_main.py +++ b/pdd/update_main.py @@ -32,6 +32,10 @@ from .git_update import git_update from .agentic_common import get_available_agents from .agentic_update import run_agentic_update +from .fingerprint_transaction import ( + FingerprintFinalizeError, + FingerprintTransaction, +) from .sync_determine_operation import calculate_sha256, extract_include_deps, read_fingerprint from .validate_prompt_includes import sanitize_prompt_output from . import DEFAULT_TIME @@ -1078,64 +1082,11 @@ def _finalize_single_file_fingerprint( model: str, source_prompt_path: Optional[Path] = None, ) -> None: - """Default fingerprint finalization for single-file/regeneration update modes. - - Writes a current `(prompt, code)` fingerprint via the existing - `pdd.operation_log` helpers so a successful `pdd update ` is not - re-detected as changed on the next run (issue #1007 / PR #1009 - Requirement 15). Skips with an `[info]` log line — unless `quiet` — for - the intentional skip cases (sync_metadata orchestrator owns the stage, - dry-run mode, `--output` redirected the write away from the canonical - source prompt, or `infer_module_identity` cannot derive - basename/language). All failures are best-effort: a `save_fingerprint` - exception surfaces as a `[warning]` line but never breaks the caller's - success tuple. - - ``source_prompt_path`` is the canonical input prompt for the module. When - callers pass a redirected output path via ``--output``, the written file - no longer represents the module's canonical prompt, so finalizing a - fingerprint against it would let later sync detection treat unrelated - canonical prompts as in sync (issue #1007 stale-state class). Skip in - that case. - """ - if sync_metadata: - if not quiet: - rprint( - "[info][metadata] Skipping fingerprint finalization: " - "orchestrator owns fingerprint stage[/info]" - ) - return - if dry_run: - if not quiet: - rprint( - "[info][metadata] Skipping fingerprint finalization: dry-run mode[/info]" - ) - return - if source_prompt_path is not None and _is_output_redirected( - Path(prompt_path), Path(source_prompt_path) - ): - if not quiet: - rprint( - "[info][metadata] Skipping fingerprint finalization: " - "output redirected[/info]" - ) - return - - # Wrap the import itself so the user's successful update tuple is never - # broken by an import-time failure (e.g. `_clear_run_report_before_fingerprint` - # gets renamed in a future operation_log refactor — it's a private - # underscore-prefixed name and therefore more fragile than the public - # `clear_run_report` / `infer_module_identity` / `save_fingerprint` - # alongside it). An ImportError raised here would propagate up to - # `update_main`'s outer `except Exception: return None`, converting a - # successful `(prompt, cost, model)` tuple to None — which violates the - # issue #1106 acceptance criterion: best-effort metadata cleanup must - # never fail the successful update tuple. + """Finalize one update through the shared commit-or-fail transaction.""" try: from .operation_log import ( _clear_run_report_before_fingerprint, infer_module_identity, - save_fingerprint, ) except ImportError as exc: if not quiet: @@ -1153,54 +1104,52 @@ def _finalize_single_file_fingerprint( ) return - # Reuse the shared helper so the single-file finalize path enforces the - # same invariant the `log_operation` decorator and repo-mode block already - # do: a fresh fingerprint must never coexist with a stale `_run.json` - # (issue #1106). The helper re-checks that the run report is actually - # gone after `clear_run_report()` and emits a console warning if a - # silent `os.remove` failure left it behind — see - # `pdd.operation_log._clear_run_report_before_fingerprint`. The warning - # surfaces unconditionally (the helper does not consult `quiet`): the - # contract the issue text quotes is "print a warning" without qualifying - # on quiet mode, and the user should learn that runtime verification - # state still describes the pre-mutation files even when --quiet - # suppresses other chatter (why: informational about a real metadata - # problem, not status fluff). - # Issue #1211: pass the same `paths` hint we use for save_fingerprint so - # the clear targets the subproject's .pdd/meta (matched on the prompt's - # nearest .pddrc), not a parent CWD orphan. Without this we cleared - # parent metadata while writing the fresh fingerprint to the subproject, - # leaving stale subproject _run.json beside it. update_paths = {"prompt": Path(prompt_path), "code": Path(code_path)} - try: - fingerprint_allowed = _clear_run_report_before_fingerprint( - basename, language, paths=update_paths - ) - except Exception as exc: - # Defensive: surrounding pattern in this function treats metadata - # cleanup as best-effort; an unexpected raise must not break the - # successful update tuple. Warn and skip the save, matching the - # `save_fingerprint` except-arm below. - if not quiet: - rprint( - f"[warning][metadata] Run report clear failed: {exc}[/warning]" - ) - return - if not fingerprint_allowed: - return + with FingerprintTransaction( + basename, + language, + operation="update", + paths=update_paths, + cost=cost, + model=model, + ) as transaction: + skip_reason: Optional[str] = None + if sync_metadata: + skip_reason = "orchestrator owns fingerprint stage" + elif dry_run: + skip_reason = "dry-run mode" + elif source_prompt_path is not None and _is_output_redirected( + Path(prompt_path), Path(source_prompt_path) + ): + skip_reason = "output redirected" + + if skip_reason is not None: + transaction.skip(skip_reason) + if not quiet: + rprint( + "[info][metadata] Skipping fingerprint finalization: " + f"{skip_reason}[/info]" + ) + return - try: - save_fingerprint( - basename, - language, - operation="update", - paths=update_paths, - cost=cost, - model=model, - ) - except Exception as exc: - if not quiet: - rprint(f"[warning][metadata] Fingerprint save failed: {exc}[/warning]") + try: + fingerprint_allowed = _clear_run_report_before_fingerprint( + basename, + language, + paths=update_paths, + ) + except Exception as exc: + transaction.skip("run report clear failed") + raise FingerprintFinalizeError( + f"[update] fingerprint finalization failed for " + f"{transaction.fingerprint_path}: run report clear failed: {exc}" + ) from exc + if not fingerprint_allowed: + transaction.skip("run report not cleared") + raise FingerprintFinalizeError( + f"[update] fingerprint finalization failed for " + f"{transaction.fingerprint_path}: run report not cleared" + ) def update_main( @@ -1386,8 +1335,7 @@ def update_main( # Save fingerprint so the file isn't detected as changed next run if "Success" in result.get("status", ""): from .operation_log import ( - clear_run_report, - get_run_report_path, + _clear_run_report_before_fingerprint, infer_module_identity, save_fingerprint, ) @@ -1403,67 +1351,30 @@ def update_main( "prompt": Path(prompt_path), "code": Path(code_path), } - # Clear stale run report first so it can't outlive - # the prompt/code pair it described. Best-effort: - # never fail the update because of metadata I/O, - # but surface failures as a non-fatal warning so - # the user knows runtime verification state may - # still describe the pre-mutation files. try: - _stale_report_path = get_run_report_path( - basename, language, paths=_update_paths + _fingerprint_allowed = _clear_run_report_before_fingerprint( + basename, + language, + paths=_update_paths, ) - except Exception: - _stale_report_path = None - _pre_existed = bool( - _stale_report_path is not None - and _stale_report_path.exists() - ) - try: - clear_run_report(basename, language, paths=_update_paths) except Exception as exc: - if not quiet: - rprint( - f"[warning][metadata] Run report clear failed for " - f"{basename} ({language}): {exc}[/warning]" - ) - # Defensive: clear_run_report() in pdd.operation_log - # silently swallows OSError on the actual unlink - # (see pdd/operation_log.py:317-320), so if the - # report file existed before the call but still - # exists afterwards, the deletion failed silently. - # Surface that as a non-fatal warning so the user - # knows runtime verification state may still - # describe the pre-mutation files. - _stale_remains = False - if _pre_existed and _stale_report_path is not None: - try: - _still_there = _stale_report_path.exists() - except Exception: - _still_there = False - if _still_there: - _stale_remains = True - if not quiet: - rprint( - f"[warning][metadata] Run report clear failed for " - f"{basename} ({language}): " - f"still exists after clear_run_report: " - f"{_stale_report_path}; skipping fingerprint update so a " - f"fresh fingerprint does not coexist with a stale " - f"run report (issue #1057)." - f"[/warning]" - ) - if not _stale_remains: - try: - save_fingerprint( - basename, language, - operation="update", - paths=_update_paths, - cost=result.get("cost", 0.0), - model=result.get("model", "unknown"), - ) - except Exception: - pass # Best-effort; don't fail the update + raise FingerprintFinalizeError( + f"[update] fingerprint finalization failed for " + f"{basename}/{language}: run report clear failed: {exc}" + ) from exc + if not _fingerprint_allowed: + raise FingerprintFinalizeError( + f"[update] fingerprint finalization failed for " + f"{basename}/{language}: run report not cleared" + ) + save_fingerprint( + basename, + language, + operation="update", + paths=_update_paths, + cost=result.get("cost", 0.0), + model=result.get("model", "unknown"), + ) else: if "Success" in result.get("status", ""): try: @@ -2023,6 +1934,10 @@ def _metadata_column_value(prompt_file_key: str) -> str: return modified_prompt, total_cost, model_name + except FingerprintFinalizeError as e: + if not quiet: + rprint(f"[bold red]Error:[/bold red] {e}") + raise click.exceptions.Exit(1) from e except (ValueError, git.InvalidGitRepositoryError) as e: if not quiet: rprint(f"[bold red]Input error:[/bold red] {str(e)}") diff --git a/tests/test_auto_deps_main.py b/tests/test_auto_deps_main.py index b6f7814229..c7e6100ef8 100644 --- a/tests/test_auto_deps_main.py +++ b/tests/test_auto_deps_main.py @@ -35,7 +35,7 @@ def tmp_dir(): @pytest.fixture(autouse=True) -def _isolate_metadata_finalization(request): +def _isolate_metadata_finalization(request, tmp_path): """ Prevent tests in this module from writing real fingerprint/run-report files into the repository's ``.pdd/meta`` directory. Tests that need to @@ -51,8 +51,12 @@ def _isolate_metadata_finalization(request): if "use_real_finalization" in request.fixturenames: yield return - with patch("pdd.auto_deps_main.save_fingerprint"), \ - patch("pdd.auto_deps_main.clear_run_report"): + with patch("pdd.auto_deps_main.FingerprintTransaction"), \ + patch("pdd.auto_deps_main.clear_run_report"), \ + patch( + "pdd.auto_deps_main.get_run_report_path", + return_value=tmp_path / "absent_run.json", + ): yield @@ -757,7 +761,7 @@ def test_auto_deps_main_updates_architecture_json_after_write( # prompt's. The fingerprint records the derivative under its own # ``(basename, language)`` so canonical metadata stays untouched. # --------------------------------------------------------------------------- -@patch("pdd.auto_deps_main.save_fingerprint") +@patch("pdd.auto_deps_main.FingerprintTransaction") @patch("pdd.auto_deps_main.clear_run_report") @patch("pdd.auto_deps_main.infer_module_identity") @patch("pdd.auto_deps_main.construct_paths") @@ -767,7 +771,7 @@ def test_auto_deps_metadata_finalizes_with_output_identity_in_default_mode( mock_construct_paths, mock_infer_identity, mock_clear_run_report, - mock_save_fingerprint, + mock_transaction, mock_ctx, tmp_path: Path, ): @@ -797,8 +801,8 @@ def test_auto_deps_metadata_finalizes_with_output_identity_in_default_mode( mock_infer_identity.assert_called_once_with(Path(output_path)) mock_clear_run_report.assert_called_once_with("child_python_with", "deps", paths=ANY) - mock_save_fingerprint.assert_called_once() - fp_kwargs = mock_save_fingerprint.call_args.kwargs + mock_transaction.assert_called_once() + fp_kwargs = mock_transaction.call_args.kwargs assert fp_kwargs["basename"] == "child_python_with" assert fp_kwargs["language"] == "deps" assert fp_kwargs["operation"] == "auto-deps" @@ -813,7 +817,7 @@ def test_auto_deps_metadata_finalizes_with_output_identity_in_default_mode( # ``clear_run_report`` and ``save_fingerprint`` run with that canonical # identity. # --------------------------------------------------------------------------- -@patch("pdd.auto_deps_main.save_fingerprint") +@patch("pdd.auto_deps_main.FingerprintTransaction") @patch("pdd.auto_deps_main.clear_run_report") @patch("pdd.auto_deps_main.infer_module_identity") @patch("pdd.auto_deps_main.construct_paths") @@ -823,7 +827,7 @@ def test_auto_deps_metadata_finalizes_with_canonical_identity_inplace( mock_construct_paths, mock_infer_identity, mock_clear_run_report, - mock_save_fingerprint, + mock_transaction, mock_ctx, tmp_path: Path, ): @@ -857,8 +861,8 @@ def test_auto_deps_metadata_finalizes_with_canonical_identity_inplace( # Fingerprint persisted with the canonical identity and the cleaned # output prompt path (which equals the original prompt in this case). - mock_save_fingerprint.assert_called_once() - fp_kwargs = mock_save_fingerprint.call_args.kwargs + mock_transaction.assert_called_once() + fp_kwargs = mock_transaction.call_args.kwargs assert fp_kwargs["basename"] == "child" assert fp_kwargs["language"] == "python" assert fp_kwargs["operation"] == "auto-deps" @@ -871,7 +875,7 @@ def test_auto_deps_metadata_finalizes_with_canonical_identity_inplace( # 19. Metadata finalization is skipped when identity cannot be inferred, # i.e. ``infer_module_identity`` returns ``(None, None)``. # --------------------------------------------------------------------------- -@patch("pdd.auto_deps_main.save_fingerprint") +@patch("pdd.auto_deps_main.FingerprintTransaction") @patch("pdd.auto_deps_main.clear_run_report") @patch("pdd.auto_deps_main.infer_module_identity") @patch("pdd.auto_deps_main.construct_paths") @@ -881,7 +885,7 @@ def test_auto_deps_metadata_skipped_on_unknown_identity( mock_construct_paths, mock_infer_identity, mock_clear_run_report, - mock_save_fingerprint, + mock_transaction, mock_ctx, tmp_path: Path, ): @@ -913,7 +917,7 @@ def test_auto_deps_metadata_skipped_on_unknown_identity( ) mock_clear_run_report.assert_not_called() - mock_save_fingerprint.assert_not_called() + mock_transaction.assert_not_called() # Auto-deps still returns its successful result. assert modified_prompt == "Modified prompt with includes" assert total_cost == pytest.approx(0.123456) @@ -921,24 +925,23 @@ def test_auto_deps_metadata_skipped_on_unknown_identity( # --------------------------------------------------------------------------- -# 20. Metadata finalization: clear_run_report failure must not abort the -# subsequent fingerprint save. +# 20. Metadata finalization: clear_run_report failure is a command failure. # --------------------------------------------------------------------------- -@patch("pdd.auto_deps_main.save_fingerprint") +@patch("pdd.auto_deps_main.FingerprintTransaction") @patch("pdd.auto_deps_main.clear_run_report") @patch("pdd.auto_deps_main.infer_module_identity") @patch("pdd.auto_deps_main.construct_paths") @patch("pdd.auto_deps_main.insert_includes") -def test_auto_deps_clear_run_report_error_does_not_block_fingerprint( +def test_auto_deps_clear_run_report_error_fails_finalization( mock_insert_includes, mock_construct_paths, mock_infer_identity, mock_clear_run_report, - mock_save_fingerprint, + mock_transaction, mock_ctx, tmp_path: Path, ): - """If clearing the stale run report fails, the fingerprint must still be saved. + """If stale runtime state cannot be cleared, auto-deps must fail closed. Uses an in-place overwrite (``output == prompt_file``) so finalization is actually attempted — the differing-output guard would otherwise skip @@ -955,18 +958,21 @@ def test_auto_deps_clear_run_report_error_does_not_block_fingerprint( mock_infer_identity.return_value = ("child", "python") mock_clear_run_report.side_effect = OSError("permission denied") - auto_deps_main( - ctx=mock_ctx, - prompt_file=prompt_file, - directory_path="context/", - auto_deps_csv_path=None, - output=None, - force_scan=False, - ) + from pdd.fingerprint_transaction import FingerprintFinalizeError + + with pytest.raises(FingerprintFinalizeError, match="run report cleanup"): + auto_deps_main( + ctx=mock_ctx, + prompt_file=prompt_file, + directory_path="context/", + auto_deps_csv_path=None, + output=None, + force_scan=False, + ) mock_clear_run_report.assert_called_once_with("child", "python", paths=ANY) - mock_save_fingerprint.assert_called_once() - fp_kwargs = mock_save_fingerprint.call_args.kwargs + mock_transaction.assert_called_once() + fp_kwargs = mock_transaction.call_args.kwargs assert fp_kwargs["basename"] == "child" assert fp_kwargs["language"] == "python" diff --git a/tests/test_ci_drift_heal.py b/tests/test_ci_drift_heal.py index 13b07a5fa0..77c7aca9cd 100644 --- a/tests/test_ci_drift_heal.py +++ b/tests/test_ci_drift_heal.py @@ -3288,7 +3288,7 @@ def test_returns_true_when_result_ok(self, tmp_path): return_value=("foo", "python")), \ patch("pdd.sync_determine_operation.read_fingerprint", return_value=fingerprint), \ - patch("pdd.operation_log.save_fingerprint") as mock_save: + patch("pdd.fingerprint_transaction.FingerprintTransaction") as mock_save: assert _run_metadata_sync_safe(str(prompt), str(code)) is True mock_save.assert_called_once() saved_paths = mock_save.call_args.kwargs["paths"] diff --git a/tests/test_fingerprint_invariant.py b/tests/test_fingerprint_invariant.py new file mode 100644 index 0000000000..16a5f5da0a --- /dev/null +++ b/tests/test_fingerprint_invariant.py @@ -0,0 +1,118 @@ +"""Post-command freshness property tests for issue #1926.""" + +from __future__ import annotations + +import ast +from pathlib import Path + +import pytest + +import pdd.sync_determine_operation as sync_state +from pdd.fingerprint_transaction import FingerprintTransaction + + +MUTATING_OPERATIONS = ( + "sync", + "generate", + "example", + "update", + "fix", + "auto-deps", + "ci-heal", +) + + +@pytest.mark.parametrize("operation", MUTATING_OPERATIONS) +def test_successful_mutation_is_immediately_fresh( + operation: str, + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Every supported mutator must converge after its transaction commits.""" + + (tmp_path / ".pddrc").write_text("{}", encoding="utf-8") + paths = { + "prompt": tmp_path / "prompts" / "widget_python.prompt", + "code": tmp_path / "src" / "widget.py", + "example": tmp_path / "examples" / "widget_example.py", + "test": tmp_path / "tests" / "test_widget.py", + } + for key, path in paths.items(): + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(f"{operation}: {key}\n", encoding="utf-8") + + monkeypatch.chdir(tmp_path) + monkeypatch.setattr(sync_state, "get_pdd_file_paths", lambda *args, **kwargs: paths) + + with FingerprintTransaction("widget", "python", operation, paths=paths): + pass + + decision = sync_state.sync_determine_operation( + "widget", + "python", + target_coverage=90.0, + log_mode=True, + skip_tests=True, + skip_verify=True, + ) + if operation == "auto-deps": + # Auto-deps intentionally schedules generation before declaring the + # workflow complete. Prove that the required follow-up also converges. + assert decision.operation == "generate", decision.reason + with FingerprintTransaction("widget", "python", "generate", paths=paths): + pass + decision = sync_state.sync_determine_operation( + "widget", + "python", + target_coverage=90.0, + log_mode=True, + skip_tests=True, + skip_verify=True, + ) + assert decision.operation == "nothing", decision.reason + + +def test_all_production_finalizers_route_through_transaction() -> None: + """Static guard: new command-specific fingerprint writers are forbidden.""" + + repo_root = Path(__file__).resolve().parents[1] + required_modules = { + "pdd/operation_log.py": "save_fingerprint", + "pdd/sync_orchestration.py": "_save_fingerprint_atomic", + "pdd/sync_main.py": "sync_main", + "pdd/auto_deps_main.py": "auto_deps_main", + "pdd/metadata_sync.py": "run_metadata_sync", + "pdd/ci_drift_heal.py": "_run_metadata_sync_safe", + "pdd/update_main.py": "_finalize_single_file_fingerprint", + "pdd/pin_example_hack.py": "_save_operation_fingerprint", + } + + for relative_path, function_name in required_modules.items(): + tree = ast.parse((repo_root / relative_path).read_text(encoding="utf-8")) + function = next( + node + for node in ast.walk(tree) + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) + and node.name == function_name + ) + referenced_names = { + node.id for node in ast.walk(function) if isinstance(node, ast.Name) + } + assert "FingerprintTransaction" in referenced_names, ( + f"{relative_path}:{function_name} bypasses FingerprintTransaction" + ) + + +def test_only_transaction_module_performs_direct_fingerprint_json_write() -> None: + """No top-level PDD module may reintroduce the legacy JSON writer.""" + repo_root = Path(__file__).resolve().parents[1] + transaction_source = (repo_root / "pdd/fingerprint_transaction.py").read_text( + encoding="utf-8" + ) + assert "os.replace" in transaction_source + + for path in (repo_root / "pdd").glob("*.py"): + if path.name in {"fingerprint_transaction.py", "sync_orchestration.py"}: + continue + source = path.read_text(encoding="utf-8") + assert "json.dump(asdict(fingerprint)" not in source, path diff --git a/tests/test_fingerprint_transaction.py b/tests/test_fingerprint_transaction.py new file mode 100644 index 0000000000..3d91c889fe --- /dev/null +++ b/tests/test_fingerprint_transaction.py @@ -0,0 +1,164 @@ +"""Unit and regression tests for transactional fingerprint persistence.""" + +from __future__ import annotations + +import json +from pathlib import Path +from unittest.mock import patch + +import pytest + +from pdd.fingerprint_transaction import ( + FingerprintFinalizeError, + FingerprintTransaction, +) +from pdd.operation_log import get_fingerprint_path, save_fingerprint + + +def _project(tmp_path: Path, name: str = "project") -> tuple[Path, dict[str, Path]]: + root = tmp_path / name + (root / "prompts").mkdir(parents=True) + (root / "src").mkdir() + (root / "examples").mkdir() + (root / "tests").mkdir() + (root / ".pddrc").write_text("{}", encoding="utf-8") + paths = { + "prompt": root / "prompts" / "widget_python.prompt", + "code": root / "src" / "widget.py", + "example": root / "examples" / "widget_example.py", + "test": root / "tests" / "test_widget.py", + } + for key, path in paths.items(): + path.write_text(f"{key} v1\n", encoding="utf-8") + return root, paths + + +def test_transaction_writes_complete_atomic_fingerprint(tmp_path: Path) -> None: + """A clean exit writes the full payload through a same-directory rename.""" + _root, paths = _project(tmp_path) + destination = get_fingerprint_path("widget", "Python", paths=paths) + + real_replace = __import__("os").replace + replace_calls: list[tuple[Path, Path]] = [] + + def recording_replace(source, target): + replace_calls.append((Path(source), Path(target))) + real_replace(source, target) + + with patch("pdd.fingerprint_transaction.os.replace", side_effect=recording_replace): + with FingerprintTransaction("widget", "Python", "generate", paths=paths): + pass + + payload = json.loads(destination.read_text(encoding="utf-8")) + assert payload["command"] == "generate" + assert all(payload[f"{name}_hash"] for name in ("prompt", "code", "example", "test")) + assert replace_calls + temp_path, target_path = replace_calls[0] + assert temp_path.parent == destination.parent + assert target_path == destination + + +def test_null_prompt_hash_fails_without_overwriting_existing_state(tmp_path: Path) -> None: + """Null prompt hashes fail without replacing known-good state (#1305).""" + root, paths = _project(tmp_path) + paths["prompt"].unlink() + destination = root / ".pdd" / "meta" / "widget_python.json" + destination.parent.mkdir(parents=True) + destination.write_text('{"known": "good"}\n', encoding="utf-8") + + with pytest.raises(FingerprintFinalizeError, match="prompt_hash is null"): + with FingerprintTransaction("widget", "python", "example", paths=paths): + pass + + assert json.loads(destination.read_text(encoding="utf-8")) == {"known": "good"} + assert not list(destination.parent.glob("*.tmp")) + + +def test_destination_is_resolved_once_before_cwd_changes( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Changing cwd after entry cannot redirect the resolved metadata root.""" + root_a, paths_a = _project(tmp_path, "a") + root_b, _paths_b = _project(tmp_path, "b") + + transaction = FingerprintTransaction("widget", "python", "update", paths=paths_a) + monkeypatch.chdir(root_b) + with transaction: + pass + + assert (root_a / ".pdd" / "meta" / "widget_python.json").is_file() + assert not (root_b / ".pdd" / "meta" / "widget_python.json").exists() + + +def test_body_exception_and_explicit_skip_do_not_write(tmp_path: Path) -> None: + """Exceptions and intentional skips never write a fingerprint.""" + root, paths = _project(tmp_path) + destination = root / ".pdd" / "meta" / "widget_python.json" + + with pytest.raises(RuntimeError, match="body failed"): + with FingerprintTransaction("widget", "python", "fix", paths=paths): + raise RuntimeError("body failed") + assert not destination.exists() + + with FingerprintTransaction("widget", "python", "fix", paths=paths) as transaction: + transaction.skip("dry-run mode") + assert not destination.exists() + + +def test_replace_failure_is_explicit_and_leaves_no_partial_file(tmp_path: Path) -> None: + """Rename failures include diagnostics and clean their temporary file.""" + root, paths = _project(tmp_path) + destination = root / ".pdd" / "meta" / "widget_python.json" + + with patch("pdd.fingerprint_transaction.os.replace", side_effect=OSError("disk full")): + with pytest.raises(FingerprintFinalizeError) as exc_info: + with FingerprintTransaction("widget", "python", "auto-deps", paths=paths): + pass + + message = str(exc_info.value) + assert "[auto-deps]" in message + assert str(destination) in message + assert "disk full" in message + assert not destination.exists() + assert not list(destination.parent.glob("*.tmp")) + + +def test_save_fingerprint_is_a_thin_transaction_wrapper(tmp_path: Path) -> None: + """The legacy public helper delegates without retaining write logic.""" + _root, paths = _project(tmp_path) + + with patch("pdd.fingerprint_transaction.FingerprintTransaction") as transaction_cls: + save_fingerprint( + "widget", + "python", + "generate", + paths=paths, + cost=1.25, + model="model", + ) + + transaction_cls.assert_called_once_with( + basename="widget", + language="python", + operation="generate", + paths=paths, + cost=1.25, + model="model", + ) + transaction_cls.return_value.__enter__.assert_called_once() + transaction_cls.return_value.__exit__.assert_called_once() + + +def test_include_dependency_override_is_persisted(tmp_path: Path) -> None: + """Pre-rewrite dependency snapshots survive prompt transformations.""" + root, paths = _project(tmp_path) + dependency = root / "context.md" + dependency.write_text("context", encoding="utf-8") + override = {str(dependency): "captured-before-rewrite"} + + with FingerprintTransaction("widget", "python", "sync", paths=paths) as transaction: + transaction.set_include_deps_override(override) + + destination = root / ".pdd" / "meta" / "widget_python.json" + assert json.loads(destination.read_text(encoding="utf-8"))["include_deps"] == override diff --git a/tests/test_metadata_sync.py b/tests/test_metadata_sync.py index b0521f40d1..a8c28c5bc2 100644 --- a/tests/test_metadata_sync.py +++ b/tests/test_metadata_sync.py @@ -286,7 +286,7 @@ def test_dry_run_does_not_write_to_prompt_file(tmp_path: Path) -> None: }), encoding="utf-8") with patch.object(ms, "update_architecture_from_prompt", return_value=_arch_update_ok()), \ patch.object(ms, "clear_run_report") as mock_clear, \ - patch.object(ms, "save_fingerprint") as mock_save: + patch.object(ms, "FingerprintTransaction") as mock_save: run_metadata_sync(ws["prompt_path"], dry_run=True, architecture_path=arch_path) assert ws["prompt_path"].read_text(encoding="utf-8") == original mock_clear.assert_not_called() @@ -302,7 +302,7 @@ def test_dry_run_does_not_call_clear_run_report(tmp_path: Path) -> None: def test_dry_run_does_not_call_save_fingerprint(tmp_path: Path) -> None: ws = _make_workspace(tmp_path) - with patch.object(ms, "save_fingerprint") as mock_save: + with patch.object(ms, "FingerprintTransaction") as mock_save: run_metadata_sync(ws["prompt_path"], dry_run=True) mock_save.assert_not_called() @@ -461,7 +461,7 @@ def test_tag_refresh_is_atomic_temp_then_rename(tmp_path: Path) -> None: def test_tags_stage_preserves_existing_pdd_tags(tmp_path: Path) -> None: ws = _make_workspace(tmp_path) # already has tags before = ws["prompt_path"].read_text(encoding="utf-8") - with patch.object(ms, "save_fingerprint"), patch.object(ms, "clear_run_report"): + with patch.object(ms, "FingerprintTransaction"), patch.object(ms, "clear_run_report"): result = run_metadata_sync(ws["prompt_path"], dry_run=False) after = ws["prompt_path"].read_text(encoding="utf-8") assert before == after @@ -480,7 +480,7 @@ def test_tags_stage_prepends_arch_tags_when_missing(tmp_path: Path) -> None: ]), encoding="utf-8") with patch.object(ms, "update_architecture_from_prompt", return_value=_arch_update_ok()), \ patch.object(ms, "clear_run_report"), \ - patch.object(ms, "save_fingerprint"): + patch.object(ms, "FingerprintTransaction"): result = run_metadata_sync(ws["prompt_path"], dry_run=False, architecture_path=arch_path) after = ws["prompt_path"].read_text(encoding="utf-8") assert "" in after @@ -521,7 +521,7 @@ def _fake_upd(**kwargs: Any) -> Dict[str, Any]: with patch.object(ms, "update_architecture_from_prompt", side_effect=_fake_upd), \ patch.object(ms, "clear_run_report"), \ - patch.object(ms, "save_fingerprint"): + patch.object(ms, "FingerprintTransaction"): run_metadata_sync(ws["prompt_path"], dry_run=False, architecture_path=arch_path) # The architecture stage must receive the in-memory (possibly refreshed) prompt content. assert "prompt_content_override" in captured @@ -541,7 +541,7 @@ def test_architecture_stage_handles_update_returning_failure(tmp_path: Path) -> return_value={"success": False, "updated": False, "changes": {}, "error": "kaboom"}), \ patch.object(ms, "clear_run_report"), \ - patch.object(ms, "save_fingerprint") as mock_save: + patch.object(ms, "FingerprintTransaction") as mock_save: result = run_metadata_sync(ws["prompt_path"], dry_run=False, architecture_path=arch_path) assert result.stages["architecture"].status == "failed" assert "kaboom" in (result.stages["architecture"].reason or "") @@ -565,7 +565,7 @@ def test_architecture_stage_skipped_with_correct_reason(tmp_path: Path) -> None: def test_run_report_clears_via_clear_run_report(tmp_path: Path) -> None: ws = _make_workspace(tmp_path) with patch.object(ms, "clear_run_report") as mock_clear, \ - patch.object(ms, "save_fingerprint"): + patch.object(ms, "FingerprintTransaction"): result = run_metadata_sync(ws["prompt_path"], dry_run=False) mock_clear.assert_called_once_with("demo", "python", paths=ANY) assert result.stages["run_report"].status == "ok" @@ -575,7 +575,7 @@ def test_run_report_skipped_when_identity_cannot_be_inferred(tmp_path: Path) -> ws = _make_workspace(tmp_path) with patch.object(ms, "infer_module_identity", return_value=(None, None)), \ patch.object(ms, "clear_run_report") as mock_clear, \ - patch.object(ms, "save_fingerprint") as mock_save: + patch.object(ms, "FingerprintTransaction") as mock_save: result = run_metadata_sync(ws["prompt_path"], dry_run=False) assert result.stages["run_report"].status == "skipped" mock_clear.assert_not_called() @@ -595,7 +595,7 @@ def test_fingerprint_gated_on_no_prior_failure(tmp_path: Path) -> None: with patch.object(ms, "update_architecture_from_prompt", side_effect=RuntimeError("arch boom")), \ patch.object(ms, "clear_run_report"), \ - patch.object(ms, "save_fingerprint") as mock_save: + patch.object(ms, "FingerprintTransaction") as mock_save: result = run_metadata_sync(ws["prompt_path"], dry_run=False, architecture_path=arch_path) assert result.stages["architecture"].status == "failed" assert result.stages["fingerprint"].status == "skipped" @@ -612,9 +612,10 @@ def _record_clear(*a: Any, **kw: Any) -> None: def _record_save(*a: Any, **kw: Any) -> None: call_order.append("save_fingerprint") + return MagicMock(__enter__=MagicMock(return_value=MagicMock()), __exit__=MagicMock(return_value=False)) with patch.object(ms, "clear_run_report", side_effect=_record_clear), \ - patch.object(ms, "save_fingerprint", side_effect=_record_save): + patch.object(ms, "FingerprintTransaction", side_effect=_record_save): run_metadata_sync(ws["prompt_path"], dry_run=False) # save_fingerprint must run AFTER clear_run_report. assert call_order == ["clear_run_report", "save_fingerprint"], call_order @@ -622,7 +623,7 @@ def _record_save(*a: Any, **kw: Any) -> None: def test_fingerprint_dry_run_does_not_persist(tmp_path: Path) -> None: ws = _make_workspace(tmp_path) - with patch.object(ms, "save_fingerprint") as mock_save: + with patch.object(ms, "FingerprintTransaction") as mock_save: result = run_metadata_sync(ws["prompt_path"], dry_run=True) mock_save.assert_not_called() assert result.stages["fingerprint"].status == "dry_run" @@ -643,7 +644,7 @@ def test_architecture_stage_skipped_when_tags_failed(tmp_path: Path) -> None: patch.object(ms, "update_architecture_from_prompt", return_value=_arch_update_ok()) as mock_upd, \ patch.object(ms, "clear_run_report") as mock_clear, \ - patch.object(ms, "save_fingerprint") as mock_fp: + patch.object(ms, "FingerprintTransaction") as mock_fp: result = run_metadata_sync(ws["prompt_path"], dry_run=False, architecture_path=arch_path) assert result.stages["tags"].status == "failed" # Architecture stage MUST be gated — no on-disk write when tags failed. diff --git a/tests/test_operation_log.py b/tests/test_operation_log.py index 2dc9fa7942..c12181cbe4 100644 --- a/tests/test_operation_log.py +++ b/tests/test_operation_log.py @@ -235,7 +235,10 @@ def test_load_operation_log_compatibility(temp_pdd_env): def test_save_fingerprint(temp_pdd_env): """Test saving fingerprint state in Fingerprint dataclass format.""" basename, lang = "state", "go" - paths = {"prompt": Path("prompts/state_go.prompt")} + prompt_path = temp_pdd_env.parents[1] / "prompts" / "state_go.prompt" + prompt_path.parent.mkdir(parents=True) + prompt_path.write_text("state prompt", encoding="utf-8") + paths = {"prompt": prompt_path} operation_log.save_fingerprint(basename, lang, "op1", paths, 0.5, "gpt-4") @@ -299,10 +302,12 @@ def test_log_operation_decorator_success(temp_pdd_env): def my_command(prompt_file: str): return {"status": "ok"}, 0.15, "gpt-3.5" - prompt_path = "prompts/feat_logic_python.prompt" + prompt_path = temp_pdd_env.parents[1] / "prompts" / "feat_logic_python.prompt" + prompt_path.parent.mkdir(parents=True) + prompt_path.write_text("feature prompt", encoding="utf-8") # Run - result = my_command(prompt_file=prompt_path) + result = my_command(prompt_file=str(prompt_path)) assert result == ({"status": "ok"}, 0.15, "gpt-3.5") @@ -471,8 +476,10 @@ def test_log_operation_decorator_success_clears_run_report(temp_pdd_env): def ok_example(prompt_file: str): return "ok", 0.0, "mock" - prompt_path = f"prompts/{basename}_{lang}.prompt" - ok_example(prompt_file=prompt_path) + prompt_path = temp_pdd_env.parents[1] / "prompts" / f"{basename}_{lang}.prompt" + prompt_path.parent.mkdir(parents=True, exist_ok=True) + prompt_path.write_text("example prompt", encoding="utf-8") + ok_example(prompt_file=str(prompt_path)) assert not rr_path.exists(), ( "clears_run_report must remove the stale run report on success" @@ -600,13 +607,16 @@ def test_fingerprint_path_extension_consistency(tmp_path): # Patch module to use temp directory with patch("pdd.operation_log.META_DIR", str(meta_dir)): + prompt_path = tmp_path / "prompts" / "test_python.prompt" + prompt_path.parent.mkdir(parents=True) + prompt_path.write_text("test prompt", encoding="utf-8") # Write a fingerprint using operation_log save_fingerprint( basename=basename, language=language, operation="test_operation", - paths={"prompt": Path("prompts/test.prompt")}, + paths={"prompt": prompt_path}, cost=0.123, model="test-model" ) @@ -643,11 +653,15 @@ def test_fingerprint_format_compatibility(tmp_path): with patch("pdd.operation_log.META_DIR", str(meta_dir)), \ patch("pdd.sync_determine_operation.get_meta_dir", return_value=meta_dir): + prompt_path = tmp_path / "prompts" / "format_test_python.prompt" + prompt_path.parent.mkdir(parents=True) + prompt_path.write_text("format prompt", encoding="utf-8") + save_fingerprint( basename=basename, language=language, operation="test_op", - paths={}, + paths={"prompt": prompt_path}, cost=0.1, model="test" ) @@ -899,18 +913,18 @@ def test_save_fingerprint_skips_resolution_when_paths_provided_issue_983(temp_pd mock_get_paths.assert_not_called() -def test_save_fingerprint_warns_on_path_resolution_failure_issue_983(temp_pdd_env): +def test_save_fingerprint_fails_on_path_resolution_failure_issue_983(temp_pdd_env): """ - Issue #983: If get_pdd_file_paths raises a recoverable error during - path resolution, save_fingerprint should warn and produce null hashes - (graceful degradation) rather than crashing. + Issue #1926: unresolved paths may not produce a null-hash fingerprint. """ with patch( "pdd.sync_determine_operation.get_pdd_file_paths", side_effect=OSError("prompts dir not found"), ), patch("pdd.operation_log.logger") as mock_logger: - # Should NOT raise - operation_log.save_fingerprint("badmod", "python", operation="generate") + from pdd.fingerprint_transaction import FingerprintFinalizeError + + with pytest.raises(FingerprintFinalizeError, match="prompt_hash is null"): + operation_log.save_fingerprint("badmod", "python", operation="generate") mock_logger.warning.assert_called_once() warning_args = str(mock_logger.warning.call_args) diff --git a/tests/test_update_main.py b/tests/test_update_main.py index 99d8ceea6b..c0a586802a 100644 --- a/tests/test_update_main.py +++ b/tests/test_update_main.py @@ -381,14 +381,15 @@ def test_update_main_regeneration_mode( mock_resolve_pair.return_value = ("modified_code_python.prompt", "modified_code.py") # Act - result = update_main( - ctx=mock_ctx, - input_prompt_file=None, - modified_code_file="modified_code.py", - input_code_file=None, - output=None, - use_git=False - ) + with patch("pdd.update_main.FingerprintTransaction"): + result = update_main( + ctx=mock_ctx, + input_prompt_file=None, + modified_code_file="modified_code.py", + input_code_file=None, + output=None, + use_git=False + ) # Assert # 1. It should resolve the pair to find/create the prompt file @@ -726,7 +727,8 @@ def mock_update_logic( ctx.obj = {"strength": 0.5, "temperature": 0.1, "verbose": False, "time": 0.25, "quiet": False} # Run update_main in repo mode - result = update_main(ctx=ctx, input_prompt_file=None, modified_code_file=None, input_code_file=None, output=None, use_git=False, repo=True) + with patch("pdd.operation_log.save_fingerprint"): + result = update_main(ctx=ctx, input_prompt_file=None, modified_code_file=None, input_code_file=None, output=None, use_git=False, repo=True) # Assert that the update function was called for each pair (all 3 marked as changed) assert mock_update_file_pair.call_count == 3 @@ -775,16 +777,17 @@ def mock_update_logic( ctx = click.Context(click.Command('update')) ctx.obj = {"strength": 0.5, "temperature": 0.1, "verbose": False, "time": 0.25, "quiet": False} - result = update_main( - ctx=ctx, - input_prompt_file=None, - modified_code_file=None, - input_code_file=None, - output=None, - use_git=False, - repo=True, - budget=1.0, - ) + with patch("pdd.operation_log.save_fingerprint"): + result = update_main( + ctx=ctx, + input_prompt_file=None, + modified_code_file=None, + input_code_file=None, + output=None, + use_git=False, + repo=True, + budget=1.0, + ) # First two updates run (0.6 + 0.6), then cap is reached and third is skipped. assert mock_update_file_pair.call_count == 2 @@ -3055,7 +3058,7 @@ def test_default_single_file_update_writes_fresh_fingerprint( with patch("pdd.update_main.get_available_agents", return_value=[]), \ patch("pdd.operation_log.infer_module_identity", return_value=("modified_code", "python")) as mock_infer, \ - patch("pdd.operation_log.save_fingerprint") as mock_save_fp, \ + patch("pdd.update_main.FingerprintTransaction") as mock_save_fp, \ patch("pdd.metadata_sync.run_metadata_sync") as mock_sync: result = update_main( ctx=mock_ctx, @@ -3254,15 +3257,18 @@ def test_finalize_single_file_fingerprint_skips_save_when_run_report_survives_cl import pdd.operation_log as ol monkeypatch.setattr(ol.os, "remove", lambda *a, **kw: None) - _finalize_single_file_fingerprint( - prompt_path=prompt_path, - code_path=code_path, - sync_metadata=False, - dry_run=False, - quiet=False, - cost=0.0, - model="test-model", - ) + from pdd.fingerprint_transaction import FingerprintFinalizeError + + with pytest.raises(FingerprintFinalizeError, match="run report not cleared"): + _finalize_single_file_fingerprint( + prompt_path=prompt_path, + code_path=code_path, + sync_metadata=False, + dry_run=False, + quiet=False, + cost=0.0, + model="test-model", + ) # Acceptance criteria: # - The stale run report still exists (because os.remove was nulled). @@ -3320,15 +3326,18 @@ def test_finalize_single_file_fingerprint_warns_about_stale_run_report_even_when import pdd.operation_log as ol monkeypatch.setattr(ol.os, "remove", lambda *a, **kw: None) - _finalize_single_file_fingerprint( - prompt_path=prompt_path, - code_path=code_path, - sync_metadata=False, - dry_run=False, - quiet=True, # explicit: warning must surface anyway - cost=0.0, - model="test-model", - ) + from pdd.fingerprint_transaction import FingerprintFinalizeError + + with pytest.raises(FingerprintFinalizeError, match="run report not cleared"): + _finalize_single_file_fingerprint( + prompt_path=prompt_path, + code_path=code_path, + sync_metadata=False, + dry_run=False, + quiet=True, # explicit: warning must surface anyway + cost=0.0, + model="test-model", + ) assert not (meta_dir / "foo_python.json").exists() # Rich's Console wraps long lines on narrow terminals; normalize @@ -3444,30 +3453,32 @@ def test_default_single_file_update_skips_fingerprint_when_identity_unknown( mock_save_fp.assert_not_called() -def test_default_single_file_update_swallows_fingerprint_save_failure( +def test_default_single_file_update_fails_when_fingerprint_save_fails( mock_ctx, minimal_input_files, mock_construct_paths, mock_update_prompt, mock_open_file, ): - # A save_fingerprint exception must not break the success return — the - # update tuple is the contract; fingerprint write is best-effort. Use the - # canonical input path so the output-redirected guard does not skip - # save_fingerprint before the OSError can be raised. + # Issue #1926: a successful artifact write may not hide fingerprint + # persistence failure; the command must exit non-zero. + from pdd.fingerprint_transaction import FingerprintFinalizeError + with patch("pdd.update_main.get_available_agents", return_value=[]), \ patch("pdd.operation_log.infer_module_identity", return_value=("mod", "python")), \ - patch("pdd.operation_log.save_fingerprint", side_effect=OSError("disk full")): - result = update_main( - ctx=mock_ctx, - input_prompt_file="updated_prompt.prompt", - modified_code_file=minimal_input_files["modified_code_file"], - input_code_file=minimal_input_files["input_code_file"], - output="updated_prompt.prompt", - use_git=False, - ) - - assert result == ("updated prompt text", 0.123456, "test-model") + patch( + "pdd.update_main.FingerprintTransaction", + side_effect=FingerprintFinalizeError("disk full"), + ): + with pytest.raises(click.exceptions.Exit): + update_main( + ctx=mock_ctx, + input_prompt_file="updated_prompt.prompt", + modified_code_file=minimal_input_files["modified_code_file"], + input_code_file=minimal_input_files["input_code_file"], + output="updated_prompt.prompt", + use_git=False, + ) def test_default_single_file_update_skips_fingerprint_when_output_redirected( @@ -3486,7 +3497,7 @@ def test_default_single_file_update_skips_fingerprint_when_output_redirected( # stale (the same stale-state class the issue is closing). with patch("pdd.update_main.get_available_agents", return_value=[]), \ patch("pdd.operation_log.infer_module_identity", return_value=("mod", "python")) as mock_infer, \ - patch("pdd.operation_log.save_fingerprint") as mock_save_fp: + patch("pdd.update_main.FingerprintTransaction") as mock_save_fp: result = update_main( ctx=mock_ctx, input_prompt_file="some_prompt_file.prompt", @@ -3497,8 +3508,11 @@ def test_default_single_file_update_skips_fingerprint_when_output_redirected( ) assert result is not None - mock_save_fp.assert_not_called() - mock_infer.assert_not_called() + mock_save_fp.assert_called_once() + mock_infer.assert_called_once() + mock_save_fp.return_value.__enter__.return_value.skip.assert_called_once_with( + "output redirected" + ) def test_sync_metadata_true_with_redirected_output_skips_orchestrator( @@ -3846,7 +3860,7 @@ def _update(prompt_file, code_file, ctx, repo, simple=False, strength=None, temp @patch("pdd.update_main.get_git_changed_files", return_value=set()) @patch("pdd.update_main.update_file_pair") @patch("pdd.pddrc_initializer.ensure_pddrc_for_scan") -def test_repo_mode_clear_run_report_failure_warns_and_continues( +def test_repo_mode_clear_run_report_failure_fails_closed( mock_pddrc, mock_update_file_pair, mock_git_changed, @@ -3855,10 +3869,8 @@ def test_repo_mode_clear_run_report_failure_warns_and_continues( temp_git_repo, capsys, ): - # Regression for issue #1057: when clear_run_report raises in repo-mode - # legacy fingerprinting, we must surface a non-fatal warning (when not - # quiet) and still proceed to save_fingerprint, so the user is told that - # runtime verification state may still describe the pre-mutation files. + # Issue #1926: metadata cleanup failure makes finalization, and therefore + # the mutating command, fail. def _update(prompt_file, code_file, ctx, repo, simple=False, strength=None, temperature=None): return { "prompt_file": prompt_file, @@ -3882,28 +3894,23 @@ def _update(prompt_file, code_file, ctx, repo, simple=False, strength=None, temp # quiet=False so the warning is emitted ctx.obj = {"strength": 0.5, "temperature": 0.1, "verbose": False, "time": 0.25, "quiet": False} - result = update_main( - ctx=ctx, - input_prompt_file=None, - modified_code_file=None, - input_code_file=None, - output=None, - use_git=False, - repo=True, - sync_metadata=False, - ) + from pdd.fingerprint_transaction import FingerprintFinalizeError + + with pytest.raises(FingerprintFinalizeError, match="run report clear failed"): + update_main( + ctx=ctx, + input_prompt_file=None, + modified_code_file=None, + input_code_file=None, + output=None, + use_git=False, + repo=True, + sync_metadata=False, + ) - assert result is not None assert mock_sync.call_count == 0 - # clear_run_report attempted for each successful pair - assert mock_clear_rr.call_count == mock_update_file_pair.call_count - # save_fingerprint still called per pair despite clear failure - assert mock_save_fp.call_count == mock_update_file_pair.call_count - assert mock_save_fp.call_count >= 1 - # Warning surfaced to the user - out = capsys.readouterr().out - assert "Run report clear failed" in out - assert "disk full" in out + assert mock_clear_rr.call_count == 1 + mock_save_fp.assert_not_called() @patch("pdd.architecture_sync.update_architecture_from_prompt", return_value={"success": False, "updated": False, "changes": {}}) @@ -3911,7 +3918,7 @@ def _update(prompt_file, code_file, ctx, repo, simple=False, strength=None, temp @patch("pdd.update_main.get_git_changed_files", return_value=set()) @patch("pdd.update_main.update_file_pair") @patch("pdd.pddrc_initializer.ensure_pddrc_for_scan") -def test_repo_mode_clear_run_report_silent_unlink_failure_warns( +def test_repo_mode_clear_run_report_silent_unlink_failure_fails_closed( mock_pddrc, mock_update_file_pair, mock_git_changed, @@ -3958,22 +3965,24 @@ def _update(prompt_file, code_file, ctx, repo, simple=False, strength=None, temp # quiet=False so the defensive warning is emitted ctx.obj = {"strength": 0.5, "temperature": 0.1, "verbose": False, "time": 0.25, "quiet": False} - result = update_main( - ctx=ctx, - input_prompt_file=None, - modified_code_file=None, - input_code_file=None, - output=None, - use_git=False, - repo=True, - sync_metadata=False, - ) + from pdd.fingerprint_transaction import FingerprintFinalizeError + + with pytest.raises(FingerprintFinalizeError, match="run report not cleared"): + update_main( + ctx=ctx, + input_prompt_file=None, + modified_code_file=None, + input_code_file=None, + output=None, + use_git=False, + repo=True, + sync_metadata=False, + ) - assert result is not None assert mock_sync.call_count == 0 # clear_run_report attempted for each successful pair, but it silently # did nothing (no exception raised, no file removed). - assert mock_clear_rr.call_count == mock_update_file_pair.call_count + assert mock_clear_rr.call_count == 1 # save_fingerprint must be SKIPPED when the stale run report remains, # so we don't claim finalized metadata while runtime verification still # describes the pre-update files (issue #1057). @@ -3981,7 +3990,6 @@ def _update(prompt_file, code_file, ctx, repo, simple=False, strength=None, temp # Defensive warning surfaced to the user because the report file still # exists after clear_run_report returned. out = capsys.readouterr().out - assert "Run report clear failed" in out assert "still exists after" in out # The file is indeed still on disk (the whole point of this regression). assert stale_report.exists() @@ -4421,6 +4429,7 @@ def _run_prd_sync_update(repo_root, ctx, sync_metadata=False): patch("pdd.update_main.git.Repo") as mock_repo, patch("pdd.update_main.os.getcwd", return_value=str(repo_root)), patch("pdd.pddrc_initializer.ensure_pddrc_for_scan"), + patch("pdd.operation_log.save_fingerprint"), ): mock_repo.return_value.working_tree_dir = str(repo_root) return update_main( From 63ad408cd96943b9c1de3f283ace64738cde3b05 Mon Sep 17 00:00:00 2001 From: Serhan Date: Fri, 10 Jul 2026 18:11:45 -0700 Subject: [PATCH 3/5] Enforce transactional fingerprint finalization --- .pdd/meta/fingerprint_transaction_python.json | 19 + .../fingerprint_transaction_python_run.json | 11 + README.md | 2 +- architecture.json | 21 +- context/fingerprint_transaction_example.py | 59 +++ pdd/auto_deps_main.py | 113 +++--- pdd/ci_drift_heal.py | 30 +- pdd/commands/fix.py | 2 +- pdd/commands/maintenance.py | 3 + pdd/fingerprint_transaction.py | 246 ++++++++++++ pdd/metadata_sync.py | 19 +- pdd/operation_log.py | 185 +++++---- pdd/pin_example_hack.py | 86 ++-- pdd/prompts/auto_deps_main_python.prompt | 17 +- pdd/prompts/ci_drift_heal_python.prompt | 3 +- pdd/prompts/commands/fix_python.prompt | 2 +- .../fingerprint_transaction_python.prompt | 93 ++--- pdd/prompts/metadata_sync_python.prompt | 8 +- pdd/prompts/operation_log_python.prompt | 13 +- pdd/prompts/sync_main_python.prompt | 4 +- pdd/prompts/sync_orchestration_python.prompt | 7 +- pdd/prompts/update_main_python.prompt | 19 +- pdd/sync_orchestration.py | 137 ++++--- pdd/update_main.py | 196 ++++----- tests/test_auto_deps_main.py | 72 ++-- tests/test_fingerprint_invariant.py | 340 ++++++++++++++++ tests/test_fingerprint_transaction.py | 373 ++++++++++++++++++ tests/test_issue_1714_sync_stall.py | 3 +- tests/test_metadata_sync.py | 2 +- tests/test_operation_log.py | 89 +++-- tests/test_update_main.py | 211 +++++----- 31 files changed, 1708 insertions(+), 677 deletions(-) create mode 100644 .pdd/meta/fingerprint_transaction_python.json create mode 100644 .pdd/meta/fingerprint_transaction_python_run.json create mode 100644 context/fingerprint_transaction_example.py create mode 100644 pdd/fingerprint_transaction.py create mode 100644 tests/test_fingerprint_invariant.py create mode 100644 tests/test_fingerprint_transaction.py diff --git a/.pdd/meta/fingerprint_transaction_python.json b/.pdd/meta/fingerprint_transaction_python.json new file mode 100644 index 0000000000..439babe813 --- /dev/null +++ b/.pdd/meta/fingerprint_transaction_python.json @@ -0,0 +1,19 @@ +{ + "pdd_version": "0.0.301.dev0", + "timestamp": "2026-07-11T00:48:46.432564+00:00", + "command": "fix", + "prompt_hash": "fd3e7d5d66c0676602864da3c13655d9d5ce1dd7c47daf687ed700f148e9ec27", + "code_hash": "e268904612b5ef15567878153db131176b5dbe5889830db1d313170c1ccce9ea", + "example_hash": "1c8d97703bf0d55809b9508bc2be174dfa40b1f55465aec2567b95109ef38398", + "test_hash": "7d7c640c05c9eb5d9bba0d2654df887f586b7944b2767ee751ba155a2324d4c0", + "test_files": { + "test_fingerprint_transaction.py": "7d7c640c05c9eb5d9bba0d2654df887f586b7944b2767ee751ba155a2324d4c0" + }, + "include_deps": { + "context/fingerprint_transaction_example.py": "1c8d97703bf0d55809b9508bc2be174dfa40b1f55465aec2567b95109ef38398", + "context/python_preamble.prompt": "0388ed131bf986f8752e1bc4c81e4da0460cfe2908ec8c60b1314edbab768254", + "pdd/json_atomic.py": "1e464b0e63e8f7a48727a0b2c0cd5e91ae766bd8182c205bf069aa8f6ad81351", + "pdd/operation_log.py": "98c26af6e32c82e1bd26e1abd03feb7a3e229178bf4c66ce01bd003d2bcaf566", + "pdd/sync_determine_operation.py": "e616ac746674a9f1e8be84bf3093d5e1f45ff0175bc3767b758f2d17b2c54132" + } +} diff --git a/.pdd/meta/fingerprint_transaction_python_run.json b/.pdd/meta/fingerprint_transaction_python_run.json new file mode 100644 index 0000000000..7248d78003 --- /dev/null +++ b/.pdd/meta/fingerprint_transaction_python_run.json @@ -0,0 +1,11 @@ +{ + "timestamp": "2026-07-11T00:48:20.573849+00:00", + "exit_code": 0, + "tests_passed": 16, + "tests_failed": 0, + "coverage": 100.0, + "test_hash": "7d7c640c05c9eb5d9bba0d2654df887f586b7944b2767ee751ba155a2324d4c0", + "test_files": { + "test_fingerprint_transaction.py": "7d7c640c05c9eb5d9bba0d2654df887f586b7944b2767ee751ba155a2324d4c0" + } +} \ No newline at end of file diff --git a/README.md b/README.md index fb820fea3d..7f7af5233b 100644 --- a/README.md +++ b/README.md @@ -2756,7 +2756,7 @@ Options: - `--git`: Use git history to find the original code file, eliminating the need for the `INPUT_CODE_FILE` argument. - `--extensions EXTENSIONS`: In repository-wide mode, filter the update to only include files with the specified comma-separated extensions (e.g., `py,js,ts`). - `--simple`: Use the legacy 2-stage LLM update process instead of the default agentic mode. Useful when agentic CLIs are not available or for faster updates. -- `--sync-metadata`: After the prompt update, run the shared metadata-sync orchestrator so prompt PDD tags, `architecture.json` entries, run reports, and fingerprint state are reconciled in one step. Works in single-file, regeneration, and repo modes. **Fingerprint note:** default single-file/regeneration `pdd update ` already finalizes the per-target fingerprint (`.pdd/meta/_.json`) on success, and logs a skip reason when finalization is intentionally bypassed; `--sync-metadata` does not gate that behavior. Before writing the single-file/regeneration fingerprint, the command clears the affected module's stale `.pdd/meta/__run.json` runtime-verification report and re-checks it is gone; if the stale report survives the clear attempt (for example, silent `os.remove` failure on permissions/locks), the fingerprint write is skipped with a yellow warning so a fresh fingerprint cannot coexist with stale runtime state — best-effort, the successful `(prompt, cost, model)` update tuple is preserved either way (closing issue [#1106](https://github.com/promptdriven/pdd/issues/1106)). The stale-report warning intentionally surfaces even under `--quiet`, because it describes a real metadata-consistency problem the operator should learn about regardless of other log suppression. Default repo-mode `pdd update --repo` likewise finalizes per-pair fingerprints and, before writing each fingerprint, clears the affected module's stale `.pdd/meta/__run.json` runtime-verification report so metadata and runtime state stay in lock-step (clear failures are surfaced as non-fatal warnings, and if the stale `_run.json` still exists after the clear attempt the fingerprint write is skipped so a fresh fingerprint cannot coexist with stale runtime state — closing issue [#1057](https://github.com/promptdriven/pdd/issues/1057)). Without this flag, the broader prompt-tag/architecture/run-report orchestrator is not run and those layers must be reconciled with separate commands. **Scope note:** the `tags` stage currently *preserves* existing PDD tags and only *seeds* tags from the matching `architecture.json` entry when a prompt has none — LLM-first **refresh** of stale-but-present tags is tracked at issue [#870](https://github.com/promptdriven/pdd/issues/870) and is not invoked by this orchestrator. When a prompt has zero PDD tags AND no architecture entry, the `tags` stage reports `skipped` (never `ok`) so operators see honest status. On any stage `failed`, `pdd update --sync-metadata` exits non-zero so CI auto-heal does not treat a half-finalized update as healed. +- `--sync-metadata`: After the prompt update, run the shared metadata-sync orchestrator so prompt PDD tags, `architecture.json` entries, run reports, and fingerprint state are reconciled in one step. Works in single-file, regeneration, and repo modes. **Fingerprint note:** without this flag, every successful single-file/regeneration update and every successful `--repo` pair finalizes through the shared `FingerprintTransaction` path. The command first resolves the complete unit path set, clears the affected stale `_run.json`, verifies it is gone, and atomically writes the new fingerprint. Identity, cleanup, hashing, or persistence failure is a hard non-zero command failure; the update cannot return a false-green success tuple after mutating an artifact. With `--sync-metadata`, the orchestrator owns that fingerprint stage, so the default finalizer intentionally skips instead of double-writing. The stale-report warning still surfaces under `--quiet` because it describes a real consistency failure. Without this flag, the broader prompt-tag/architecture stages are not run and must be reconciled separately. **Scope note:** the `tags` stage currently *preserves* existing PDD tags and only *seeds* tags from the matching `architecture.json` entry when a prompt has none — LLM-first **refresh** of stale-but-present tags is tracked at issue [#870](https://github.com/promptdriven/pdd/issues/870) and is not invoked by this orchestrator. When a prompt has zero PDD tags AND no architecture entry, the `tags` stage reports `skipped` (never `ok`) so operators see honest status. On any stage `failed`, `pdd update --sync-metadata` exits non-zero so CI auto-heal does not treat a half-finalized update as healed. Example (Metadata Sync): ```bash diff --git a/architecture.json b/architecture.json index 1e68f2bb0e..ff5a09dcdb 100644 --- a/architecture.json +++ b/architecture.json @@ -5493,7 +5493,7 @@ "code_generator_main_python.prompt", "agentic_sync_runner_python.prompt", "compressed_sync_context_python.prompt", - "fingerprint_transaction_python.prompt" + "operation_log_python.prompt" ], "priority": 148, "filename": "sync_main_python.prompt", @@ -7437,7 +7437,7 @@ "functions": [ { "name": "log_operation", - "signature": "(updates_fingerprint: bool, updates_run_report: bool, clears_run_report: bool) -> Callable", + "signature": "(operation: str, updates_fingerprint: bool = False, updates_run_report: bool = False, clears_run_report: bool = False) -> Callable", "returns": "Callable" }, { @@ -7449,6 +7449,11 @@ "name": "log_event", "signature": "(basename: str, language: str, event: str, details: Mapping[str, Any] | None = None, project_root=None, paths=None, compression=None, agentic_fallback=None) -> None", "returns": "None" + }, + { + "name": "resolve_fingerprint_paths", + "signature": "(basename: str, language: str, prompt_file: Union[str, Path], *, paths: Optional[Mapping[str, Any]] = None) -> Dict[str, Any]", + "returns": "Dict[str, Any]" } ] } @@ -8819,8 +8824,7 @@ "auto_include_python.prompt", "agentic_langtest_python.prompt", "architecture_sync_python.prompt", - "metadata_sync_python.prompt", - "fingerprint_transaction_python.prompt" + "metadata_sync_python.prompt" ], "priority": 80, "filename": "ci_drift_heal_python.prompt", @@ -9663,8 +9667,7 @@ "dependencies": [ "architecture_sync_python.prompt", "sync_determine_operation_python.prompt", - "operation_log_python.prompt", - "fingerprint_transaction_python.prompt" + "operation_log_python.prompt" ], "priority": 5, "filename": "metadata_sync_python.prompt", @@ -11313,8 +11316,8 @@ } }, { - "reason": "Single enforced code path for all PDD fingerprint writes — atomic temp-file + os.replace with commit-or-fail semantics.", - "description": "Context manager that owns: eager identity and path resolution at entry, atomic file write (temp-file + os.replace) at exit, null-hash validation, commit-or-fail exit semantics, and dry-run/redirect skip control. Used by every PDD mutating command (sync, generate, example, update, fix, auto-deps, ci-heal).", + "reason": "Single enforced code path for complete, atomic PDD fingerprint finalization with commit-or-fail semantics.", + "description": "Context manager that eagerly resolves the fingerprint destination, builds and validates the canonical payload, and either writes through the shared atomic JSON primitive or buffers the identical payload in sync's outer atomic state. Used by every PDD mutating command (sync, generate, example, update, fix, auto-deps, CI heal).", "dependencies": [ "sync_determine_operation_python.prompt" ], @@ -11331,7 +11334,7 @@ "functions": [ { "name": "FingerprintTransaction.__init__", - "signature": "(basename: str, language: str, operation: str, paths: Optional[Dict[str, Path]] = None, cost: float = 0.0, model: str = '') -> None", + "signature": "(basename: str, language: str, operation: str, paths: Optional[Dict[str, Path]] = None, cost: float = 0.0, model: str = '', *, atomic_state: Optional[Any] = None) -> None", "returns": "None" }, { diff --git a/context/fingerprint_transaction_example.py b/context/fingerprint_transaction_example.py new file mode 100644 index 0000000000..1df8fa7abd --- /dev/null +++ b/context/fingerprint_transaction_example.py @@ -0,0 +1,59 @@ +"""Examples of the shared fingerprint finalization boundary.""" +from __future__ import annotations + +from pathlib import Path +from typing import Any + +from pdd.fingerprint_transaction import FingerprintTransaction + + +def finalize_generated_unit( + prompt_path: Path, + code_path: Path, + *, + dry_run: bool = False, +) -> Path: + """Persist a complete fingerprint, or explicitly skip a dry run.""" + transaction = FingerprintTransaction( + "sample", + "python", + "generate", + paths={"prompt": prompt_path, "code": code_path}, + cost=0.01, + model="example-model", + ) + with transaction: + if dry_run: + transaction.skip("dry-run") + return transaction.fingerprint_path + + +class FingerprintBuffer: + """Minimal duck-typed buffer used by an outer atomic-state context.""" + + def __init__(self) -> None: + self.pending: tuple[dict[str, Any], Path, str | None] | None = None + + def set_fingerprint( + self, + payload: dict[str, Any], + path: Path, + *, + operation: str | None = None, + ) -> None: + self.pending = (payload, path, operation) + + +def buffer_sync_fingerprint( + paths: dict[str, Path], + buffer: FingerprintBuffer, +) -> None: + """Build the canonical payload while deferring persistence to sync.""" + with FingerprintTransaction( + "sample", + "python", + "sync", + paths=paths, + atomic_state=buffer, + ): + pass diff --git a/pdd/auto_deps_main.py b/pdd/auto_deps_main.py index 101697c4cb..b764b119e5 100644 --- a/pdd/auto_deps_main.py +++ b/pdd/auto_deps_main.py @@ -13,10 +13,14 @@ from .validate_prompt_includes import sanitize_prompt_output from .auto_deps_architecture import merge_auto_deps_includes_from_cwd from .operation_log import ( + _clear_run_report_before_fingerprint, + get_fingerprint_path, infer_module_identity, + resolve_fingerprint_paths, save_fingerprint, - clear_run_report, - get_run_report_path, +) +from .fingerprint_transaction import ( + FingerprintFinalizeError, ) @@ -188,8 +192,8 @@ def auto_deps_main( # written (``output_path``); in in-place mode this resolves to the # canonical ``_`` module, in the default CLI # flow it resolves to the ``__with_deps`` - # derivative. Errors are surfaced as warnings and never mask a - # successful auto-deps result. + # derivative. Finalization is part of command success: a real + # mutation may not return green without a persisted fingerprint. # # ``_skip_finalization=True`` is set by ``pdd sync`` because it # invokes auto-deps with a temp ``__with_deps`` @@ -201,76 +205,55 @@ def auto_deps_main( # run-report clear on its side). if _skip_finalization: return cleaned_prompt, total_cost, model_name - try: - basename, language = infer_module_identity(Path(output_path)) - if basename is None or language is None: - if not quiet: - console.print( - f"[yellow]Warning: Could not infer module identity for " - f"{output_path}; skipping fingerprint finalization.[/yellow]" - ) - else: - # Issue #1211: route clear/verify/save through the same - # `paths` hint (the prompt path we just wrote) so all - # three target the subproject's .pdd/meta — not a parent - # CWD orphan — when auto-deps is run from above the - # subproject root. - _autodeps_paths = {"prompt": Path(output_path)} - # Clear stale run report; do not let its failure block fingerprint save - try: - clear_run_report(basename, language, paths=_autodeps_paths) - except Exception as cr_exc: - if not quiet: - console.print( - f"[yellow]Warning: Failed to clear run report for " - f"{basename}_{language}: {cr_exc}[/yellow]" - ) - # Defensive: clear_run_report() in pdd.operation_log silently swallows - # OSError on the actual unlink, so verify the report is really gone. - try: - _stale_report_path = get_run_report_path( - basename, language, paths=_autodeps_paths - ) - if _stale_report_path.exists(): - if not quiet: - console.print( - f"[yellow]Warning: clear_run_report did not remove " - f"{_stale_report_path}; downstream sync may still see " - f"stale results.[/yellow]" - ) - except Exception as _vrf_exc: - if not quiet: - console.print( - f"[yellow]Warning: could not verify run-report removal: " - f"{_vrf_exc}[/yellow]" - ) - try: - save_fingerprint( - basename=basename, - language=language, - operation="auto-deps", - paths=_autodeps_paths, - cost=total_cost, - model=model_name, - ) - except Exception as fp_exc: - if not quiet: - console.print( - f"[yellow]Warning: Failed to save fingerprint for " - f"{basename}_{language}: {fp_exc}[/yellow]" - ) - except Exception as meta_exc: - # Never mask a successful auto-deps result on metadata errors + basename, language = infer_module_identity(Path(output_path)) + if basename is None or language is None: if not quiet: console.print( - f"[yellow]Warning: Metadata finalization encountered an error: {meta_exc}[/yellow]" + "[yellow]Skipping fingerprint finalization for " + f"unmanaged output {output_path}: could not infer " + "module identity.[/yellow]" ) + return cleaned_prompt, total_cost, model_name + + # Issue #1211: route clear and commit through the same known path + # so both target the subproject's .pdd/meta directory. + _autodeps_paths = resolve_fingerprint_paths( + basename, + language, + output_path, + paths={"prompt": Path(output_path)}, + ) + if not _clear_run_report_before_fingerprint( + basename, + language, + paths=_autodeps_paths, + ): + raise FingerprintFinalizeError( + "auto-deps", + get_fingerprint_path( + basename, + language, + paths=_autodeps_paths, + ), + "run report not cleared", + ) + save_fingerprint( + basename=basename, + language=language, + operation="auto-deps", + paths=_autodeps_paths, + cost=total_cost, + model=model_name, + ) return cleaned_prompt, total_cost, model_name except click.Abort: # Re-raise to allow orchestrators (e.g. pdd sync) to stop the loop raise + except FingerprintFinalizeError: + # Commit-or-fail: the Click boundary must see a non-zero failure. + raise except Exception as exc: if not quiet: console.print(f"[red]Error in auto-deps: {exc}[/red]") diff --git a/pdd/ci_drift_heal.py b/pdd/ci_drift_heal.py index a87dd695d0..d707e27adf 100644 --- a/pdd/ci_drift_heal.py +++ b/pdd/ci_drift_heal.py @@ -792,21 +792,28 @@ def _run_metadata_sync_safe( ok = bool(getattr(result, "ok", False)) if ok: try: - from pdd.operation_log import infer_module_identity, save_fingerprint + from pdd.operation_log import ( + infer_module_identity, + resolve_fingerprint_paths, + save_fingerprint, + ) from pdd.sync_determine_operation import read_fingerprint basename, language = infer_module_identity(str(p)) if basename is None or language is None: raise ValueError("could not determine module identity from prompt path") - # Issue #1211: build paths directly from the known subproject - # paths rather than calling get_pdd_file_paths (which resolves - # from CWD and would return parent-repo paths in parent-CWD mode). - # Example/test-file hash validation is skipped here since those - # keys are not present in this minimal paths dict; that gap is - # tracked at issue #870 alongside LLM-first tag refresh. + # Issue #1211: seed resolution with the known subproject paths. + # The shared resolver anchors discovery at the real prompt, keeps + # these explicit paths authoritative, and adds existing + # example/test files so their hashes are not erased (#1926). if code_p is None: raise ValueError("authoritative prompt/code paths unavailable: code_path not provided") - paths: Dict[str, Any] = {"prompt": p, "code": code_p} + paths: Dict[str, Any] = resolve_fingerprint_paths( + basename, + language, + p, + paths={"prompt": p, "code": code_p}, + ) # Preserve the previous user-facing command so the released # `sync_determine_operation._is_workflow_complete` (which only # accepts verify/test/fix/update as complete) keeps recognizing @@ -833,9 +840,12 @@ def _run_metadata_sync_safe( or not fingerprint.code_hash ): raise ValueError("fingerprint missing prompt/code hashes") - except Exception: + except Exception as exc: basename = str(prompt_path) - print(f"metadata finalization failed: fingerprint refresh failed for {basename}") + print( + "metadata finalization failed: fingerprint refresh failed for " + f"{basename}: {exc}" + ) return False meta_rel = f".pdd/meta/{_safe_basename(basename)}_{language}.json" print(f"metadata finalized for {basename}: {meta_rel}") diff --git a/pdd/commands/fix.py b/pdd/commands/fix.py index a49dbd765f..5fe5a47ade 100644 --- a/pdd/commands/fix.py +++ b/pdd/commands/fix.py @@ -122,7 +122,7 @@ def _is_user_story_file(value: str) -> bool: help="Behavior when context compression fails (default: full).", ) @click.pass_context -@log_operation(operation="fix", clears_run_report=True) +@log_operation(operation="fix", clears_run_report=True, updates_fingerprint=True) @track_cost def fix( ctx: click.Context, diff --git a/pdd/commands/maintenance.py b/pdd/commands/maintenance.py index b8ca343784..c8b0fce853 100644 --- a/pdd/commands/maintenance.py +++ b/pdd/commands/maintenance.py @@ -14,6 +14,7 @@ from ..track_cost import track_cost from ..core.errors import handle_error from ..sync_determine_operation import AmbiguousModuleError +from ..fingerprint_transaction import FingerprintFinalizeError from ..core.utils import _run_setup_utility, echo_model_line from ..evidence_manifest import ( collect_sync_evidence_paths, @@ -992,6 +993,8 @@ def auto_deps( return result, total_cost, model_name except click.Abort: raise + except FingerprintFinalizeError as exception: + raise click.ClickException(str(exception)) from exception except Exception as exception: handle_error(exception, "auto-deps", ctx.obj.get("quiet", False)) return None diff --git a/pdd/fingerprint_transaction.py b/pdd/fingerprint_transaction.py new file mode 100644 index 0000000000..51463477e0 --- /dev/null +++ b/pdd/fingerprint_transaction.py @@ -0,0 +1,246 @@ +"""Transactional fingerprint finalization. + +Every successful fingerprint write is prepared and validated here. Callers may +either commit immediately or provide the sync orchestrator's atomic-state buffer; +both paths use the same payload construction and null-hash checks. + +``os.replace`` is atomic on POSIX when source and destination share a filesystem. +Windows has weaker replacement guarantees when the destination already exists. +""" +from __future__ import annotations + +import logging +from dataclasses import asdict +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Dict, Mapping, Optional + +from . import __version__ +from .json_atomic import atomic_write_json +from .operation_log import get_fingerprint_path +from .sync_determine_operation import ( + Fingerprint, + calculate_current_hashes, + get_pdd_file_paths, + read_fingerprint, +) + +logger = logging.getLogger(__name__) + + +class FingerprintFinalizeError(RuntimeError): + """Raised when a fingerprint cannot be finalized safely.""" + + def __init__(self, operation: str, fingerprint_path: Path, cause: object): + self.operation = operation + self.fingerprint_path = Path(fingerprint_path) + self.cause = cause + super().__init__( + f"[{operation}] fingerprint finalization failed for " + f"{self.fingerprint_path}: {cause}" + ) + + +def _coerce_paths(paths: Mapping[str, Any]) -> Dict[str, Any]: + """Normalize path hints without changing their caller-selected scope.""" + normalized: Dict[str, Any] = {} + for key, value in paths.items(): + if key == "test_files": + if value is None: + normalized[key] = [] + else: + normalized[key] = [ + item if isinstance(item, Path) else Path(item) + for item in value + ] + elif value is None or isinstance(value, Path): + normalized[key] = value + else: + normalized[key] = Path(value) + return normalized + + +class FingerprintTransaction: + """Commit a complete fingerprint on clean context-manager exit. + + ``atomic_state`` is the sync orchestrator's optional metadata buffer. It is + deliberately duck-typed so this leaf module never imports the orchestrator. + """ + + def __init__( + self, + basename: str, + language: str, + operation: str, + paths: Optional[Dict[str, Path]] = None, + cost: float = 0.0, + model: str = "", + *, + atomic_state: Optional[Any] = None, + ) -> None: + self._basename = basename + self._language = language.lower() + self._operation = operation + self._cost = float(cost or 0.0) + self._model = model + self._atomic_state = atomic_state + self._skipped = False + self._skip_reason: Optional[str] = None + self._include_deps_override: Optional[Dict[str, str]] = None + + fallback_path = ( + Path(".pdd") + / "meta" + / f"{basename.replace('/', '_')}_{self._language}.json" + ) + try: + resolved_paths: Mapping[str, Any] + if paths: + # Preserve issue #983: explicit paths are authoritative and must + # not be replaced by CWD-based discovery. + resolved_paths = paths + else: + resolved_paths = get_pdd_file_paths(basename, self._language) + self._paths = _coerce_paths(resolved_paths) + self._fingerprint_path = get_fingerprint_path( + basename, + self._language, + paths=self._paths, + ) + except Exception as exc: + raise FingerprintFinalizeError( + operation, + fallback_path, + f"path resolution failed: {exc}", + ) from exc + + @property + def fingerprint_path(self) -> Path: + """Return the eagerly resolved destination path.""" + return self._fingerprint_path + + def __enter__(self) -> "FingerprintTransaction": + return self + + def skip(self, reason: str) -> None: + """Suppress finalization for an intentional non-mutating path.""" + self._skipped = True + self._skip_reason = str(reason) + logger.info( + "FingerprintTransaction.skip for %s/%s (%s): %s", + self._basename, + self._language, + self._operation, + self._skip_reason, + ) + + def set_include_deps_override(self, deps: Dict[str, str]) -> None: + """Use a pre-mutation include graph for hashing and persistence.""" + self._include_deps_override = dict(deps) + + def _validate_hashes(self, current_hashes: Mapping[str, Any]) -> None: + """Reject fingerprints that cannot represent their existing inputs.""" + if current_hashes.get("prompt_hash") is None: + raise FingerprintFinalizeError( + self._operation, + self._fingerprint_path, + "prompt_hash is null", + ) + + for key in ("code", "example", "test"): + value = self._paths.get(key) + if isinstance(value, Path) and value.exists(): + hash_field = f"{key}_hash" + if current_hashes.get(hash_field) is None: + raise FingerprintFinalizeError( + self._operation, + self._fingerprint_path, + f"{hash_field} is null for existing path {value}", + ) + + expected_tests = self._paths.get("test_files") or [] + actual_tests = current_hashes.get("test_files") or {} + for test_path in expected_tests: + if isinstance(test_path, Path) and test_path.exists(): + if actual_tests.get(test_path.name) is None: + raise FingerprintFinalizeError( + self._operation, + self._fingerprint_path, + f"test_files hash is null for existing path {test_path}", + ) + + def _build_payload(self) -> Dict[str, Any]: + previous = read_fingerprint( + self._basename, + self._language, + paths=self._paths, + ) + stored_deps = ( + self._include_deps_override + if self._include_deps_override is not None + else (previous.include_deps if previous else None) + ) + current_hashes = calculate_current_hashes( + self._paths, + stored_include_deps=stored_deps, + ) + self._validate_hashes(current_hashes) + + include_deps = ( + self._include_deps_override + if self._include_deps_override is not None + else current_hashes.get("include_deps") + ) + fingerprint = Fingerprint( + pdd_version=__version__, + timestamp=datetime.now(timezone.utc).isoformat(), + command=self._operation, + prompt_hash=current_hashes.get("prompt_hash"), + code_hash=current_hashes.get("code_hash"), + example_hash=current_hashes.get("example_hash"), + test_hash=current_hashes.get("test_hash"), + test_files=current_hashes.get("test_files"), + include_deps=include_deps, + ) + return asdict(fingerprint) + + def _commit(self) -> None: + try: + payload = self._build_payload() + if self._atomic_state is not None: + setter = getattr(self._atomic_state, "set_fingerprint", None) + if not callable(setter): + raise TypeError("atomic_state does not provide set_fingerprint()") + setter( + payload, + self._fingerprint_path, + operation=self._operation, + ) + else: + atomic_write_json(self._fingerprint_path, payload) + except FingerprintFinalizeError: + raise + except Exception as exc: + raise FingerprintFinalizeError( + self._operation, + self._fingerprint_path, + exc, + ) from exc + + def __exit__(self, exc_type, exc_val, exc_tb) -> bool: + if exc_type is not None: + return False + if self._skipped: + logger.debug( + "Fingerprint commit suppressed for %s/%s (%s): %s", + self._basename, + self._language, + self._operation, + self._skip_reason, + ) + return False + self._commit() + return False + + +__all__ = ["FingerprintFinalizeError", "FingerprintTransaction"] diff --git a/pdd/metadata_sync.py b/pdd/metadata_sync.py index 00c92e99d4..96242fc60e 100644 --- a/pdd/metadata_sync.py +++ b/pdd/metadata_sync.py @@ -17,7 +17,12 @@ update_architecture_from_prompt, ) from .architecture_registry import find_architecture_for_project -from .operation_log import clear_run_report, infer_module_identity, save_fingerprint +from .operation_log import ( + clear_run_report, + infer_module_identity, + resolve_fingerprint_paths, + save_fingerprint, +) _THEME = Theme( @@ -482,12 +487,18 @@ def run_metadata_sync( basename, language = infer_module_identity(prompt_path) if not basename or not language: reason = "could not infer (basename, language) for fingerprint" - result.stages["fingerprint"] = StageStatus(status="skipped", reason=reason) - _stage_log_exit("fingerprint", "skipped", reason) + result.stages["fingerprint"] = StageStatus(status="failed", reason=reason) + _stage_log_exit("fingerprint", "failed", reason) else: paths: Dict[str, Path] = {"prompt": prompt_path} if code_path is not None: paths["code"] = code_path + paths = resolve_fingerprint_paths( + basename, + language, + prompt_path, + paths=paths, + ) detail = f"basename={basename} language={language}" if dry_run: result.stages["fingerprint"] = StageStatus( @@ -534,4 +545,4 @@ def run_metadata_sync( result.stages["fingerprint"] = StageStatus(status="failed", reason=reason) _stage_log_exit("fingerprint", "failed", reason) - return result \ No newline at end of file + return result diff --git a/pdd/operation_log.py b/pdd/operation_log.py index 2bcf9f8415..b85daf91b9 100644 --- a/pdd/operation_log.py +++ b/pdd/operation_log.py @@ -391,6 +391,68 @@ def _prompts_root_for_fingerprint(prompt_file: Union[str, Path]) -> Path: return prompt_path.parent +def resolve_fingerprint_paths( + basename: str, + language: str, + prompt_file: Union[str, Path], + *, + paths: Optional[Mapping[str, Any]] = None, +) -> Dict[str, Any]: + """Resolve a complete unit path set without losing explicit caller paths. + + Mutating commands often know only the prompt and the artifact they wrote. + Persisting that partial set would replace previously valid example/test + hashes with ``null`` and make the next sync report those untouched files as + changed. Resolve the remaining paths from the prompt's own project, then + overlay every explicit path so caller-selected locations stay authoritative + (issues #983, #1211/#1290, and #1305). + + Resolution is best-effort because derivative/new prompt outputs may not yet + be registered in project configuration. The explicit touched paths remain + available for hard hash validation even when optional sibling discovery + cannot complete. + """ + explicit: Dict[str, Any] = dict(paths or {}) + explicit_prompt = Path(prompt_file) + explicit["prompt"] = explicit_prompt + try: + from .sync_determine_operation import get_pdd_file_paths + + discovered: Dict[str, Any] = get_pdd_file_paths( + basename, + language, + prompts_dir=str(_prompts_root_for_fingerprint(prompt_file)), + ) + except (ImportError, OSError, ValueError) as exc: + logger.warning( + "Could not resolve complete fingerprint paths for %s/%s: %s", + basename, + language, + exc, + ) + return explicit + + resolved_prompt = discovered.get("prompt") + if not isinstance(resolved_prompt, Path) or not resolved_prompt.exists(): + logger.warning( + "Resolved fingerprint prompt is missing for %s/%s: %s", + basename, + language, + resolved_prompt, + ) + return explicit + + # A CLI may supply a relative prompt spelling while path discovery returns + # the real absolute file. Keep the discovered prompt when that spelling + # does not exist from the current process CWD; otherwise an otherwise good + # resolution would be overwritten with a null-hash path (issue #1305). + overlay = dict(explicit) + if not explicit_prompt.exists(): + overlay.pop("prompt", None) + discovered.update(overlay) + return discovered + + def load_operation_log( basename: str, language: str, @@ -578,56 +640,22 @@ def save_fingerprint( cost: float = 0.0, model: str = "unknown" ) -> None: - """ - Save the current fingerprint/state to the state file. + """Finalize the current fingerprint through the shared transaction. - Writes the full Fingerprint dataclass format compatible with read_fingerprint() - in sync_determine_operation.py. This ensures manual commands (generate, example) - don't break sync's fingerprint tracking. + Fingerprint persistence is part of command success. Any path, hash, or + atomic-write failure therefore propagates as ``FingerprintFinalizeError``. """ - from dataclasses import asdict - from datetime import timezone - from .sync_determine_operation import calculate_current_hashes, Fingerprint, read_fingerprint - from . import __version__ + from .fingerprint_transaction import FingerprintTransaction - # Issue #983: when the caller provides `paths`, use them directly — do - # NOT call get_pdd_file_paths. Issue #1211: at the same time, use those - # caller-supplied paths to detect the subproject root so the meta dir - # anchors at the .pddrc rather than the run CWD. - if not paths: - from .sync_determine_operation import get_pdd_file_paths - try: - paths = get_pdd_file_paths(basename, language) - except (ImportError, OSError, ValueError) as e: - logger.warning("Could not resolve paths for %s/%s: %s", basename, language, e) - paths = {} - - path = get_fingerprint_path(basename, language, paths=paths) - - # Issue #522: Pass stored include deps for prompt hash calculation - prev_fp = read_fingerprint(basename, language, paths=paths) - stored_deps = prev_fp.include_deps if prev_fp else None - current_hashes = calculate_current_hashes(paths, stored_include_deps=stored_deps) if paths else {} - - # Create Fingerprint with same format as _save_fingerprint_atomic - fingerprint = Fingerprint( - pdd_version=__version__, - timestamp=datetime.now(timezone.utc).isoformat(), - command=operation, - prompt_hash=current_hashes.get('prompt_hash'), - code_hash=current_hashes.get('code_hash'), - example_hash=current_hashes.get('example_hash'), - test_hash=current_hashes.get('test_hash'), - test_files=current_hashes.get('test_files'), - include_deps=current_hashes.get('include_deps'), # Issue #522 - ) - - try: - with open(path, 'w', encoding='utf-8') as f: - json.dump(asdict(fingerprint), f, indent=2) - except Exception as e: - console = Console() - console.print(f"[yellow]Warning: Failed to save fingerprint to {path}: {e}[/yellow]") + with FingerprintTransaction( + basename=basename, + language=language, + operation=operation, + paths=paths, + cost=cost, + model=model, + ): + pass def save_run_report( @@ -782,7 +810,20 @@ def wrapper(*args: Any, **kwargs: Any) -> Any: pass elif basename and language: append_log_entry(basename, language, entry, paths=log_paths) - if success: + no_op_fix = ( + operation == "fix" + and cost == 0.0 + and str(model).strip().lower() + in {"", "none", "unknown", "n/a"} + ) + if success and no_op_fix: + logger.info( + "Skipping fix metadata finalization for %s/%s: " + "no LLM invocation", + basename, + language, + ) + elif success: fingerprint_allowed = True # Clear the stale run report only after the command # succeeds, so a failed run cannot erase existing @@ -795,6 +836,20 @@ def wrapper(*args: Any, **kwargs: Any) -> Any: fingerprint_allowed = _clear_run_report_before_fingerprint( basename, language, paths=log_paths ) + if updates_fingerprint and not fingerprint_allowed: + from .fingerprint_transaction import ( + FingerprintFinalizeError, + ) + + raise FingerprintFinalizeError( + operation, + get_fingerprint_path( + basename, + language, + paths=log_paths, + ), + "run report not cleared", + ) if updates_fingerprint and fingerprint_allowed: # Issue #1305 + #1211: save_fingerprint hashes only # Path values, so a bare {"prompt": } hint @@ -814,35 +869,11 @@ def wrapper(*args: Any, **kwargs: Any) -> Any: # resolution fails, so anchoring still works. The # #983 contract is preserved: the caller resolves the # paths, so save_fingerprint does not. - from .sync_determine_operation import get_pdd_file_paths - try: - prompts_root = _prompts_root_for_fingerprint( - prompt_file - ) - fingerprint_paths: Dict[str, Any] = get_pdd_file_paths( - basename, language, prompts_dir=str(prompts_root) - ) - # Existence-gate the resolution: if it silently - # resolved the prompt to a path that is not on - # disk (a mis-resolution that did not raise), the - # other paths are wrong too, so the fingerprint - # would be null and mis-anchored. Fall back to the - # real prompt path so prompt_hash is real and the - # write still anchors at the subproject. - resolved_prompt = fingerprint_paths.get("prompt") - if not ( - isinstance(resolved_prompt, Path) - and resolved_prompt.exists() - ): - fingerprint_paths = {"prompt": Path(prompt_file)} - except (ImportError, OSError, ValueError) as e: - logger.warning( - "Could not resolve paths for %s/%s: %s", - basename, - language, - e, - ) - fingerprint_paths = {"prompt": Path(prompt_file)} + fingerprint_paths = resolve_fingerprint_paths( + basename, + language, + prompt_file, + ) save_fingerprint( basename, language, diff --git a/pdd/pin_example_hack.py b/pdd/pin_example_hack.py index 15b4889cfd..f599086e89 100644 --- a/pdd/pin_example_hack.py +++ b/pdd/pin_example_hack.py @@ -14,7 +14,6 @@ from pathlib import Path from typing import Dict, Any, Optional, List, Callable from dataclasses import asdict, dataclass, field -import tempfile import sys import click @@ -52,6 +51,7 @@ from .get_run_command import get_run_command_for_file from .pytest_output import extract_failing_files_from_output, _find_project_root from . import DEFAULT_STRENGTH +from .json_atomic import atomic_write_json # --- Helper Functions --- @@ -67,6 +67,7 @@ class PendingStateUpdate: fingerprint: Optional[Dict[str, Any]] = None run_report_path: Optional[Path] = None fingerprint_path: Optional[Path] = None + fingerprint_operation: Optional[str] = None class AtomicStateUpdate: @@ -105,39 +106,39 @@ def set_run_report(self, report: Dict[str, Any], path: Path): self.pending.run_report = report self.pending.run_report_path = path - def set_fingerprint(self, fingerprint: Dict[str, Any], path: Path): + def set_fingerprint( + self, + fingerprint: Dict[str, Any], + path: Path, + *, + operation: Optional[str] = None, + ): """Buffer a fingerprint for atomic write.""" self.pending.fingerprint = fingerprint self.pending.fingerprint_path = path + self.pending.fingerprint_operation = operation def _atomic_write(self, data: Dict[str, Any], target_path: Path) -> None: """Write data to file atomically using temp file + rename pattern.""" - target_path.parent.mkdir(parents=True, exist_ok=True) - - # Write to temp file in same directory (required for atomic rename) - fd, temp_path = tempfile.mkstemp( - dir=target_path.parent, - prefix=f".{target_path.stem}_", - suffix=".tmp" - ) - self._temp_files.append(temp_path) - - try: - with os.fdopen(fd, 'w') as f: - json.dump(data, f, indent=2, default=str) - - # Atomic rename - guaranteed atomic on POSIX systems - os.replace(temp_path, target_path) - self._temp_files.remove(temp_path) # Successfully moved, stop tracking - except Exception: - # Leave temp file for rollback to clean up - raise + atomic_write_json(target_path, data) def _commit(self): """Commit all pending state updates atomically.""" # Write fingerprint first (checkpoint), then run_report if self.pending.fingerprint and self.pending.fingerprint_path: - self._atomic_write(self.pending.fingerprint, self.pending.fingerprint_path) + try: + self._atomic_write( + self.pending.fingerprint, + self.pending.fingerprint_path, + ) + except Exception as exc: + from .fingerprint_transaction import FingerprintFinalizeError + + raise FingerprintFinalizeError( + self.pending.fingerprint_operation or "unknown", + self.pending.fingerprint_path, + exc, + ) from exc if self.pending.run_report and self.pending.run_report_path: self._atomic_write(self.pending.run_report, self.pending.run_report_path) @@ -246,31 +247,18 @@ def _save_operation_fingerprint(basename: str, language: str, operation: str, model: The model used. atomic_state: Optional AtomicStateUpdate for atomic writes (Issue #159 fix). """ - from datetime import datetime, timezone - from .sync_determine_operation import calculate_current_hashes, Fingerprint - from . import __version__ - - current_hashes = calculate_current_hashes(paths) - fingerprint = Fingerprint( - pdd_version=__version__, - timestamp=datetime.now(timezone.utc).isoformat(), - command=operation, - prompt_hash=current_hashes.get('prompt_hash'), - code_hash=current_hashes.get('code_hash'), - example_hash=current_hashes.get('example_hash'), - test_hash=current_hashes.get('test_hash'), - test_files=current_hashes.get('test_files'), # Bug #156 - ) - - fingerprint_file = META_DIR / f"{_safe_basename(basename)}_{language.lower()}.json" - if atomic_state: - # Buffer for atomic write - atomic_state.set_fingerprint(asdict(fingerprint), fingerprint_file) - else: - # Legacy direct write - META_DIR.mkdir(parents=True, exist_ok=True) - with open(fingerprint_file, 'w') as f: - json.dump(asdict(fingerprint), f, indent=2, default=str) + from .fingerprint_transaction import FingerprintTransaction + + with FingerprintTransaction( + basename=basename, + language=language, + operation=operation, + paths=paths, + cost=cost, + model=model, + atomic_state=atomic_state, + ): + pass def _python_cov_target_for_code_file(code_file: Path) -> str: """Return a `pytest-cov` `--cov` target for a Python code file. @@ -1761,4 +1749,4 @@ def __init__(self, rc, out, err): PDD_DIR.mkdir(exist_ok=True) META_DIR.mkdir(exist_ok=True) result = sync_orchestration(basename="my_calculator", language="python", quiet=True) - print(json.dumps(result, indent=2)) \ No newline at end of file + print(json.dumps(result, indent=2)) diff --git a/pdd/prompts/auto_deps_main_python.prompt b/pdd/prompts/auto_deps_main_python.prompt index 9621b2468b..f57cbda3bd 100644 --- a/pdd/prompts/auto_deps_main_python.prompt +++ b/pdd/prompts/auto_deps_main_python.prompt @@ -84,22 +84,23 @@ Use a `.lock` suffix file (e.g., `{csv_path}.lock`) for the lock. - Error handling: • On `click.Abort` exception: re-raise to allow the orchestrator to stop the sync loop. + • On `FingerprintFinalizeError`: re-raise so a mutated prompt cannot return a false success. • On any other exception: print a red error message (unless `quiet`) and return an error tuple `("", 0.0, f"Error: {exc}")` instead of calling `sys.exit(1)`. This allows the orchestrator to handle failures gracefully. - Metadata finalization (on successful runs only — skip on the error path above): • After the modified prompt and any CSV output have been written successfully, finalize fingerprint and include-dependency state for the file that was actually written (`output_file_paths["output"]`) so downstream commands see a consistent view. This must run for *every* successful auto-deps mutation — both the default CLI flow (where the output is `__with_deps.prompt`) and the in-place flow (where the output equals the input `prompt_file`). Issue #989 requires this for every mutated prompt file; do not gate the finalizer on `output_path == prompt_file`. - • Use the existing helpers from `pdd.operation_log` (`infer_module_identity`, `clear_run_report`, `get_run_report_path`) and `FingerprintTransaction` from `pdd.fingerprint_transaction`. Behavior must be language-agnostic (hash-based). + • Use the shared `pdd.operation_log` helpers (`infer_module_identity`, `resolve_fingerprint_paths`, `_clear_run_report_before_fingerprint`, `get_fingerprint_path`, and `save_fingerprint`). The public save wrapper is the only persistence route and delegates to `FingerprintTransaction`; behavior remains language-agnostic and hash-based. • Derive identity from the *output* path, not the input prompt: - - Resolve `(basename, language)` via `infer_module_identity(Path(output_path))`. The output path is the file whose hash and dependency state the fingerprint records; tying the identity to that file keeps `.pdd/meta` self-consistent even when the canonical prompt is untouched (default mode) or when the file is the canonical prompt itself (in-place mode). `infer_module_identity` returns `(None, None)` for unrecognized prompt names — check `basename is None or language is None` explicitly and skip finalization (logged as a yellow warning) rather than guessing. - - Clear any stale per-module run report via `clear_run_report(basename, language, paths={"prompt": Path(output_path)})` because the prompt content changed. If `clear_run_report` raises, log a warning and continue — it must not abort the subsequent fingerprint transaction. - - After invoking `clear_run_report`, verify the report path is actually gone via `get_run_report_path(basename, language, paths={"prompt": Path(output_path)})`; if it still exists, log a yellow warning (unless `quiet`). This is defensive because the helper silently swallows unlink failures. - - Write/update the fingerprint via `FingerprintTransaction(basename, language, operation="auto-deps", paths={"prompt": Path(output_path)}, cost=total_cost, model=model_name)` used as a context manager. When a pre-captured include graph is available, call `transaction.set_include_deps_override(include_deps)` before exiting the context. `FingerprintFinalizeError` propagates — a finalization failure exits non-zero. When `_skip_finalization=True`, call `transaction.skip(reason="_skip_finalization set by sync")` instead of omitting the transaction entirely. + - Resolve `(basename, language)` via `infer_module_identity(Path(output_path))`. The output path is the file whose hash and dependency state the fingerprint records. If either value is missing, the destination is an unmanaged/redirected output rather than a PDD unit: emit a yellow skip message unless quiet and return the successful artifact tuple without module metadata. + - Build the complete path set with `resolve_fingerprint_paths(basename, language, output_path, paths={"prompt": Path(output_path)})`. This preserves the written prompt while retaining code/example/test hashes for registered units; unregistered derivative outputs retain the explicit prompt path. + - Call `_clear_run_report_before_fingerprint(..., paths=resolved_paths)`. If it returns false, raise `FingerprintFinalizeError` using `get_fingerprint_path(..., paths=resolved_paths)` and cause `"run report not cleared"`. + - Commit with `save_fingerprint(..., operation="auto-deps", paths=resolved_paths, cost=total_cost, model=model_name)`. Re-raise `FingerprintFinalizeError` before the broad error-tuple handler so finalization is a non-zero command failure. • Architecture file updates (architecture.json) are merged by `merge_auto_deps_includes_from_cwd` and are not finalized here — that file is not a `(basename, language)`-keyed module and has its own atomicity guarantees inside the merge helper. - • Surface non-exception failures from the architecture merge (return shape `{"updated": bool, "messages": list[str], …}`): if the helper returned `updated=False`, emit each `messages` entry as a yellow `Warning: architecture.json not updated: …` line so the success message does not mask a silent no-op. + • Surface non-exception failures from the architecture merge (return shape `{"updated": bool, "messages": list[str], "added_dependencies": list[str], …}`): when `updated=False` AND `added_dependencies` is non-empty, emit each message as a yellow `Warning: architecture.json not updated: …` line. `updated=False` with no requested additions is a legitimate no-op. • Surface invalid `` tags stripped by `sanitize_prompt_output` (it returns `(cleaned_prompt, invalid_includes: list[str])`): if `invalid_includes` is non-empty, emit a yellow warning naming the count, the output path, and the stripped entries so the user knows generated dependencies were dropped. - • NOTE: `pdd sync` invokes `auto_deps_main` with a temp `__with_deps.prompt` output and then `shutil.move`s it onto the canonical prompt. To avoid leaving an orphan `.pdd/meta/*_with_deps.json` for that temp identity after the move (and to avoid clearing the wrong run report), sync passes the internal flag `_skip_finalization=True`; the `FingerprintTransaction` for that identity calls `transaction.skip(reason="_skip_finalization set by sync")` and sync owns canonical metadata itself via `FingerprintTransaction` (routed through `_save_fingerprint_atomic`) and clears the canonical `_run.json` via `clear_run_report` immediately after the move. When `_skip_finalization=True`, this finalizer must enter the `FingerprintTransaction` context but call `transaction.skip(...)` before returning. + • NOTE: `pdd sync` invokes `auto_deps_main` with a temp `__with_deps.prompt` output and then moves it onto the canonical prompt. To avoid orphan derivative metadata, sync passes `_skip_finalization=True`; after the artifact/console work, return the successful tuple before identity lookup or metadata writes. Sync owns canonical fingerprint finalization through `_save_fingerprint_atomic` and clears the canonical run report after the move. -pdd/operation_log.py +pdd/operation_log.py diff --git a/pdd/prompts/ci_drift_heal_python.prompt b/pdd/prompts/ci_drift_heal_python.prompt index 12232efe04..d2d6bacec1 100644 --- a/pdd/prompts/ci_drift_heal_python.prompt +++ b/pdd/prompts/ci_drift_heal_python.prompt @@ -26,7 +26,6 @@ agentic_langtest_python.prompt architecture_sync_python.prompt metadata_sync_python.prompt -fingerprint_transaction_python.prompt % You are an expert Python engineer. Your goal is to write a standalone CI script that detects prompt/example drift across all PDD modules and auto-heals drifted modules. @@ -53,7 +52,7 @@ A standalone CI script (`pdd/ci_drift_heal.py`) that orchestrates drift detectio 5. **Scope control:** `--modules` limits detection to specific basenames. Without it, scan all discovered modules. Accept both space-separated and comma-separated module lists. 6. **Three-way heal dispatch:** Dispatch by `drift.operation`: - - `update`: run `pdd --force --strength 0.5 update ` then invoke the shared `run_metadata_sync` orchestrator (from `pdd.metadata_sync`) for the (prompt, code) pair before the follow-up `pdd --force --strength 0.5 example ` step. The orchestrator finalizes prompt tags, architecture entries, run-report cleanup, and fingerprint state in fixed order; after a successful orchestrator result, refresh the operation-log fingerprint using the authoritative resolved module paths (not only the prompt/code pair handed to the orchestrator) so existing example/test hashes are preserved in the final fingerprint. **Fingerprint refresh paths (issue #1211)**: after a successful `run_metadata_sync` result, build `paths = {"prompt": p, "code": code_p}` directly from the known subproject paths — do NOT call `get_pdd_file_paths(basename, language)`, which resolves from CWD and returns parent-repo paths in parent-CWD mode. Use `FingerprintTransaction(basename, language, operation="update", paths=paths, cost=0.0, model="")` for this post-orchestrator refresh; `FingerprintFinalizeError` is treated as a hard heal failure consistent with "Metadata finalization is mandatory and never silent." Pass the same `paths` dict to every `read_fingerprint` call so reads and writes all anchor at the same subproject `.pdd/meta` directory. If `code_p` is None when `result.ok` is True, raise `ValueError("authoritative prompt/code paths unavailable: code_path not provided")`. Example/test-file hash validation (checking `example_hash`/`test_files` in the fingerprint against paths keys `"example"`/`"test_files"`) is skipped since those keys are absent from this minimal dict; that gap is tracked at issue #870. If code_path is unresolved, fail closed (no repo-wide fallback). After update, run churn gate and structural invariants gate before the orchestrator call. **Metadata finalization is mandatory and never silent**: `_run_metadata_sync_safe` returning False MUST be treated as a hard heal failure in every invocation mode (PR auto-heal, push-to-main auto-heal, and preflight drift-heal). The module's `heal_module` call MUST return False, the module MUST be recorded as failed (not merely skipped), and `main` MUST return a non-zero exit code with an explicit log line `metadata finalization failed for : ` so the workflow step fails loudly rather than committing a half-synced state. This finalization-failure escalation overrides the push-to-main "advisory" clause in Requirement 15. **Module-scoped snapshot/restore (issue #1211)**: implement `_subproject_root_for(drift)` — walk up from `drift.prompt_path` then `drift.code_path` looking for `.pddrc`, fall back to `_repo_root()`. Use it inside `_snapshot_metadata_state_for(drift)` (replacing `_repo_root()`) so `.pdd/meta` paths resolve under the subproject root. Change `_restore_metadata_state_for(snapshot, root: Path)` to accept an explicit `root` parameter; at both call sites pass `_subproject_root_for(drift)`. Before calling `_run_metadata_sync_safe`, capture a per-module snapshot of `architecture.json`, the operation-log fingerprint path `.pdd/meta/_.json`, and the operation-log run-report path `.pdd/meta/__run.json` via `_snapshot_metadata_state_for(drift)`. The safe basename must match `pdd.operation_log` path semantics, so subdirectory basenames such as `commands/foo` snapshot `.pdd/meta/commands_foo_python.json` and `.pdd/meta/commands_foo_python_run.json`, not `.pdd/meta/commands/foo_python.json`. On any orchestrator-stage failure OR on follow-up `pdd example` failure, call `_revert_prompt_file(drift)` AND `_restore_metadata_state_for(snapshot, _subproject_root_for(drift))` to revert this module's writes only. Do NOT use a repo-scoped `git restore .pdd` / `git restore architecture.json` here — that would wipe earlier successful modules' state from the same run on a multi-module push-to-main heal. The legacy `_cleanup_metadata_artifacts` name is preserved as a no-op shim solely for backward import compatibility; new code uses the snapshot/restore primitives. **Preflight drift-heal uses the same orchestrator call** so CLI auto-heal and preflight share one metadata-finalization path, and the same finalization-failure escalation applies there. + - `update`: run `pdd --force --strength 0.5 update ` then invoke the shared `run_metadata_sync` orchestrator (from `pdd.metadata_sync`) for the (prompt, code) pair before the follow-up `pdd --force --strength 0.5 example ` step. The orchestrator finalizes prompt tags, architecture entries, run-report cleanup, and fingerprint state in fixed order; after a successful orchestrator result, refresh the operation-log fingerprint once more using the shared `resolve_fingerprint_paths` + `save_fingerprint` boundary. Seed resolution with the known subproject `{"prompt": p, "code": code_p}` paths so explicit locations stay authoritative, while the prompt-anchored resolver adds existing example/test paths rather than wiping their hashes. Use the same complete mapping for the preceding `read_fingerprint`, preserve a completed user-facing command (`verify`/`test`/`fix`/`update`, falling back to `fix`), and verify the resulting fingerprint has non-null prompt/code hashes. If `code_p` is missing, identity cannot be inferred, path resolution/persistence fails, or the written fingerprint is incomplete, return False with an explicit finalization diagnostic. After update, run churn gate and structural invariants gate before the orchestrator call. **Metadata finalization is mandatory and never silent**: `_run_metadata_sync_safe` returning False MUST be treated as a hard heal failure in every invocation mode (PR auto-heal, push-to-main auto-heal, and preflight drift-heal). The module's `heal_module` call MUST return False, the module MUST be recorded as failed (not merely skipped), and `main` MUST return a non-zero exit code with an explicit log line `metadata finalization failed for : ` so the workflow step fails loudly rather than committing a half-synced state. This finalization-failure escalation overrides the push-to-main "advisory" clause in Requirement 15. **Module-scoped snapshot/restore (issue #1211)**: implement `_subproject_root_for(drift)` — walk up from `drift.prompt_path` then `drift.code_path` looking for `.pddrc`, fall back to `_repo_root()`. Use it inside `_snapshot_metadata_state_for(drift)` (replacing `_repo_root()`) so `.pdd/meta` paths resolve under the subproject root. Change `_restore_metadata_state_for(snapshot, root: Path)` to accept an explicit `root` parameter; at both call sites pass `_subproject_root_for(drift)`. Before calling `_run_metadata_sync_safe`, capture a per-module snapshot of `architecture.json`, the operation-log fingerprint path `.pdd/meta/_.json`, and the operation-log run-report path `.pdd/meta/__run.json` via `_snapshot_metadata_state_for(drift)`. The safe basename must match `pdd.operation_log` path semantics, so subdirectory basenames such as `commands/foo` snapshot `.pdd/meta/commands_foo_python.json` and `.pdd/meta/commands_foo_python_run.json`, not `.pdd/meta/commands/foo_python.json`. On any orchestrator-stage failure OR on follow-up `pdd example` failure, call `_revert_prompt_file(drift)` AND `_restore_metadata_state_for(snapshot, _subproject_root_for(drift))` to revert this module's writes only. Do NOT use a repo-scoped `git restore .pdd` / `git restore architecture.json` here — that would wipe earlier successful modules' state from the same run on a multi-module push-to-main heal. The legacy `_cleanup_metadata_artifacts` name is preserved as a no-op shim solely for backward import compatibility; new code uses the snapshot/restore primitives. **Preflight drift-heal uses the same orchestrator call** so CLI auto-heal and preflight share one metadata-finalization path, and the same finalization-failure escalation applies there. - `example`: run `pdd --force --strength 0.5 example ` directly. Require both paths resolved; fail closed otherwise. In CI, when `PDD_HEAL_SKIP_REVIEW_ONLY_EXAMPLE_DRIFT=1`, the drift reason is git-reclassified as already-covered PR review work (`Code and prompt changed together...` or `Prompt changed without code changes...`), and the resolved example file already exists, skip the example rewrite and leave that reviewed artifact to the PR author; missing examples still run the heal. When `PDD_HEAL_SKIP_EXISTING_EXAMPLE_DRIFT=1` and any resolved example file already exists, skip the example rewrite as a broader fallback. - `auto-deps`: run `pdd --force --strength 0.5 auto-deps --output --csv project_dependencies.csv` directly. Do not route this through full `pdd sync`, because sync can continue into generate/crash flows after dependency insertion. - `verify`/`generate`/`test`/`crash`: run `pdd --force --strength 0.5 sync ` to let sync_determine_operation dispatch correctly. In PR auto-heal mode (no `--skip-ci`), the subprocess env MUST include `PDD_DISABLE_TEST_EXTEND=1` so this nested sync cannot escalate a narrow heal into coverage-driven `test_extend`. Do NOT use `pdd example` for these — it would overwrite user edits (verify), skip code regeneration (generate), or save a stale fingerprint. The only exceptions are the git-based clean-CI reclassifications above, where a reported `auto-deps`/`generate` operation is converted to `example` or skipped based on actual code/prompt changes in the PR. diff --git a/pdd/prompts/commands/fix_python.prompt b/pdd/prompts/commands/fix_python.prompt index e33b7d755e..49277f698e 100644 --- a/pdd/prompts/commands/fix_python.prompt +++ b/pdd/prompts/commands/fix_python.prompt @@ -44,7 +44,7 @@ - Manual: `--output-test`, `--output-code`, `--output-results`, `--loop`, `--verification-program`, `--max-attempts` (3), `--budget` (5.0), `--auto-submit`, `--agentic-fallback/--no-agentic-fallback` (T), `--compress-test-context`, `--context-compression` (choice), `--compression-fallback` (choice). - Both: `--protect-tests/--no-protect-tests` (F). - **Technical Details**: - - Decorate with `@log_operation(operation="fix", clears_run_report=True)` and `@track_cost`. + - Decorate with `@log_operation(operation="fix", clears_run_report=True, updates_fingerprint=True)` and `@track_cost` so a successful manual fix cannot return without refreshing the shared fingerprint transaction. Agentic issue mode has no local prompt identity and therefore leaves finalization to its workflow. - Use deferred imports inside the command for mode-specific functions. - Return `Optional[Tuple[result_dict, cost, model]]`. - Re-raise Click exceptions; use `handle_error(e, "fix", quiet)` for others. diff --git a/pdd/prompts/fingerprint_transaction_python.prompt b/pdd/prompts/fingerprint_transaction_python.prompt index f6012a13dd..efba6d6d98 100644 --- a/pdd/prompts/fingerprint_transaction_python.prompt +++ b/pdd/prompts/fingerprint_transaction_python.prompt @@ -1,13 +1,13 @@ context/python_preamble.prompt -Single enforced code path for all PDD fingerprint writes — atomic temp-file + os.replace with commit-or-fail semantics. +Single enforced code path for complete, atomic PDD fingerprint finalization with commit-or-fail semantics. { "type": "module", "module": { "functions": [ - {"name": "FingerprintTransaction.__init__", "signature": "(basename: str, language: str, operation: str, paths: Optional[Dict[str, Path]] = None, cost: float = 0.0, model: str = '') -> None", "returns": "None"}, + {"name": "FingerprintTransaction.__init__", "signature": "(basename: str, language: str, operation: str, paths: Optional[Dict[str, Path]] = None, cost: float = 0.0, model: str = '', *, atomic_state: Optional[Any] = None) -> None", "returns": "None"}, {"name": "FingerprintTransaction.__enter__", "signature": "() -> FingerprintTransaction", "returns": "FingerprintTransaction"}, {"name": "FingerprintTransaction.__exit__", "signature": "(exc_type, exc_val, exc_tb) -> bool", "returns": "bool"}, {"name": "FingerprintTransaction.skip", "signature": "(reason: str) -> None", "returns": "None"}, @@ -20,70 +20,63 @@ sync_determine_operation_python.prompt % Goal -Write `pdd/fingerprint_transaction.py` — the single authoritative context manager for all PDD fingerprint writes. Replaces scattered `save_fingerprint` call sites with one enforced, atomic code path. +Write `pdd/fingerprint_transaction.py`, the authoritative builder and commit boundary for every PDD fingerprint write. All mutating workflows delegate here, and none may report success after finalization fails. % Role & Scope -This module provides the `FingerprintTransaction` class used by every PDD mutating command (sync, generate, example, update, fix, auto-deps, ci-heal). It owns: eager identity and path resolution at entry, atomic file write (temp-file + `os.replace`) at exit, null-hash validation, commit-or-fail exit semantics, and dry-run/redirect skip control. It is a leaf import — it does NOT import from the seven caller modules, eliminating circular dependency risk. - - -- Does not invoke LLM calls. -- Does not read or write run reports (that remains in operation_log). -- Does not push commits or run git commands. -- Does not own the `@log_operation` decorator (that stays in operation_log). - +This module resolves the fingerprint destination once, computes the canonical payload, rejects incomplete hashes, and either writes the JSON with the shared crash-safe atomic writer or buffers the identical payload in sync's outer `AtomicStateUpdate`. It does not invoke LLMs, mutate artifacts, clear run reports, or decide workflow operations. -- Commit: the `__exit__` path that writes the fingerprint file atomically and raises `FingerprintFinalizeError` on failure. -- Skip: suppressing the commit so `__exit__` is a no-op; used for dry-run, redirected output, or run-report-not-cleared conditions. -- Null hash: a `prompt_hash`, `code_hash`, `example_hash`, or `test_hash` value that is `None` — an indication that `calculate_current_hashes` could not resolve the file, producing a fingerprint that can never converge with on-disk state. +- Immediate commit: write the validated payload with `json_atomic.atomic_write_json`, whose same-directory temp file, flush, `fsync`, and `os.replace` prevent a partial JSON destination. +- Buffered commit: pass the validated payload and eagerly resolved destination to an outer atomic-state object's `set_fingerprint`; the outer context performs the write before its command can return success. +- Intentional skip: `skip(reason)` suppresses finalization for a known non-mutating path such as dry-run or redirected output. +- Complete existing input: prompt is always required; code, example, primary test, and each `test_files` entry require a non-null hash whenever the supplied path exists. -R1 (MUST): When `__exit__` is called with `exc_type is None` (clean body) and `skip()` has NOT been called, write the fingerprint atomically (temp-file in `fingerprint_path.parent` + `os.fsync` + `os.replace`); raise `FingerprintFinalizeError` on ANY failure during this write. -R2 (MUST): When `__exit__` is called with `exc_type is not None` (exception in body), propagate the exception without touching the fingerprint file. Return `False` (do not suppress the exception). -R3 (MUST): Before writing, validate that `prompt_hash` is not `None`. If it is `None`, raise `FingerprintFinalizeError` naming the null-hash field and the fingerprint path (prevents the #1305 null-hash loop class). -R4 (MUST): Resolve `fingerprint_path` eagerly in `__init__` via `get_fingerprint_path(basename, language, paths=paths)`. Never re-resolve it later (prevents the #1211/#1290 wrong-root class). -R5 (MUST): Use a temp file in the **same directory** as the fingerprint (`dir=fingerprint_path.parent`) so that `os.replace` is a single-filesystem rename and is POSIX-atomic. Never write to a cross-device temp directory. -R6 (MUST): `skip(reason)` suppresses the commit; log the reason at INFO level via stdlib `logging`. After `skip()` is called, `__exit__` on a clean body MUST be a no-op (no file write, no error). -R7 (MUST NOT): Raise `FingerprintFinalizeError` when `skip()` has been called, even if `prompt_hash` would otherwise be `None`. -R8 (MUST): `FingerprintFinalizeError` message MUST include the fingerprint path, the operation name, and the underlying cause (if any). +R1 (MUST): On a clean body exit without `skip()`, build and validate exactly one fingerprint payload, then commit or buffer it. Any failure raises `FingerprintFinalizeError` and propagates. +R2 (MUST): On a body exception, return `False` without building or writing a fingerprint so the original exception propagates. +R3 (MUST): Reject a null `prompt_hash`. Also reject a null code/example/test hash for every corresponding supplied path that exists, and reject a missing entry for every existing supplied `test_files` path. +R4 (MUST): Resolve `fingerprint_path` eagerly in `__init__` from the same normalized path mapping later used for reading and hashing. Explicit paths are authoritative; call `get_pdd_file_paths` only when no non-empty `paths` mapping was supplied (issue #983). +R5 (MUST): Immediate commits use `atomic_write_json`; do not duplicate temp-file JSON persistence in this module or any caller. +R6 (MUST): If `atomic_state` is supplied, require a callable `set_fingerprint(payload, path, operation=...)` and buffer the same payload an immediate commit would write. Do not write the destination early. +R7 (MUST): `skip(reason)` is idempotent, logs the reason at INFO, and makes clean exit a no-op. A skipped transaction never validates hashes or raises a finalization error. +R8 (MUST): `FingerprintFinalizeError` includes the operation, resolved fingerprint path, and underlying cause. Path-discovery errors use the deterministic fallback path `.pdd/meta/_.json` in diagnostics. +R9 (MUST): An include-dependency override controls both hash calculation (`stored_include_deps`) and the `include_deps` value persisted. This preserves the pre-mutation include graph used by auto-deps/sync. % Requirements -1. **`FingerprintFinalizeError(RuntimeError)`** — raised when atomic fingerprint write fails or null-hash is detected. Message format: `"[{operation}] fingerprint finalization failed for {fingerprint_path}: {cause}"`. No additional base class is required (no PDD-specific exception hierarchy exists). - -2. **`FingerprintTransaction.__init__`**: Eagerly resolve everything needed for the commit: - - Accept `basename`, `language`, `operation`, `paths` (optional `Dict[str, Path]`), `cost`, and `model`. - - Call `get_fingerprint_path(basename, language, paths=paths)` and store as `self._fingerprint_path`. - - Store `basename`, `language`, `operation`, `cost`, `model`, and `paths` for use in `__exit__`. - - Initialize `self._skipped: bool = False`, `self._skip_reason: Optional[str] = None`, and `self._include_deps_override: Optional[Dict[str, str]] = None`. +1. Define `FingerprintFinalizeError(RuntimeError)` with public attributes `operation`, `fingerprint_path`, and `cause`. Format its message as `"[{operation}] fingerprint finalization failed for {fingerprint_path}: {cause}"`. -3. **`skip(reason: str)`**: Set `self._skipped = True` and `self._skip_reason = reason`. Log at `logging.INFO`: `"FingerprintTransaction.skip for %s/%s (%s): %s"`. Must be idempotent (calling twice is safe). +2. In `FingerprintTransaction.__init__`: + - Lowercase `language`; store operation, numeric cost, model, optional duck-typed `atomic_state`, skip state, and include override state. + - When `paths` is non-empty, normalize its values without changing scope: coerce ordinary values to `Path`, preserve `None`, and normalize `test_files` to a list of `Path` values. Never replace explicit paths with CWD discovery. + - Otherwise resolve paths via `get_pdd_file_paths(basename, language)` and normalize the result. + - Resolve and store `get_fingerprint_path(basename, language, paths=normalized_paths)` immediately. Wrap discovery/destination errors as `FingerprintFinalizeError` with `path resolution failed: ...` as the cause. + - Expose the resolved destination through a read-only `fingerprint_path` property. -4. **`set_include_deps_override(deps: Dict[str, str])`**: Store `deps` for use in `__exit__` when building the `Fingerprint` dataclass. Called by `auto_deps_main` after the prompt file is written but before `__exit__`. +3. `__enter__` returns `self`. `skip(reason)` records/logs an intentional skip. `set_include_deps_override(deps)` copies the mapping so callers cannot mutate transaction state afterward. -5. **`__enter__`**: Return `self`. No side effects. +4. Payload construction on clean exit: + - Read the previous fingerprint with the normalized paths. + - Select `stored_deps`: the explicit override when set, otherwise the previous fingerprint's `include_deps`, otherwise `None`. + - Call `calculate_current_hashes(paths, stored_include_deps=stored_deps)` exactly once. + - Apply R3 validation before constructing or writing anything. + - Build the existing `Fingerprint` dataclass with `pdd_version=__version__`, a UTC ISO timestamp, `command=operation`, all current artifact hashes/test-files hashes, and include deps from the override when present or the calculated result otherwise. Serialize with `dataclasses.asdict`. -6. **`__exit__(exc_type, exc_val, exc_tb)`**: - - If `exc_type is not None`: return `False` (propagate the exception, do not write). - - If `self._skipped`: log at DEBUG that the commit was suppressed; return `False`. - - Otherwise (clean body, not skipped): execute the atomic commit: - a. Read the previous fingerprint via `read_fingerprint(self._basename, self._language, paths=self._paths)` to obtain stored `include_deps` (use `None` when no previous fingerprint exists). - b. Call `calculate_current_hashes(self._paths, stored_include_deps=stored_deps)` to get current file hashes, where `stored_deps` is the `include_deps` from step (a) (or `None` when no previous fingerprint exists). - c. If `current_hashes.get('prompt_hash') is None`: raise `FingerprintFinalizeError` (R3). - d. Build the `Fingerprint` dataclass: `command=self._operation`, `pdd_version=__version__` (imported from `pdd`), `timestamp=datetime.now(timezone.utc).isoformat()`, `prompt_hash/code_hash/example_hash/test_hash/test_files` from `current_hashes`, and `include_deps` from `current_hashes.get('include_deps')` — the freshly extracted deps from step (b) — or `self._include_deps_override` when set, which takes priority over the computed current include_deps. The corrected logic is: `include_deps = self._include_deps_override if self._include_deps_override is not None else current_hashes.get('include_deps')`. - e. Create a temp file: `fd, tmp_path = tempfile.mkstemp(dir=self._fingerprint_path.parent, suffix=".tmp")`. - f. Write JSON via `json.dump(asdict(fingerprint), f, indent=2)` then `os.fsync(fd)` then close. - g. Call `os.replace(tmp_path, self._fingerprint_path)` for atomic rename. - h. On ANY exception during steps (a)–(g): attempt `os.unlink(tmp_path)` (suppress `OSError`); then raise `FingerprintFinalizeError` with the original exception as cause. - - Return `False` (never suppress exceptions from the body). +5. Commit the payload: + - With `atomic_state`, look up `set_fingerprint`; a missing/non-callable setter is a typed finalization failure. Call it with `(payload, fingerprint_path, operation=operation)`. + - Without `atomic_state`, call `atomic_write_json(fingerprint_path, payload)`. + - Re-raise an existing `FingerprintFinalizeError` unchanged. Wrap every other build, buffer, or atomic-write exception as `FingerprintFinalizeError` using exception chaining. -7. **Imports**: Use only stdlib and existing PDD primitives — `get_fingerprint_path` from `pdd.operation_log`; `read_fingerprint`, `calculate_current_hashes`, `Fingerprint` from `pdd.sync_determine_operation`; `__version__` from `pdd`; `datetime`, `timezone` from `datetime`. Use `dataclasses.asdict` for JSON serialization. Use `tempfile.mkstemp`, `os.replace`, `os.fsync`, `os.unlink`, `json`, `logging`. +6. `__exit__` always returns `False`. It first propagates body exceptions, then honors skip, and only then commits. Log skipped exits at DEBUG. -8. **Windows note**: Document in a module-level comment that `os.replace()` on Windows is not POSIX-atomic when the destination exists; this is the same behavior as the existing `_save_fingerprint_atomic` implementation and is a known limitation. +7. Document that `os.replace` is atomic on POSIX for same-filesystem paths and has weaker replacement guarantees on Windows. The shared writer creates its temp file in the destination directory. % Dependencies + + pdd/json_atomic.py + pdd/operation_log.py @@ -91,9 +84,9 @@ R8 (MUST): `FingerprintFinalizeError` message MUST include the fingerprint path, pdd/sync_determine_operation.py - - context/operation_log_example.py - + + context/fingerprint_transaction_example.py + % Deliverables - Code: `pdd/fingerprint_transaction.py` diff --git a/pdd/prompts/metadata_sync_python.prompt b/pdd/prompts/metadata_sync_python.prompt index 5ead5c943e..be0f478c6c 100644 --- a/pdd/prompts/metadata_sync_python.prompt +++ b/pdd/prompts/metadata_sync_python.prompt @@ -20,7 +20,6 @@ architecture_sync_python.prompt sync_determine_operation_python.prompt operation_log_python.prompt -fingerprint_transaction_python.prompt % Goal Write `pdd/metadata_sync.py` — a single-function orchestrator that runs prompt metadata finalization stages in a fixed order and returns a structured result. @@ -35,7 +34,7 @@ This module provides the shared `run_metadata_sync` entry point used by `pdd upd 2. `tags`: **preserve existing PDD metadata tags, or seed tags from the architecture entry when the prompt has none.** Uses `pdd.architecture_sync.generate_tags_from_architecture` (multi-language compatible, no language-specific AST dependency) to render the seed block. Writes the seeded prompt back to disk unless `dry_run=True`. When the prompt has zero PDD tags AND no architecture entry exists to seed from, report `skipped` with a reason naming #870 — never `ok` (claiming `ok with 0 tag(s)` would mislead operators). **LLM-first tag refresh (rewriting existing-but-stale tags) is out of scope for this orchestrator and tracked at #870**; downstream callers that need refresh should invoke that primitive directly. 3. `architecture`: call `update_architecture_from_prompt(prompt_filename, prompts_dir, architecture_path, dry_run, prompt_content_override=)` so architecture.json picks up the freshly-generated tags without a disk race. `prompt_filename` MUST be the prompts-dir-relative path (e.g. `commands/foo_python.prompt`), not just the basename — `update_architecture_from_prompt` matches architecture entries by their `filename` field which is path-aware per Issue #617. Fall back to the basename only when the prompt sits outside any `prompts/` ancestor. When the architecture lookup returns "No architecture entry found for: …", treat that as `skipped` (unregistered module is a normal state for tools/scripts), NOT as `failed`, so the fingerprint stage is not gated off. **Gated symmetrically with `run_report` and `fingerprint`**: if `prompt` or `tags` reported `failed`, mark `architecture` as `skipped` with the failing stage as the reason and do NOT invoke `update_architecture_from_prompt` — that call writes architecture.json on success, and running it after an earlier hard failure means the architecture entry can be modified on disk before the overall result reports `failed`, which on the push-to-main auto-heal path can land via the heal-commit's scoped `git add -u` even though downstream stages correctly stop. 4. `run_report`: clear or refresh stale run reports tied to this prompt/code pair (only the entries this sync invalidates — do not wipe unrelated entries). Build a `paths` hint from the known file paths — `{"prompt": prompt_path}` plus `"code": code_path` when `code_path` is set — and pass it as `paths=` to `clear_run_report` so that `.pdd/meta` is anchored at the subproject root rather than the run CWD (supports parent-directory workflows, issue #1211). **Gated symmetrically with `fingerprint`**: if `prompt`, `tags`, or `architecture` reported `failed`, mark `run_report` as `skipped` with the failing stage as the reason and do NOT invoke `clear_run_report` — the deletion is an on-disk side effect that, left to run after an earlier hard failure, would create partial `.pdd/meta` state that auto-heal's scoped staging (`git add -u` plus per-module fingerprint pathspecs) could still pick up on the push-to-main path. - 5. `fingerprint`: finalize fingerprint state LAST via `FingerprintTransaction` directly (not the `save_fingerprint` wrapper). Pass the same `paths` hint (built from `prompt_path`/`code_path`) so fingerprint reads and writes anchor at the subproject `.pdd/meta` directory (issue #1211). Gated on no prior stage reporting `failed` — `skipped` and `dry_run` are acceptable upstream statuses (matches `MetadataSyncResult.ok`). If any earlier stage failed, mark fingerprint as `skipped` with the failing stage as the reason. This prevents storing a fingerprint that records "in sync" when an earlier metadata layer is broken. `FingerprintFinalizeError` is caught by the per-stage handler (Req 4) and reported as `{"status": "failed", "reason": str(exc)}` — this is the correct behavior for `metadata_sync` as an orchestrator that surfaces failures via `MetadataSyncResult.ok` rather than propagating them. + 5. `fingerprint`: finalize fingerprint state LAST through `operation_log.save_fingerprint`, the thin shared-transaction wrapper. Start with the known prompt/code hint, then call `resolve_fingerprint_paths` so untouched existing example/test hashes are retained while explicit paths remain authoritative. Use this complete set for both `read_fingerprint` and the save, anchoring all metadata at the same subproject `.pdd/meta` (issue #1211). Preserve a previous user-facing command in `verify`/`test`/`fix`/`update`; use `fix` when no completed command exists so the internal metadata stage does not make the workflow appear incomplete. Gate on no prior stage reporting `failed`; `skipped` and `dry_run` remain acceptable upstream statuses. Any finalization exception is caught by the per-stage handler and reported as `failed`, which makes `MetadataSyncResult.ok` false. 3. **Result type** `MetadataSyncResult` — a Pydantic v2 model with: - `prompt_path: Path` - `code_path: Optional[Path]` @@ -70,11 +69,8 @@ This module provides the shared `run_metadata_sync` entry point used by `pdd upd pdd/sync_determine_operation.py - pdd/operation_log.py + pdd/operation_log.py - - pdd/fingerprint_transaction.py - % Deliverables diff --git a/pdd/prompts/operation_log_python.prompt b/pdd/prompts/operation_log_python.prompt index 6589412b98..2981ce9bb4 100644 --- a/pdd/prompts/operation_log_python.prompt +++ b/pdd/prompts/operation_log_python.prompt @@ -7,9 +7,10 @@ "type": "module", "module": { "functions": [ - {"name": "log_operation", "signature": "(updates_fingerprint: bool, updates_run_report: bool, clears_run_report: bool) -> Callable", "returns": "Callable"}, + {"name": "log_operation", "signature": "(operation: str, updates_fingerprint: bool = False, updates_run_report: bool = False, clears_run_report: bool = False) -> Callable", "returns": "Callable"}, {"name": "append_log_entry", "signature": "(basename: str, language: str, entry: Mapping[str, Any], project_root=None, paths=None) -> None", "returns": "None"}, - {"name": "log_event", "signature": "(basename: str, language: str, event: str, details: Mapping[str, Any] | None = None, project_root=None, paths=None, compression=None, agentic_fallback=None) -> None", "returns": "None"} + {"name": "log_event", "signature": "(basename: str, language: str, event: str, details: Mapping[str, Any] | None = None, project_root=None, paths=None, compression=None, agentic_fallback=None) -> None", "returns": "None"}, + {"name": "resolve_fingerprint_paths", "signature": "(basename: str, language: str, prompt_file: Union[str, Path], *, paths: Optional[Mapping[str, Any]] = None) -> Dict[str, Any]", "returns": "Dict[str, Any]"} ] } } @@ -36,10 +37,10 @@ A shared logging infrastructure module for tracking all PDD operations. Provides 3. **Module Identity Inference** - Extract basename and language from prompt file paths. Reconstructs directory context relative to the configured prompts root, so `prompts/frontend/settings/page_typescriptreact.prompt` yields basename `frontend/settings/page` and language `typescriptreact`. Paths without a `prompts/` ancestor fall back to filename-only. Language is always normalized to lowercase. **Issue #1211**: when the nearest `.pddrc` defines a non-default `prompts_dir` (e.g. `"prompts/backend"`), `infer_module_identity` MUST anchor the subdir extraction on that configured root — not the literal `"prompts"` segment. Otherwise the decorator would emit basenames like `backend/services/foo` while `construct_paths`/`sync` emit `services/foo`, splitting fingerprint filenames into two (`backend_services_foo_*.json` vs `services_foo_*.json`) and silently hiding metadata from sync. Use a helper `_prompts_dir_for_prompt(prompt_path)` that finds the nearest `.pddrc` via `construct_paths._find_nearest_pddrc_for_file`, resolves the context (file-based first, then cwd-based), and returns its `prompts_dir`. Fall back to the literal `prompts/` anchor when no `.pddrc` or no `prompts_dir` is set. -4. **State File Management** - Save fingerprints (`.json`), run reports (`_run.json`), and sync logs (`_sync.log`) to `.pdd/meta/`. All paths use sanitized basenames and `language.lower()`. Stale run reports are cleared only after the invalidating operation succeeds, and must be cleared before `save_fingerprint` so a fresh fingerprint never coexists with a stale per-module run report (issue #1057). The decorator must verify that a pre-existing run report is actually gone before saving a fingerprint; if deletion cannot be verified, call `transaction.skip(reason="run report not cleared")` rather than warning and skipping ad-hoc. `FingerprintFinalizeError` propagates to the Click boundary (non-zero exit). A failed command must never erase existing runtime verification state. - - `save_fingerprint` is a thin public wrapper that constructs a `FingerprintTransaction` and commits it. **`FingerprintFinalizeError` propagates to callers — fingerprint write failures are command failures, not warnings.** Callers needing skip control (dry-run, redirected output) construct `FingerprintTransaction` directly and call `transaction.skip(reason=...)`. +4. **State File Management** - Save fingerprints (`.json`), run reports (`_run.json`), and sync logs (`_sync.log`) to `.pdd/meta/`. All paths use sanitized basenames and `language.lower()`. Stale run reports are cleared only after the invalidating operation succeeds, and must be cleared before `save_fingerprint` so a fresh fingerprint never coexists with a stale per-module run report (issue #1057). The decorator must verify that a pre-existing run report is actually gone before saving a fingerprint; if deletion cannot be verified, raise `FingerprintFinalizeError` and fail the command. A failed command must never erase existing runtime verification state. + - `save_fingerprint` is a thin public wrapper that constructs a `FingerprintTransaction` and commits it. **`FingerprintFinalizeError` propagates to callers — fingerprint write failures are command failures, not warnings.** Intentional non-mutating modes skip before calling this wrapper; callers that need a scoped transaction may use `FingerprintTransaction.skip(reason=...)`. - `save_fingerprint` must write the full `Fingerprint` dataclass format (from `sync_determine_operation`) using `calculate_current_hashes` so that `read_fingerprint` can parse it. This ensures manual commands (generate, example) don't break sync's fingerprint tracking. Pass stored `include_deps` from previous fingerprint for prompt hash calculation. - - Metadata path resolution (issue #1211): `ensure_meta_dir`, `get_log_path`, `get_fingerprint_path`, and `get_run_report_path` all accept an optional `project_root` argument that anchors `.pdd/meta` under that root, plus an optional `paths` dict (the same `paths` flowing through `save_fingerprint`). Resolution order when `project_root` is omitted: (1) walk up from each file in `paths` until a `.pddrc` is found — this is what handles the case where the subproject .pddrc lives BELOW the run CWD; (2) walk up from CWD; (3) fall back to CWD. `save_fingerprint`, `save_run_report`, `clear_run_report`, `_clear_run_report_before_fingerprint`, and `append_log_entry` all accept `paths` and thread it through. The `@log_operation` decorator captures the inferred `prompt_file` path as an anchoring hint and passes a `paths` dict to every metadata write it makes (append_log_entry, save_fingerprint, save_run_report, _clear_run_report_before_fingerprint) so a decorated command run from a parent CWD still anchors its sync log, fingerprint, and run report at the subproject's `.pdd/meta`. **Issue #1305 / #1211**: for the `save_fingerprint` call specifically, the decorator MUST pass the authoritative, complete file-path set — `Path` objects keyed by `prompt`/`code`/`example`/`test` — and NOT a bare `{"prompt": prompt_file}` hint. `calculate_current_hashes` only hashes values that are `Path` instances, so a raw string prompt value plus absent code/example/test keys yields all-null `prompt_hash/code_hash/example_hash/test_hash`; that null fingerprint never matches the files on disk, so it never converges and CI auto-heal re-runs `pdd example`/`generate` on every pass, rewriting the generated example and overwriting maintainer cleanup. Resolve that set via `get_pdd_file_paths`, but anchor it at the prompt file's subproject by passing an absolute `prompts_dir` pointing at the prompts-root directory that contains this `prompt_file`: `get_pdd_file_paths(basename, language, prompts_dir=)`. This anchoring is REQUIRED because `get_pdd_file_paths` otherwise re-resolves the prompts root from the run CWD (`_find_pddrc_file()` walks up from CWD): a command invoked from a PARENT CWD for a nested subproject (the #1211 case — subproject `.pddrc` BELOW the run CWD) would then resolve to the parent, fail to find the prompt, point code/example/test at non-existent parent files (all-null hashes again), AND write the fingerprint to the parent `.pdd/meta` — splitting it from the sync log/run report that the prompt-path hint still anchors at the subproject. The absolute prompts root must be the BASE prompts directory (e.g. `/prompts`) — the `prompts` component nearest the prompt file, the same root `get_pdd_file_paths` uses by default — NOT a deeper, context-configured `prompts_dir` such as `prompts/commands`. `get_pdd_file_paths` searches for the prompt recursively under this root and computes `architecture.json` lookup keys *relative to it*, so a deeper root changes that key (`checkup_python.prompt` instead of `commands/checkup_python.prompt`) and silently breaks filepath resolution for subdir-context modules (e.g. `pdd/commands/checkup.py`). Compute it by taking the path up to and including the `prompts` component nearest the prompt file (resolved absolute), falling back to the prompt file's own directory when the path has no `prompts` component. Once the prompt resolves to its real subproject path, `construct_paths` detects the subproject `.pddrc` lives below the CWD and anchors code/example/test inside the subproject too, so all four hashes are real and the fingerprint lands in the subproject's `.pdd/meta`. Passing the full, subproject-anchored `Path` dict produces real, non-null hashes for both the `example` and `generate` operations and lets a second auto-heal pass on an unchanged tree detect no drift. Existence-gate the result: if resolution silently produced a `prompt` path that is not on disk (a mis-resolution that did not raise), the other paths are wrong too, so fall back to `{"prompt": Path(prompt_file)}` so `prompt_hash` is still real and the write anchors at the subproject. Likewise, if prompts-root computation or `get_pdd_file_paths` resolution raises, fall back to `{"prompt": Path(prompt_file)}` (a coerced `Path`, never a raw string) so anchoring still works. The lighter prompt-path hint may still be used for the non-fingerprint writes (append_log_entry/save_run_report/clear) purely for anchoring. Issue #983 contract preserved: when the caller passes a non-empty `paths`, `save_fingerprint` never calls `get_pdd_file_paths` — the decorator resolves the full path set itself before the call. Expose `_detect_project_root` and `_detect_project_root_from_paths` helpers for cross-module reuse by `sync_determine_operation.get_meta_dir` / `read_fingerprint` / `read_run_report`. + - Metadata path resolution (issue #1211): `ensure_meta_dir`, `get_log_path`, `get_fingerprint_path`, and `get_run_report_path` all accept an optional `project_root` argument that anchors `.pdd/meta` under that root, plus an optional `paths` dict. Resolution order when `project_root` is omitted: (1) walk up from each file in `paths` until a `.pddrc` is found; (2) walk up from CWD; (3) fall back to CWD. `save_fingerprint`, `save_run_report`, `clear_run_report`, `_clear_run_report_before_fingerprint`, and `append_log_entry` all thread the path hint through. `resolve_fingerprint_paths` is the shared caller-side resolver for mutating commands that know only some artifacts: anchor `get_pdd_file_paths` at the absolute base `prompts` directory containing the real prompt, verify its resolved prompt exists, then overlay every explicit caller path so those locations remain authoritative. If a relative CLI spelling does not exist from process CWD, keep the valid discovered prompt instead of overwriting it. Derivative/new prompts may not yet be registered; on discovery failure or a missing discovered prompt, return the explicit touched paths so their hashes still undergo hard validation. This prevents normal partial update/CI refreshes from replacing untouched example/test hashes with null while preserving issue #983: `save_fingerprint` itself never re-discovers a non-empty explicit mapping. The decorator uses this helper for the complete fingerprint path set but retains the lighter prompt hint for log/run-report anchoring. Expose `_detect_project_root` and `_detect_project_root_from_paths` for `sync_determine_operation` reuse. 5. **Event Logging** - Log special events (lock_acquired, budget_warning, etc.) to the sync log. **Issue #1211**: `log_event` accepts an optional `paths` argument and forwards it to `append_log_entry` so lifecycle events (sync_start, lock_acquired, lock_released, budget_warning/exceeded, cycle_detected, steering_override, test_extend_skipped/limit, import_validation_failed, auto_fix_attempted/success, etc.) land in the same subproject `.pdd/meta` as the fingerprints/run reports. Without this, lifecycle logs split across two trees when invoked from a parent CWD. `load_operation_log` accepts the same `paths` hint so dry-run / log-display reads find the subproject log. - Sync callers may attach per-phase compression status and agentic-fallback status to log entries/events. Store only metadata such as requested, used, phase, hashes, source counts, estimates, reason, and fallback booleans. @@ -49,7 +50,7 @@ A shared logging infrastructure module for tracking all PDD operations. Provides A decorator for CLI commands that automatically logs operations and manages state files. Parameters control which state files are updated: - `updates_fingerprint` - Save fingerprint on success via `FingerprintTransaction`; `FingerprintFinalizeError` propagates to Click boundary (non-zero exit) - `updates_run_report` - Save run report on success (if result is a dict) -- `clears_run_report` - Clear stale run report only after the wrapped command succeeds, before `FingerprintTransaction.__exit__`. Verify a pre-existing run report was removed; if it remains or cannot be inspected, call `transaction.skip(reason="run report not cleared")` rather than warning and skipping ad-hoc. `FingerprintFinalizeError` propagates to Click boundary. A failed command must not clear the existing run report (issue #1057). +- `clears_run_report` - Clear stale run report only after the wrapped command succeeds, before fingerprint finalization. Verify a pre-existing run report was removed; if it remains, raise `FingerprintFinalizeError` so the command cannot return a false success. A failed wrapped command must not clear the existing run report (issue #1057). The decorator must: - Work with Click commands diff --git a/pdd/prompts/sync_main_python.prompt b/pdd/prompts/sync_main_python.prompt index 8c1d5ae754..e55658d263 100644 --- a/pdd/prompts/sync_main_python.prompt +++ b/pdd/prompts/sync_main_python.prompt @@ -15,7 +15,7 @@ code_generator_main_python.prompt agentic_sync_runner_python.prompt compressed_sync_context_python.prompt -fingerprint_transaction_python.prompt +operation_log_python.prompt # sync_main_python.prompt % You are an expert Python engineer. Your goal is to write a Python function, 'sync_main', that will be the CLI wrapper for the sync command. This function handles interface logic (environment variables, parameter validation, path construction) and calls sync_orchestration to perform the actual sync workflow. @@ -125,7 +125,7 @@ - Merge costs: add `pre_cost` (including failed conformance attempts and the final successful generate attempt) to the one-session result's `total_cost`; pass only `remaining_budget - pre_cost` into the one-session phase. - Include snapshot status and artifact paths from generation in the one-session summary/result when available, without displaying sensitive artifact contents or forbidden detail strings. - If code generation fails (code file still missing), return failure without proceeding to one-session - - **Post-sync**: Save fingerprint via `FingerprintTransaction` (not the `save_fingerprint` wrapper) so next sync sees files as up-to-date; `FingerprintFinalizeError` propagates to Click (non-zero exit) — one-session sync MUST NOT catch it. + - **Post-sync**: Save the complete `pdd_files` mapping through `operation_log.save_fingerprint`, the shared `FingerprintTransaction` wrapper, so the next sync sees files as up to date. Do not catch its `FingerprintFinalizeError`; one-session sync must exit non-zero when finalization fails. - **Auto-submit**: On success and not local mode, call `_auto_submit_example()` to submit the example to PDD Cloud 9. **Auto-Submit Example** (`_auto_submit_example` helper): diff --git a/pdd/prompts/sync_orchestration_python.prompt b/pdd/prompts/sync_orchestration_python.prompt index d9a92d9190..556d735f7f 100644 --- a/pdd/prompts/sync_orchestration_python.prompt +++ b/pdd/prompts/sync_orchestration_python.prompt @@ -180,7 +180,6 @@ from .operation_log import ( update_log_entry, append_log_entry, log_event, - save_fingerprint, save_run_report, clear_run_report, ) @@ -292,7 +291,7 @@ Use `AtomicStateUpdate` context manager for consistent state writes: - Ensures run_report and fingerprint are both written or neither is written - Uses temp file + atomic rename pattern for crash safety - After each successful operation, save a `Fingerprint` to disk with current file hashes via `FingerprintTransaction`. `FingerprintFinalizeError` propagates through the operation dispatch to Click; it MUST NOT be caught by per-operation `except` handlers. -- **No-op fix handling**: for no-op fix operations (detected before finalization — see "Return Value Parsing" below), construct a `FingerprintTransaction` and call `transaction.skip(reason="no-op fix — no LLM invocation")` instead of omitting the transaction entirely, to preserve the single-code-path invariant. +- **No-op fix handling**: a zero-cost fix with an empty/none/unknown/n/a model performed no mutation. Detect it before finalization and do not call `_save_fingerprint_atomic`; this intentional non-mutating path must not claim `fix` completed. - The fingerprint-write leg of `AtomicStateUpdate` and `_save_fingerprint_atomic` MUST route through `FingerprintTransaction`. `_save_fingerprint_atomic` becomes a thin wrapper that constructs and enters a `FingerprintTransaction`. - **Case-insensitive language in paths**: All metadata file paths (fingerprint `.json`, run report `_run.json`, sync log `_sync.log`) must use `language.lower()` for consistent paths on case-sensitive filesystems. This applies to any direct path construction in this module (e.g., `f"{_safe_basename(basename)}_{language.lower()}_run.json"`). - **Subproject anchoring (issue #1211)**: never compose `META_DIR / ` directly for runtime writes — the module-level `META_DIR` is CWD-resolved at import time and won't follow a subproject `.pddrc` that lives below the run CWD. Instead, route fingerprint/run-report/log paths through `get_fingerprint_path(...)`, `get_run_report_path(...)`, and `get_log_path(...)` (from `operation_log`) and pass `paths=pdd_files` so the helper resolves the meta dir via upward `.pddrc` detection from those paths. `_save_run_report_atomic` both accepts `paths` and forwards it into the atomic-state file path AND into `read_fingerprint` for stored-deps lookup. Every `read_run_report` / `clear_run_report` / `log_event` / `append_log_entry` call inside `sync_orchestration` must pass `paths=pdd_files` once `pdd_files` is in scope — including the post-`pdd_files` branches in coverage retry, crash-detection, generate logic, lifecycle events (sync_start, lock_acquired, lock_released, budget_warning/exceeded, cycle_detected, steering_override, etc.), and per-operation log_entry appends. The dry-run / `_display_sync_log` branch executes BEFORE `pdd_files` is set, so it does a best-effort `get_pdd_file_paths(...)` lookup (wrapped in try/except) and forwards the result via `paths=` so log display still finds the subproject log when invoked from a parent CWD. Mismatched paths leave stale state behind and let stale parent metadata leak into decision logic. The only acceptable `META_DIR` reference outside of these helpers is the `if __name__ == '__main__':` demo block at the end of the file. @@ -375,7 +374,7 @@ Use these functions from `operation_log` module (already imported above): - `update_log_entry(entry, success, cost, model, duration, error)` - Add execution results to log entry - `append_log_entry(basename, language, entry, paths=None)` - Append entry to log file; pass `paths=pdd_files` so the log anchors at the subproject `.pdd/meta` (issue #1211) - `log_event(basename, language, event_type, details, invocation_mode, paths=None)` - Log system events (lock_acquired, budget_warning, etc.); pass `paths=pdd_files` for subproject anchoring -- `save_fingerprint(basename, language, operation, paths=None, cost, model)` - Thin wrapper around `FingerprintTransaction`; `FingerprintFinalizeError` propagates. Pass `paths=pdd_files` for subproject anchoring. Use `FingerprintTransaction` directly when skip control is needed. +- `_save_fingerprint_atomic(...)` - Thin sync adapter around `FingerprintTransaction`. Pass `paths=pdd_files`, the outer `AtomicStateUpdate`, and any include-deps override; `FingerprintFinalizeError` propagates. - `save_run_report(basename, language, report_data, paths=None)` - Save RunReport dict; pass `paths=pdd_files` for subproject anchoring (issue #1211) - `clear_run_report(basename, language, paths=None)` - Remove stale run report; pass `paths=pdd_files` for subproject anchoring @@ -445,7 +444,7 @@ Handle command results in multiple formats: - For test operation failure: if `result[4]` (error_message) is non-empty, append it to the errors list so it reaches the sync log - For crash/fix operations returning tuples: extract error message from `result[1]` on failure and append to errors list - For `ArchitectureConformanceError` exception paths, extract `total_cost` and `model_name` from the exception into a result-shaped dict before logging so failed conformance attempts do not report `$0.00`. -- For fingerprint finalization after successful operations: construct a `FingerprintTransaction` for every operation regardless of format. **For no-op fix operations**, call `transaction.skip(reason="no-op fix — no LLM invocation")` instead of skipping the transaction entirely. A no-op fix is detected when `operation == 'fix'` and `actual_cost == 0.0` and `model_name` is empty/`'none'`/`'unknown'`/`'n/a'`. This check **must run before** `FingerprintTransaction.__exit__` so that a logical failure (consecutive no-op breaker) does not persist stale "fix completed" state. +- After parsing a successful mutating result, call `_save_fingerprint_atomic`, which constructs `FingerprintTransaction` with the outer atomic-state buffer. **For no-op fix operations**, omit the adapter call because no artifact mutation occurred. A no-op fix is detected when `operation == 'fix'`, `actual_cost == 0.0`, and `model_name` is empty/`'none'`/`'unknown'`/`'n/a'`. This check must run before finalization so the consecutive-no-op breaker cannot persist stale "fix completed" state. ### Non-Python Language Handling and Agentic Mode diff --git a/pdd/prompts/update_main_python.prompt b/pdd/prompts/update_main_python.prompt index 0c2ba06d4e..590c3a9d88 100644 --- a/pdd/prompts/update_main_python.prompt +++ b/pdd/prompts/update_main_python.prompt @@ -67,16 +67,13 @@ Supports three modes: true update, regeneration, and repository-wide updates. Ro - **Failure reporting**: When any stage in `MetadataSyncResult` is not `ok`, print a Rich error line per failed stage with the stage name and reason. The `metadata` column makes it obvious which layer is incomplete. - **Backward compatibility**: When `sync_metadata=False` (default), behavior is unchanged for repo mode — it keeps its current post-update fingerprint and architecture-sync calls. Single-file and regeneration modes still run the default fingerprint finalization defined in Requirement 15 (this is the only metadata write they perform by default). 15. Default fingerprint finalization for single-file and regeneration modes: - - After a successful single-file or regeneration update, after the prompt has been written and before returning a non-None `(prompt, cost, model)` tuple, write a current fingerprint for the affected `(prompt, code)` pair using an explicit `FingerprintTransaction`. - - Reuse `infer_module_identity(prompt_path)` from `pdd.operation_log`; construct `FingerprintTransaction(basename, language, operation="update", paths={"prompt": Path(prompt_path), "code": Path(code_path)}, cost=cost, model=model)` (omitting `"code"` when code path is not available) so `.pdd/meta` anchors at the subproject root when running from a parent directory (issue #1211). - - Skip this default finalization in repo mode because repo mode already performs post-update fingerprinting. - - Skip conditions — call `transaction.skip(reason=...)` rather than returning early from the transaction: - - If `sync_metadata=True`: `transaction.skip(reason="orchestrator owns fingerprint stage")`; print `[info][metadata] Skipping fingerprint finalization: orchestrator owns fingerprint stage[/info]` unless `quiet`. - - If `dry_run=True`: `transaction.skip(reason="dry-run mode")`; print `[info][metadata] Skipping fingerprint finalization: dry-run mode[/info]` unless `quiet`. - - If the written prompt path differs from the canonical source prompt (output redirected): `transaction.skip(reason="output redirected")`; print `[info][metadata] Skipping fingerprint finalization: output redirected[/info]` unless `quiet`. This guard applies in addition to (and independently of) the `sync_metadata=True` skip. - - If `infer_module_identity(prompt_path)` returns `(None, None)`: `transaction.skip(reason="unable to infer module identity")`; print `[info][metadata] Skipping fingerprint finalization: unable to infer module identity for [/info]` unless `quiet`. - - `FingerprintFinalizeError` propagates — the successful return tuple is NOT preserved on fingerprint write failure; Click exits non-zero. This is the enforced commit-or-fail contract (issue #1926). - - Stale `_run.json` cleanup MUST reuse `pdd.operation_log._clear_run_report_before_fingerprint(basename, language, paths={"prompt": Path(prompt_path), "code": Path(code_path)})` (omitting `"code"` when unavailable), which clears the run report then re-checks existence and returns `False` (with a yellow console warning) when a silent `os.remove` left it on disk. When the helper returns `False`, call `transaction.skip(reason="run report not cleared")` so a fresh fingerprint never coexists with stale runtime state (issue #1106). The helper's warning surfaces unconditionally — it does NOT honour the caller's `quiet` flag, intentionally, because it describes a real metadata problem. Wrap the helper call in `try/except`: on an unexpected exception, emit `[warning][metadata] Run report clear failed: [/warning]` unless `quiet` and call `transaction.skip(reason="run report clear failed")` (best-effort cleanup must never block the transaction context). The function-local `from .operation_log import (_clear_run_report_before_fingerprint, infer_module_identity)` and `from .fingerprint_transaction import FingerprintTransaction, FingerprintFinalizeError` MUST be wrapped in `try/except ImportError`: on ImportError, emit `[warning][metadata] Could not import finalization helpers: [/warning]` unless `quiet` and return without entering the transaction. + - After a successful single-file or regeneration update, after the prompt has been written and before returning a non-None `(prompt, cost, model)` tuple, call one `_finalize_single_file_fingerprint` helper. It delegates persistence to `operation_log.save_fingerprint`, the shared `FingerprintTransaction` wrapper. + - Resolve identity with `infer_module_identity(prompt_path)`, then build a complete unit mapping with `resolve_fingerprint_paths(basename, language, prompt_path, paths={"prompt": Path(prompt_path), "code": Path(code_path)})`. The explicit written paths remain authoritative while existing example/test hashes are preserved, and the prompt path anchors `.pdd/meta` at a nested subproject (issues #1211/#1290). + - Repo mode invokes the same helper once per successful pair from its post-update loop; individual pair workers do not finalize a second time. + - Intentional non-mutating conditions return before finalization: `sync_metadata=True` (orchestrator owns the stage), `dry_run=True`, or redirected output. Print the existing informational skip line unless `quiet`. + - Inability to infer identity is NOT an intentional skip after a real write: raise `FingerprintFinalizeError("update", prompt_path, "unable to infer module identity")`. + - Stale `_run.json` cleanup uses `_clear_run_report_before_fingerprint(..., paths=complete_paths)`. An exception or a false return becomes `FingerprintFinalizeError`; never warn-and-return after the artifact mutated. The helper's own warning about a surviving report remains visible even under quiet mode. + - Import failures for finalization helpers are wrapped as `FingerprintFinalizeError`. `save_fingerprint` errors propagate. At the command boundary, print the explicit diagnostic and raise `click.exceptions.Exit(1)` so every single-file and repo-mode finalization failure is non-zero. % Modes 1. **True update**: `input_prompt_file` + `modified_code_file` + (use_git OR `input_code_file`). @@ -115,7 +112,7 @@ Supports three modes: true update, regeneration, and repository-wide updates. Ro - After resolving pairs, call `ensure_pddrc_for_scan` from `pddrc_initializer` to auto-create `.pddrc` if needed. - **Change detection**: Use `get_git_changed_files` + `is_code_changed` to filter pairs. Also include pairs with empty (0-byte) prompt files regardless of code changes. Include dependency hashes from fingerprints are checked so that changes to shared include files (preambles, examples) trigger updates. - Progress bar: Rich Progress with spinner, bar, percentage, time remaining, and running total cost. -- **Post-update fingerprinting** (legacy, `sync_metadata=False` only): After each successful pair update, infer module identity via `infer_module_identity`, then call `clear_run_report(basename, language, paths={"prompt": Path(prompt_path), "code": Path(code_path)})` BEFORE `save_fingerprint` so stale `.pdd/meta/__run.json` runtime-verification state from before the mutation is invalidated rather than left to coexist with the freshly written fingerprint. Pass the same `paths` hint to `get_run_report_path` and `save_fingerprint` so all metadata writes anchor at the subproject root (issue #1211). Wrap `clear_run_report` in its own try/except and surface failures as a non-fatal warning (mirroring the single-file finalize path in `_finalize_target_fingerprint`); after the clear attempt, re-check `get_run_report_path(basename, language, paths=...).exists()` and **skip `save_fingerprint` for that pair (with a yellow warning) when the stale `_run.json` still exists** — this is the safer behavior required by issue #1057 so a fresh fingerprint can never coexist with stale runtime-verification state. When `sync_metadata=True`, the per-pair `run_metadata_sync` call replaces this — do NOT also call `clear_run_report` or `save_fingerprint` here, or the fingerprint will be written before the orchestrator's gates run and a stale fingerprint can record "in sync" even when an earlier metadata stage failed. +- **Post-update fingerprinting** (legacy, `sync_metadata=False` only): After every successful pair, call the same `_finalize_single_file_fingerprint` helper used by single-file modes. It resolves the complete unit paths, clears and verifies the stale run report, and delegates to the shared save wrapper. Any `FingerprintFinalizeError` prints an explicit diagnostic and raises `click.exceptions.Exit(1)`; repo mode may not keep processing after a pair mutated without a durable fingerprint. When `sync_metadata=True`, the per-pair orchestrator replaces this path, so do not also finalize early. - **Post-update architecture sync** (legacy, `sync_metadata=False` only): Use `find_architecture_for_project` to locate architecture files; for each successfully updated prompt, call `update_architecture_from_prompt` if architecture file exists. When `sync_metadata=True`, this is handled inside `run_metadata_sync` (which also gates the write on prior-stage success), so do NOT call it again from `update_main`. - **Post-update PRD sync** (both branches): If architecture entries were updated, find PRD file and use `run_agentic_task` to review/update PRD content. Unpack the agent result as `(llm_success, llm_output, llm_cost, _llm_model)`, add `llm_cost` to the repository total when present, and treat `llm_success=False` as an error status without writing the PRD. Parse `...` tags from `llm_output` only when the call succeeded; otherwise leave the PRD unchanged. PRD sync stays in `update_main` even under `sync_metadata=True` — the orchestrator deliberately does NOT call it (see Requirement 11). Key off the count of pairs whose `MetadataSyncResult.stages['architecture']` reported `updated`/`ok` (or, in the legacy branch, off the count returned by the inline arch loop). - Summary table: `prompt file`, `status`, `cost`, `model`, `error` columns. When `sync_metadata=True`, append a `metadata` column (one of `synced`, `partial:`, `failed:`, `skipped`, `dry-run`) so partial heals surface visibly. Show architecture/PRD status if relevant. diff --git a/pdd/sync_orchestration.py b/pdd/sync_orchestration.py index 4e7e70c145..cd342150a4 100644 --- a/pdd/sync_orchestration.py +++ b/pdd/sync_orchestration.py @@ -38,13 +38,13 @@ update_log_entry, append_log_entry, log_event, - save_fingerprint, save_run_report, clear_run_report, get_log_path, get_run_report_path, get_fingerprint_path, ) +from .json_atomic import atomic_write_json from .sync_determine_operation import ( sync_determine_operation, get_pdd_file_paths, @@ -454,6 +454,7 @@ class PendingStateUpdate: fingerprint: Optional[Dict[str, Any]] = None run_report_path: Optional[Path] = None fingerprint_path: Optional[Path] = None + fingerprint_operation: Optional[str] = None @dataclass @@ -500,41 +501,75 @@ def set_run_report(self, report: Dict[str, Any], path: Path): self.pending.run_report = report self.pending.run_report_path = path - def set_fingerprint(self, fingerprint: Dict[str, Any], path: Path): + def set_fingerprint( + self, + fingerprint: Dict[str, Any], + path: Path, + *, + operation: Optional[str] = None, + ): """Buffer a fingerprint for atomic write.""" self.pending.fingerprint = fingerprint self.pending.fingerprint_path = path + self.pending.fingerprint_operation = operation def _atomic_write(self, data: Dict[str, Any], target_path: Path) -> None: """Write data to file atomically using temp file + rename pattern.""" - target_path.parent.mkdir(parents=True, exist_ok=True) + atomic_write_json(target_path, data) - # Write to temp file in same directory (required for atomic rename) - fd, temp_path = tempfile.mkstemp( - dir=target_path.parent, - prefix=f".{target_path.stem}_", - suffix=".tmp" - ) - self._temp_files.append(temp_path) + def _commit(self): + """Commit all pending state updates atomically.""" + snapshots: List[FileRollbackSnapshot] = [] + for path in ( + self.pending.fingerprint_path, + self.pending.run_report_path, + ): + if path is None: + continue + if path.exists(): + snapshots.append( + FileRollbackSnapshot(path=path, existed=True, content=path.read_bytes()) + ) + else: + snapshots.append(FileRollbackSnapshot(path=path, existed=False)) try: - with os.fdopen(fd, 'w') as f: - json.dump(data, f, indent=2, default=str) + # Write fingerprint first (checkpoint), then run_report. If either + # leg fails, restore both destinations from the snapshots above so + # the outer state context cannot leave a half-committed pair. + if self.pending.fingerprint and self.pending.fingerprint_path: + self._atomic_write( + self.pending.fingerprint, + self.pending.fingerprint_path, + ) + if self.pending.run_report and self.pending.run_report_path: + self._atomic_write( + self.pending.run_report, + self.pending.run_report_path, + ) + except Exception as exc: + for snapshot in reversed(snapshots): + try: + if snapshot.existed: + _atomic_write_bytes(snapshot.path, snapshot.content or b"") + elif snapshot.path.exists(): + snapshot.path.unlink() + except OSError as rollback_exc: + logger.error( + "Failed to roll back atomic metadata path %s: %s", + snapshot.path, + rollback_exc, + ) - # Atomic rename - guaranteed atomic on POSIX systems - os.replace(temp_path, target_path) - self._temp_files.remove(temp_path) # Successfully moved, stop tracking - except Exception: - # Leave temp file for rollback to clean up - raise + if self.pending.fingerprint_path is not None: + from .fingerprint_transaction import FingerprintFinalizeError - def _commit(self): - """Commit all pending state updates atomically.""" - # Write fingerprint first (checkpoint), then run_report - if self.pending.fingerprint and self.pending.fingerprint_path: - self._atomic_write(self.pending.fingerprint, self.pending.fingerprint_path) - if self.pending.run_report and self.pending.run_report_path: - self._atomic_write(self.pending.run_report, self.pending.run_report_path) + raise FingerprintFinalizeError( + self.pending.fingerprint_operation or "unknown", + self.pending.fingerprint_path, + f"atomic state commit failed: {exc}", + ) from exc + raise def _rollback(self): """Clean up any temp files without committing changes.""" @@ -682,43 +717,21 @@ def _save_fingerprint_atomic(basename: str, language: str, operation: str, include_deps_override: Pre-captured include deps (Issue #522). Used when auto-deps may have stripped tags before fingerprint save. """ - if atomic_state: - # Buffer for atomic write - from datetime import datetime, timezone - from .sync_determine_operation import calculate_current_hashes, Fingerprint, read_fingerprint - from . import __version__ - - # Issue #522: Use override deps if provided (captured before auto-deps), - # otherwise fall back to stored deps from previous fingerprint - if include_deps_override is not None: - stored_deps = include_deps_override - else: - prev_fp = read_fingerprint(basename, language, paths=paths) - stored_deps = prev_fp.include_deps if prev_fp else None - current_hashes = calculate_current_hashes(paths, stored_include_deps=stored_deps) - # If override provided and current extraction found nothing, use the override - if include_deps_override and not current_hashes.get('include_deps'): - current_hashes['include_deps'] = include_deps_override - fingerprint = Fingerprint( - pdd_version=__version__, - timestamp=datetime.now(timezone.utc).isoformat(), - command=operation, - prompt_hash=current_hashes.get('prompt_hash'), - code_hash=current_hashes.get('code_hash'), - example_hash=current_hashes.get('example_hash'), - test_hash=current_hashes.get('test_hash'), - test_files=current_hashes.get('test_files'), # Bug #156 - include_deps=current_hashes.get('include_deps'), # Issue #522 - ) - - # Issue #1211: route the atomic fingerprint file through the - # paths-aware helper so subprojects whose .pddrc is below run CWD - # get the file under /.pdd/meta, not parent CWD. - fingerprint_file = get_fingerprint_path(basename, language, paths=paths) - atomic_state.set_fingerprint(asdict(fingerprint), fingerprint_file) - else: - # Direct write using operation_log - save_fingerprint(basename, language, operation, paths, cost, model) + from .fingerprint_transaction import FingerprintTransaction + + transaction = FingerprintTransaction( + basename=basename, + language=language, + operation=operation, + paths=paths, + cost=cost, + model=model, + atomic_state=atomic_state, + ) + if include_deps_override is not None: + transaction.set_include_deps_override(include_deps_override) + with transaction: + pass def _python_cov_target_for_code_file(code_file: Path) -> str: """Return a `pytest-cov` `--cov` target for a Python code file. diff --git a/pdd/update_main.py b/pdd/update_main.py index 0543cbb2f3..c06aef22af 100644 --- a/pdd/update_main.py +++ b/pdd/update_main.py @@ -34,6 +34,9 @@ from .agentic_update import run_agentic_update from .sync_determine_operation import calculate_sha256, extract_include_deps, read_fingerprint from .validate_prompt_includes import sanitize_prompt_output +from .fingerprint_transaction import ( + FingerprintFinalizeError, +) from . import DEFAULT_TIME # Issue #1714: bound the PRD-sync agentic step below the 600s @@ -1080,16 +1083,15 @@ def _finalize_single_file_fingerprint( ) -> None: """Default fingerprint finalization for single-file/regeneration update modes. - Writes a current `(prompt, code)` fingerprint via the existing - `pdd.operation_log` helpers so a successful `pdd update ` is not - re-detected as changed on the next run (issue #1007 / PR #1009 - Requirement 15). Skips with an `[info]` log line — unless `quiet` — for - the intentional skip cases (sync_metadata orchestrator owns the stage, - dry-run mode, `--output` redirected the write away from the canonical - source prompt, or `infer_module_identity` cannot derive - basename/language). All failures are best-effort: a `save_fingerprint` - exception surfaces as a `[warning]` line but never breaks the caller's - success tuple. + Writes a complete unit fingerprint through the shared operation-log + transaction so a successful `pdd update ` is not re-detected as + changed on the next run (issue #1007 / PR #1009 Requirement 15). Only + genuinely non-mutating paths skip: the metadata orchestrator owns the + stage, dry-run mode, or `--output` redirected the write away from the + canonical source prompt. Identity, run-report cleanup, path resolution, + and persistence failures are hard ``FingerprintFinalizeError`` failures; + a real artifact mutation may not retain the caller's success tuple unless + its fingerprint commits. ``source_prompt_path`` is the canonical input prompt for the module. When callers pass a redirected output path via ``--output``, the written file @@ -1121,37 +1123,27 @@ def _finalize_single_file_fingerprint( ) return - # Wrap the import itself so the user's successful update tuple is never - # broken by an import-time failure (e.g. `_clear_run_report_before_fingerprint` - # gets renamed in a future operation_log refactor — it's a private - # underscore-prefixed name and therefore more fragile than the public - # `clear_run_report` / `infer_module_identity` / `save_fingerprint` - # alongside it). An ImportError raised here would propagate up to - # `update_main`'s outer `except Exception: return None`, converting a - # successful `(prompt, cost, model)` tuple to None — which violates the - # issue #1106 acceptance criterion: best-effort metadata cleanup must - # never fail the successful update tuple. try: from .operation_log import ( _clear_run_report_before_fingerprint, + get_fingerprint_path, infer_module_identity, + resolve_fingerprint_paths, save_fingerprint, ) except ImportError as exc: - if not quiet: - rprint( - f"[warning][metadata] Could not import finalization helpers: " - f"{exc}[/warning]" - ) - return + raise FingerprintFinalizeError( + "update", + prompt_path, + f"could not import finalization helpers: {exc}", + ) from exc basename, language = infer_module_identity(prompt_path) if not (basename and language): - if not quiet: - rprint( - "[info][metadata] Skipping fingerprint finalization: " - f"unable to infer module identity for {prompt_path}[/info]" - ) - return + raise FingerprintFinalizeError( + "update", + prompt_path, + "unable to infer module identity", + ) # Reuse the shared helper so the single-file finalize path enforces the # same invariant the `log_operation` decorator and repo-mode block already @@ -1171,24 +1163,30 @@ def _finalize_single_file_fingerprint( # nearest .pddrc), not a parent CWD orphan. Without this we cleared # parent metadata while writing the fresh fingerprint to the subproject, # leaving stale subproject _run.json beside it. - update_paths = {"prompt": Path(prompt_path), "code": Path(code_path)} + update_paths = resolve_fingerprint_paths( + basename, + language, + prompt_path, + paths={"prompt": Path(prompt_path), "code": Path(code_path)}, + ) try: fingerprint_allowed = _clear_run_report_before_fingerprint( - basename, language, paths=update_paths + basename, + language, + paths=update_paths, ) except Exception as exc: - # Defensive: surrounding pattern in this function treats metadata - # cleanup as best-effort; an unexpected raise must not break the - # successful update tuple. Warn and skip the save, matching the - # `save_fingerprint` except-arm below. - if not quiet: - rprint( - f"[warning][metadata] Run report clear failed: {exc}[/warning]" - ) - return + raise FingerprintFinalizeError( + "update", + get_fingerprint_path(basename, language, paths=update_paths), + f"run report clear failed: {exc}", + ) from exc if not fingerprint_allowed: - return - + raise FingerprintFinalizeError( + "update", + get_fingerprint_path(basename, language, paths=update_paths), + "run report not cleared", + ) try: save_fingerprint( basename, @@ -1198,9 +1196,14 @@ def _finalize_single_file_fingerprint( cost=cost, model=model, ) + except FingerprintFinalizeError: + raise except Exception as exc: - if not quiet: - rprint(f"[warning][metadata] Fingerprint save failed: {exc}[/warning]") + raise FingerprintFinalizeError( + "update", + get_fingerprint_path(basename, language, paths=update_paths), + exc, + ) from exc def update_main( @@ -1385,85 +1388,24 @@ def update_main( if not sync_metadata: # Save fingerprint so the file isn't detected as changed next run if "Success" in result.get("status", ""): - from .operation_log import ( - clear_run_report, - get_run_report_path, - infer_module_identity, - save_fingerprint, - ) - basename, language = infer_module_identity(prompt_path) - if basename and language: - # Issue #1211: route all three metadata calls - # (get_run_report_path / clear_run_report / - # save_fingerprint) through the same `paths` hint - # so they hit the subproject .pdd/meta — not a - # parent CWD orphan — when the user invokes - # update from above the subproject root. - _update_paths = { - "prompt": Path(prompt_path), - "code": Path(code_path), - } - # Clear stale run report first so it can't outlive - # the prompt/code pair it described. Best-effort: - # never fail the update because of metadata I/O, - # but surface failures as a non-fatal warning so - # the user knows runtime verification state may - # still describe the pre-mutation files. - try: - _stale_report_path = get_run_report_path( - basename, language, paths=_update_paths - ) - except Exception: - _stale_report_path = None - _pre_existed = bool( - _stale_report_path is not None - and _stale_report_path.exists() + try: + _finalize_single_file_fingerprint( + Path(prompt_path), + Path(code_path), + sync_metadata=False, + dry_run=False, + quiet=quiet, + cost=result.get("cost", 0.0), + model=result.get("model", "unknown"), + source_prompt_path=Path(prompt_path), ) - try: - clear_run_report(basename, language, paths=_update_paths) - except Exception as exc: - if not quiet: - rprint( - f"[warning][metadata] Run report clear failed for " - f"{basename} ({language}): {exc}[/warning]" - ) - # Defensive: clear_run_report() in pdd.operation_log - # silently swallows OSError on the actual unlink - # (see pdd/operation_log.py:317-320), so if the - # report file existed before the call but still - # exists afterwards, the deletion failed silently. - # Surface that as a non-fatal warning so the user - # knows runtime verification state may still - # describe the pre-mutation files. - _stale_remains = False - if _pre_existed and _stale_report_path is not None: - try: - _still_there = _stale_report_path.exists() - except Exception: - _still_there = False - if _still_there: - _stale_remains = True - if not quiet: - rprint( - f"[warning][metadata] Run report clear failed for " - f"{basename} ({language}): " - f"still exists after clear_run_report: " - f"{_stale_report_path}; skipping fingerprint update so a " - f"fresh fingerprint does not coexist with a stale " - f"run report (issue #1057)." - f"[/warning]" - ) - if not _stale_remains: - try: - save_fingerprint( - basename, language, - operation="update", - paths=_update_paths, - cost=result.get("cost", 0.0), - model=result.get("model", "unknown"), - ) - except Exception: - pass # Best-effort; don't fail the update + except FingerprintFinalizeError as exc: + if not quiet: + rprint( + "[bold red]Fingerprint finalization failed:" + f"[/bold red] {exc}" + ) + raise click.exceptions.Exit(1) from exc else: if "Success" in result.get("status", ""): try: @@ -2038,6 +1980,10 @@ def _metadata_column_value(prompt_file_key: str) -> str: # (#871). Letting the bare `except Exception` below swallow this would # silently convert it to exit 0. raise + except FingerprintFinalizeError as e: + if not quiet: + rprint(f"[bold red]Fingerprint finalization failed:[/bold red] {e}") + raise click.exceptions.Exit(1) from e except Exception as e: if not quiet: rprint(f"[bold red]Error:[/bold red] {str(e)}") diff --git a/tests/test_auto_deps_main.py b/tests/test_auto_deps_main.py index b6f7814229..2fe33fbef9 100644 --- a/tests/test_auto_deps_main.py +++ b/tests/test_auto_deps_main.py @@ -52,7 +52,13 @@ def _isolate_metadata_finalization(request): yield return with patch("pdd.auto_deps_main.save_fingerprint"), \ - patch("pdd.auto_deps_main.clear_run_report"): + patch( + "pdd.auto_deps_main._clear_run_report_before_fingerprint", + return_value=True, + ), patch( + "pdd.auto_deps_main.infer_module_identity", + return_value=("test", "python"), + ): yield @@ -758,7 +764,7 @@ def test_auto_deps_main_updates_architecture_json_after_write( # ``(basename, language)`` so canonical metadata stays untouched. # --------------------------------------------------------------------------- @patch("pdd.auto_deps_main.save_fingerprint") -@patch("pdd.auto_deps_main.clear_run_report") +@patch("pdd.auto_deps_main._clear_run_report_before_fingerprint") @patch("pdd.auto_deps_main.infer_module_identity") @patch("pdd.auto_deps_main.construct_paths") @patch("pdd.auto_deps_main.insert_includes") @@ -802,7 +808,10 @@ def test_auto_deps_metadata_finalizes_with_output_identity_in_default_mode( assert fp_kwargs["basename"] == "child_python_with" assert fp_kwargs["language"] == "deps" assert fp_kwargs["operation"] == "auto-deps" - assert fp_kwargs["paths"] == {"prompt": Path(output_path)} + assert fp_kwargs["paths"]["prompt"] == Path(output_path) + assert {"prompt", "code", "example", "test"}.issubset( + fp_kwargs["paths"] + ) # --------------------------------------------------------------------------- @@ -814,7 +823,7 @@ def test_auto_deps_metadata_finalizes_with_output_identity_in_default_mode( # identity. # --------------------------------------------------------------------------- @patch("pdd.auto_deps_main.save_fingerprint") -@patch("pdd.auto_deps_main.clear_run_report") +@patch("pdd.auto_deps_main._clear_run_report_before_fingerprint") @patch("pdd.auto_deps_main.infer_module_identity") @patch("pdd.auto_deps_main.construct_paths") @patch("pdd.auto_deps_main.insert_includes") @@ -862,7 +871,10 @@ def test_auto_deps_metadata_finalizes_with_canonical_identity_inplace( assert fp_kwargs["basename"] == "child" assert fp_kwargs["language"] == "python" assert fp_kwargs["operation"] == "auto-deps" - assert fp_kwargs["paths"] == {"prompt": Path(output_path)} + assert fp_kwargs["paths"]["prompt"] == Path(output_path) + assert {"prompt", "code", "example", "test"}.issubset( + fp_kwargs["paths"] + ) assert fp_kwargs["model"] == "test-model" assert fp_kwargs["cost"] == pytest.approx(0.123456) @@ -872,7 +884,7 @@ def test_auto_deps_metadata_finalizes_with_canonical_identity_inplace( # i.e. ``infer_module_identity`` returns ``(None, None)``. # --------------------------------------------------------------------------- @patch("pdd.auto_deps_main.save_fingerprint") -@patch("pdd.auto_deps_main.clear_run_report") +@patch("pdd.auto_deps_main._clear_run_report_before_fingerprint") @patch("pdd.auto_deps_main.infer_module_identity") @patch("pdd.auto_deps_main.construct_paths") @patch("pdd.auto_deps_main.insert_includes") @@ -886,11 +898,8 @@ def test_auto_deps_metadata_skipped_on_unknown_identity( tmp_path: Path, ): """ - ``infer_module_identity`` returns ``(None, None)`` (a tuple, not None) - for unrecognized prompt names. The finalization block must handle that - explicitly and skip both ``clear_run_report`` and ``save_fingerprint`` - rather than crash. Use an in-place overwrite so finalization is actually - attempted (otherwise the differing-output guard would short-circuit it). + An output without PDD's basename/language naming is an unmanaged redirect, + so it intentionally skips module metadata. """ prompt_file = str(tmp_path / "weird_name_no_language.prompt") Path(prompt_file).write_text("orig", encoding="utf-8") @@ -903,7 +912,7 @@ def test_auto_deps_metadata_skipped_on_unknown_identity( mock_insert_includes.return_value = _make_insert_includes_return() mock_infer_identity.return_value = (None, None) - modified_prompt, total_cost, model_name = auto_deps_main( + result = auto_deps_main( ctx=mock_ctx, prompt_file=prompt_file, directory_path="context/", @@ -912,24 +921,21 @@ def test_auto_deps_metadata_skipped_on_unknown_identity( force_scan=False, ) + assert result == _make_insert_includes_return()[:1] + (0.123456, "test-model") mock_clear_run_report.assert_not_called() mock_save_fingerprint.assert_not_called() - # Auto-deps still returns its successful result. - assert modified_prompt == "Modified prompt with includes" - assert total_cost == pytest.approx(0.123456) - assert model_name == "test-model" # --------------------------------------------------------------------------- -# 20. Metadata finalization: clear_run_report failure must not abort the -# subsequent fingerprint save. +# 20. Metadata finalization: a stale run report that cannot be cleared is a +# hard failure and must block the fingerprint write. # --------------------------------------------------------------------------- @patch("pdd.auto_deps_main.save_fingerprint") -@patch("pdd.auto_deps_main.clear_run_report") +@patch("pdd.auto_deps_main._clear_run_report_before_fingerprint") @patch("pdd.auto_deps_main.infer_module_identity") @patch("pdd.auto_deps_main.construct_paths") @patch("pdd.auto_deps_main.insert_includes") -def test_auto_deps_clear_run_report_error_does_not_block_fingerprint( +def test_auto_deps_clear_run_report_error_blocks_fingerprint( mock_insert_includes, mock_construct_paths, mock_infer_identity, @@ -938,7 +944,7 @@ def test_auto_deps_clear_run_report_error_does_not_block_fingerprint( mock_ctx, tmp_path: Path, ): - """If clearing the stale run report fails, the fingerprint must still be saved. + """A fresh fingerprint must not coexist with stale runtime state. Uses an in-place overwrite (``output == prompt_file``) so finalization is actually attempted — the differing-output guard would otherwise skip @@ -953,22 +959,20 @@ def test_auto_deps_clear_run_report_error_does_not_block_fingerprint( ) mock_insert_includes.return_value = _make_insert_includes_return() mock_infer_identity.return_value = ("child", "python") - mock_clear_run_report.side_effect = OSError("permission denied") + mock_clear_run_report.return_value = False - auto_deps_main( - ctx=mock_ctx, - prompt_file=prompt_file, - directory_path="context/", - auto_deps_csv_path=None, - output=None, - force_scan=False, - ) + with pytest.raises(Exception, match="run report not cleared"): + auto_deps_main( + ctx=mock_ctx, + prompt_file=prompt_file, + directory_path="context/", + auto_deps_csv_path=None, + output=None, + force_scan=False, + ) mock_clear_run_report.assert_called_once_with("child", "python", paths=ANY) - mock_save_fingerprint.assert_called_once() - fp_kwargs = mock_save_fingerprint.call_args.kwargs - assert fp_kwargs["basename"] == "child" - assert fp_kwargs["language"] == "python" + mock_save_fingerprint.assert_not_called() # --------------------------------------------------------------------------- diff --git a/tests/test_fingerprint_invariant.py b/tests/test_fingerprint_invariant.py new file mode 100644 index 0000000000..60a47cf4db --- /dev/null +++ b/tests/test_fingerprint_invariant.py @@ -0,0 +1,340 @@ +"""Post-command freshness invariant for every mutating command (#1926). + +The command bodies are kept offline by replacing only their LLM-producing +work. Their production metadata boundaries remain real: decorators, +single/repo update finalization, auto-deps, sync, and CI heal all have to +persist a fingerprint which the real drift classifier accepts as fresh. +""" +from __future__ import annotations + +import json +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import patch + +import click +import pytest + +from pdd.auto_deps_main import auto_deps_main +from pdd.ci_drift_heal import _run_metadata_sync_safe +from pdd.operation_log import log_operation +from pdd.sync_determine_operation import ( + calculate_current_hashes, + read_fingerprint, + sync_determine_operation, +) +from pdd.sync_orchestration import _save_fingerprint_atomic +from pdd.update_main import _finalize_single_file_fingerprint + + +MUTATING_COMMANDS = ( + ("sync", "sync"), + ("generate", "decorator"), + ("example", "decorator"), + ("update", "update"), + ("update --all", "update"), + ("auto-deps", "auto-deps"), + ("fix", "decorator"), + ("CI heal", "ci-heal"), +) + + +@pytest.fixture +def fingerprint_unit( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> tuple[Path, dict[str, Path]]: + root = tmp_path / "project" + paths = { + "prompt": root / "prompts" / "sample_python.prompt", + "code": root / "src" / "sample.py", + "example": root / "examples" / "sample_example.py", + "test": root / "tests" / "test_sample.py", + } + paths["test_files"] = [paths["test"]] + for path in paths.values(): + if isinstance(path, list): + continue + path.parent.mkdir(parents=True, exist_ok=True) + + (root / ".pddrc").write_text( + 'version: "1.0"\n' + "contexts:\n" + " default:\n" + " defaults:\n" + ' generate_output_path: "src/"\n' + ' example_output_path: "examples/"\n' + ' test_output_path: "tests/"\n' + ' default_language: "python"\n', + encoding="utf-8", + ) + paths["prompt"].write_text("% Goal\nCreate sample.\n", encoding="utf-8") + paths["code"].write_text("def sample():\n return 1\n", encoding="utf-8") + paths["example"].write_text( + "from sample import sample\nprint(sample())\n", + encoding="utf-8", + ) + paths["test"].write_text( + "def test_sample():\n assert True\n", + encoding="utf-8", + ) + monkeypatch.chdir(root) + return root, paths + + +def _run_decorated_finalizer(operation: str, paths: dict[str, Path]) -> None: + @log_operation( + operation=operation, + clears_run_report=True, + updates_fingerprint=True, + ) + def mutation(*, prompt_file: str): + if operation == "generate": + return "written", False, 0.25, "invariant-model" + if operation == "fix": + return {}, "fixed", True, 1, 0.25, "invariant-model" + return "written", 0.25, "invariant-model" + + with patch( + "pdd.sync_determine_operation.get_pdd_file_paths", + return_value=paths, + ): + mutation(prompt_file=str(paths["prompt"])) + + +def _run_update_finalizer(paths: dict[str, Path]) -> None: + with patch( + "pdd.operation_log.infer_module_identity", + return_value=("sample", "python"), + ), patch( + "pdd.operation_log._clear_run_report_before_fingerprint", + return_value=True, + ): + _finalize_single_file_fingerprint( + prompt_path=paths["prompt"], + code_path=paths["code"], + sync_metadata=False, + dry_run=False, + quiet=True, + cost=0.25, + model="invariant-model", + ) + + +def _run_auto_deps_finalizer(root: Path, paths: dict[str, Path]) -> None: + original = paths["prompt"].read_text(encoding="utf-8") + csv_path = root / "project_dependencies.csv" + ctx = click.Context(click.Command("auto-deps")) + ctx.obj = { + "quiet": True, + "force": True, + "strength": 0.5, + "temperature": 0.0, + "time": 0.25, + } + with patch( + "pdd.auto_deps_main.construct_paths", + return_value=( + {}, + {"prompt_file": original}, + {"output": str(paths["prompt"]), "csv": str(csv_path)}, + "python", + ), + ), patch( + "pdd.auto_deps_main.insert_includes", + return_value=(original, "", 0.25, "invariant-model"), + ), patch( + "pdd.auto_deps_main.sanitize_prompt_output", + return_value=(original, []), + ), patch( + "pdd.auto_deps_main.merge_auto_deps_includes_from_cwd", + return_value={"updated": False, "added_dependencies": []}, + ), patch( + "pdd.auto_deps_main._clear_run_report_before_fingerprint", + return_value=True, + ): + result = auto_deps_main( + ctx=ctx, + prompt_file=str(paths["prompt"]), + directory_path=str(root), + auto_deps_csv_path=str(csv_path), + output=str(paths["prompt"]), + ) + + assert result[0] == original + fingerprint = read_fingerprint("sample", "python", paths=paths) + assert fingerprint is not None + assert fingerprint.command == "auto-deps" + + +def _run_ci_heal_finalizer(paths: dict[str, Path]) -> None: + with patch( + "pdd.metadata_sync.run_metadata_sync", + return_value=SimpleNamespace(ok=True), + ): + assert _run_metadata_sync_safe( + str(paths["prompt"]), + str(paths["code"]), + ) + + +def _run_command_finalizer( + command: str, + route: str, + root: Path, + paths: dict[str, Path], +) -> None: + if route == "sync": + _save_fingerprint_atomic( + "sample", + "python", + "sync", + paths, + 0.25, + "invariant-model", + ) + elif route == "decorator": + _run_decorated_finalizer(command, paths) + elif route == "update": + _run_update_finalizer(paths) + elif route == "auto-deps": + _run_auto_deps_finalizer(root, paths) + # Standalone auto-deps intentionally advances the workflow to + # generate; model that required continuation before asserting the + # terminal freshness invariant. + _run_decorated_finalizer("generate", paths) + elif route == "ci-heal": + _run_ci_heal_finalizer(paths) + else: # pragma: no cover - registry is closed above + raise AssertionError(f"unknown finalization route: {route}") + + +@pytest.mark.parametrize( + ("command", "route"), + MUTATING_COMMANDS, + ids=[command for command, _route in MUTATING_COMMANDS], +) +def test_mutating_command_leaves_unit_fresh( + command: str, + route: str, + fingerprint_unit: tuple[Path, dict[str, Path]], +) -> None: + """A green mutating command must leave a durable, fresh fingerprint.""" + root, paths = fingerprint_unit + + _run_command_finalizer(command, route, root, paths) + + fingerprint_path = root / ".pdd" / "meta" / "sample_python.json" + assert fingerprint_path.exists(), f"{command} returned without a fingerprint" + payload = json.loads(fingerprint_path.read_text(encoding="utf-8")) + assert payload["prompt_hash"], f"{command} wrote a null prompt hash" + + with patch( + "pdd.sync_determine_operation.get_pdd_file_paths", + return_value=paths, + ): + decision = sync_determine_operation( + "sample", + "python", + target_coverage=90.0, + log_mode=True, + skip_tests=True, + skip_verify=True, + ) + + assert decision.operation in {"nothing", "all_synced"}, ( + f"{command} reported success but the touched unit is still stale: " + f"{decision.operation} ({decision.reason})" + ) + + +def test_example_from_parent_cwd_has_no_null_hashes_or_wrong_root( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Combined regressions for #1305 and #1211/#1290.""" + parent = tmp_path / "parent" + subproject = parent / "extensions" / "app" + parent.mkdir() + (subproject / "prompts").mkdir(parents=True) + (subproject / "src").mkdir() + (subproject / "examples").mkdir() + (subproject / "tests").mkdir() + (subproject / ".pddrc").write_text( + 'version: "1.0"\ncontexts:\n default:\n defaults:\n' + ' generate_output_path: "src/"\n' + ' example_output_path: "examples/"\n' + ' test_output_path: "tests/"\n', + encoding="utf-8", + ) + paths = { + "prompt": subproject / "prompts" / "sample_python.prompt", + "code": subproject / "src" / "sample.py", + "example": subproject / "examples" / "sample_example.py", + "test": subproject / "tests" / "test_sample.py", + } + paths["test_files"] = [paths["test"]] + paths["prompt"].write_text("% Goal\nNested sample.\n", encoding="utf-8") + paths["code"].write_text("VALUE = 1\n", encoding="utf-8") + paths["example"].write_text("print(1)\n", encoding="utf-8") + paths["test"].write_text("def test_value(): assert True\n", encoding="utf-8") + monkeypatch.chdir(parent) + + _run_decorated_finalizer("example", paths) + + fingerprint_path = subproject / ".pdd" / "meta" / "sample_python.json" + assert fingerprint_path.exists() + assert not (parent / ".pdd" / "meta" / "sample_python.json").exists() + payload = json.loads(fingerprint_path.read_text(encoding="utf-8")) + expected = calculate_current_hashes(paths) + assert payload["prompt_hash"] == expected["prompt_hash"] + assert payload["code_hash"] == expected["code_hash"] + assert payload["example_hash"] == expected["example_hash"] + assert payload["test_hash"] == expected["test_hash"] + + fingerprint = read_fingerprint("sample", "python", paths=paths) + assert fingerprint is not None + with patch( + "pdd.sync_determine_operation.get_pdd_file_paths", + return_value=paths, + ): + decision = sync_determine_operation( + "sample", + "python", + 90.0, + log_mode=True, + skip_tests=True, + skip_verify=True, + ) + assert decision.operation in {"nothing", "all_synced"} + + +@pytest.mark.parametrize( + ("relative_path", "required_text"), + ( + ( + "pdd/commands/generate.py", + '@log_operation(operation="generate", clears_run_report=True, updates_fingerprint=True)', + ), + ( + "pdd/commands/generate.py", + '@log_operation(operation="example", clears_run_report=True, updates_fingerprint=True)', + ), + ( + "pdd/commands/fix.py", + '@log_operation(operation="fix", clears_run_report=True, updates_fingerprint=True)', + ), + ("pdd/update_main.py", "_finalize_single_file_fingerprint("), + ("pdd/auto_deps_main.py", "save_fingerprint("), + ("pdd/sync_orchestration.py", "FingerprintTransaction("), + ("pdd/ci_drift_heal.py", "save_fingerprint("), + ), +) +def test_mutating_command_is_registered_with_shared_finalization_route( + relative_path: str, + required_text: str, +) -> None: + """Keep the executable command registry aligned with the property harness.""" + repo_root = Path(__file__).parents[1] + source = (repo_root / relative_path).read_text(encoding="utf-8") + assert required_text in source diff --git a/tests/test_fingerprint_transaction.py b/tests/test_fingerprint_transaction.py new file mode 100644 index 0000000000..98ecf6c8ed --- /dev/null +++ b/tests/test_fingerprint_transaction.py @@ -0,0 +1,373 @@ +"""Regression coverage for transactional fingerprint finalization (#1926).""" +from __future__ import annotations + +import json +from pathlib import Path +from unittest.mock import patch + +import pytest + +from pdd.fingerprint_transaction import ( + FingerprintFinalizeError, + FingerprintTransaction, +) +from pdd.operation_log import save_fingerprint +from pdd.sync_determine_operation import calculate_current_hashes + + +def _unit(tmp_path: Path) -> tuple[dict[str, Path], Path]: + root = tmp_path / "project" + (root / "prompts").mkdir(parents=True) + (root / "pdd").mkdir() + (root / "tests").mkdir() + (root / ".pdd" / "meta").mkdir(parents=True) + (root / ".pddrc").write_text("contexts: {}\n", encoding="utf-8") + paths = { + "prompt": root / "prompts" / "sample_python.prompt", + "code": root / "pdd" / "sample.py", + "test": root / "tests" / "test_sample.py", + } + paths["prompt"].write_text("% Goal\nCreate sample.\n", encoding="utf-8") + paths["code"].write_text("def sample():\n return 1\n", encoding="utf-8") + paths["test"].write_text("def test_sample():\n assert True\n", encoding="utf-8") + return paths, root + + +def test_clean_exit_writes_complete_fingerprint(tmp_path: Path) -> None: + paths, root = _unit(tmp_path) + + with FingerprintTransaction("sample", "python", "generate", paths): + pass + + fingerprint_path = root / ".pdd" / "meta" / "sample_python.json" + data = json.loads(fingerprint_path.read_text(encoding="utf-8")) + expected = calculate_current_hashes(paths) + assert data["command"] == "generate" + assert data["prompt_hash"] == expected["prompt_hash"] + assert data["code_hash"] == expected["code_hash"] + assert data["test_hash"] == expected["test_hash"] + assert data["prompt_hash"] is not None + + +def test_body_exception_preserves_existing_fingerprint(tmp_path: Path) -> None: + paths, root = _unit(tmp_path) + fingerprint_path = root / ".pdd" / "meta" / "sample_python.json" + fingerprint_path.write_text('{"sentinel": true}\n', encoding="utf-8") + + with pytest.raises(ValueError, match="body failed"): + with FingerprintTransaction("sample", "python", "generate", paths): + raise ValueError("body failed") + + assert json.loads(fingerprint_path.read_text(encoding="utf-8")) == { + "sentinel": True + } + + +def test_skip_is_idempotent_and_does_not_write(tmp_path: Path) -> None: + paths, root = _unit(tmp_path) + transaction = FingerprintTransaction("sample", "python", "update", paths) + + with transaction: + transaction.skip("dry-run") + transaction.skip("dry-run") + + assert not (root / ".pdd" / "meta" / "sample_python.json").exists() + + +def test_null_prompt_hash_is_hard_failure_and_preserves_previous_file( + tmp_path: Path, +) -> None: + paths, root = _unit(tmp_path) + paths["prompt"].unlink() + fingerprint_path = root / ".pdd" / "meta" / "sample_python.json" + fingerprint_path.write_text('{"old": "state"}\n', encoding="utf-8") + + with pytest.raises(FingerprintFinalizeError) as raised: + with FingerprintTransaction("sample", "python", "fix", paths): + pass + + message = str(raised.value) + assert "[fix]" in message + assert str(fingerprint_path) in message + assert "prompt_hash is null" in message + assert json.loads(fingerprint_path.read_text(encoding="utf-8")) == { + "old": "state" + } + + +def test_existing_non_prompt_artifact_cannot_have_null_hash( + tmp_path: Path, +) -> None: + paths, _root = _unit(tmp_path) + hashes = calculate_current_hashes(paths) + hashes["code_hash"] = None + + with patch( + "pdd.fingerprint_transaction.calculate_current_hashes", + return_value=hashes, + ): + with pytest.raises(FingerprintFinalizeError, match="code_hash is null"): + with FingerprintTransaction("sample", "python", "generate", paths): + pass + + +def test_atomic_write_failure_has_context_and_leaves_destination_intact( + tmp_path: Path, +) -> None: + paths, root = _unit(tmp_path) + fingerprint_path = root / ".pdd" / "meta" / "sample_python.json" + fingerprint_path.write_text('{"old": true}\n', encoding="utf-8") + + with patch( + "pdd.fingerprint_transaction.atomic_write_json", + side_effect=OSError("disk full"), + ): + with pytest.raises(FingerprintFinalizeError) as raised: + with FingerprintTransaction("sample", "python", "example", paths): + pass + + assert "[example]" in str(raised.value) + assert str(fingerprint_path) in str(raised.value) + assert "disk full" in str(raised.value) + assert json.loads(fingerprint_path.read_text(encoding="utf-8")) == { + "old": True + } + + +def test_explicit_paths_eagerly_anchor_nested_subproject( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + parent = tmp_path / "parent" + nested = parent / "nested" + (nested / "prompts").mkdir(parents=True) + (nested / "src").mkdir() + (nested / ".pddrc").write_text("contexts: {}\n", encoding="utf-8") + prompt = nested / "prompts" / "sample_python.prompt" + code = nested / "src" / "sample.py" + prompt.write_text("% Goal\nNested.\n", encoding="utf-8") + code.write_text("VALUE = 1\n", encoding="utf-8") + monkeypatch.chdir(parent) + + transaction = FingerprintTransaction( + "sample", + "python", + "update", + {"prompt": prompt, "code": code}, + ) + assert transaction.fingerprint_path == ( + nested / ".pdd" / "meta" / "sample_python.json" + ) + with transaction: + pass + + assert transaction.fingerprint_path.exists() + assert not (parent / ".pdd" / "meta" / "sample_python.json").exists() + + +def test_include_override_controls_hash_and_persisted_graph(tmp_path: Path) -> None: + paths, root = _unit(tmp_path) + dependency = root / "schema.md" + dependency.write_text("contract-v1\n", encoding="utf-8") + override = {str(dependency): "pre-captured-hash"} + expected = calculate_current_hashes(paths, stored_include_deps=override) + + transaction = FingerprintTransaction( + "sample", + "python", + "auto-deps", + paths, + ) + transaction.set_include_deps_override(override) + with transaction: + pass + + data = json.loads( + (root / ".pdd" / "meta" / "sample_python.json").read_text( + encoding="utf-8" + ) + ) + assert data["prompt_hash"] == expected["prompt_hash"] + assert data["include_deps"] == override + + +def test_atomic_state_uses_same_payload_builder_without_early_write( + tmp_path: Path, +) -> None: + paths, root = _unit(tmp_path) + + class Buffer: + payload = None + path = None + operation = None + + def set_fingerprint(self, payload, path, *, operation=None): + self.payload = payload + self.path = path + self.operation = operation + + buffer = Buffer() + with FingerprintTransaction( + "sample", + "python", + "test", + paths, + atomic_state=buffer, + ): + pass + + destination = root / ".pdd" / "meta" / "sample_python.json" + assert not destination.exists() + assert buffer.path == destination + assert buffer.operation == "test" + assert buffer.payload["prompt_hash"] is not None + + +def test_outer_atomic_state_rolls_back_fingerprint_if_run_report_commit_fails( + tmp_path: Path, +) -> None: + """The buffered sync path must not expose half of its state pair.""" + from pdd.sync_orchestration import AtomicStateUpdate + + meta = tmp_path / ".pdd" / "meta" + meta.mkdir(parents=True) + fingerprint_path = meta / "sample_python.json" + run_report_path = meta / "sample_python_run.json" + old_fingerprint = b'{"old": "fingerprint"}\n' + old_run_report = b'{"old": "run-report"}\n' + fingerprint_path.write_bytes(old_fingerprint) + run_report_path.write_bytes(old_run_report) + + state = AtomicStateUpdate("sample", "python") + real_atomic_write = state._atomic_write + write_count = 0 + + def fail_second_write(data, target_path): + nonlocal write_count + write_count += 1 + if write_count == 2: + raise OSError("run-report disk failure") + real_atomic_write(data, target_path) + + state._atomic_write = fail_second_write + with pytest.raises(FingerprintFinalizeError, match="atomic state commit failed"): + with state: + state.set_fingerprint( + {"prompt_hash": "new"}, + fingerprint_path, + operation="test", + ) + state.set_run_report({"exit_code": 0}, run_report_path) + + assert fingerprint_path.read_bytes() == old_fingerprint + assert run_report_path.read_bytes() == old_run_report + + +def test_public_save_wrapper_propagates_transaction_failure(tmp_path: Path) -> None: + paths, _root = _unit(tmp_path) + paths["prompt"].unlink() + + with pytest.raises(FingerprintFinalizeError, match="prompt_hash is null"): + save_fingerprint("sample", "python", "generate", paths=paths) + + +def test_path_normalization_handles_strings_and_null_test_list( + tmp_path: Path, +) -> None: + paths, root = _unit(tmp_path) + + with FingerprintTransaction( + "sample", + "python", + "generate", + paths={"prompt": str(paths["prompt"]), "test_files": None}, + ): + pass + + payload = json.loads( + (root / ".pdd" / "meta" / "sample_python.json").read_text( + encoding="utf-8" + ) + ) + assert payload["prompt_hash"] is not None + assert payload["test_files"] == {} + + +def test_implicit_path_discovery_and_failure_are_typed(tmp_path: Path) -> None: + paths, _root = _unit(tmp_path) + with patch( + "pdd.fingerprint_transaction.get_pdd_file_paths", + return_value=paths, + ) as discover: + with FingerprintTransaction("sample", "python", "generate"): + pass + discover.assert_called_once_with("sample", "python") + + with patch( + "pdd.fingerprint_transaction.get_pdd_file_paths", + side_effect=OSError("cannot discover project"), + ): + with pytest.raises( + FingerprintFinalizeError, + match="path resolution failed: cannot discover project", + ) as raised: + FingerprintTransaction("nested/sample", "python", "generate") + assert raised.value.fingerprint_path == Path( + ".pdd/meta/nested_sample_python.json" + ) + + +def test_existing_test_file_requires_test_files_hash(tmp_path: Path) -> None: + paths, _root = _unit(tmp_path) + paths["test_files"] = [paths["test"]] + hashes = calculate_current_hashes(paths) + hashes["test_files"] = {} + + with patch( + "pdd.fingerprint_transaction.calculate_current_hashes", + return_value=hashes, + ): + with pytest.raises( + FingerprintFinalizeError, + match="test_files hash is null", + ): + with FingerprintTransaction("sample", "python", "test", paths): + pass + + +def test_atomic_state_requires_fingerprint_setter(tmp_path: Path) -> None: + paths, _root = _unit(tmp_path) + + with pytest.raises( + FingerprintFinalizeError, + match="atomic_state does not provide set_fingerprint", + ): + with FingerprintTransaction( + "sample", + "python", + "sync", + paths, + atomic_state=object(), + ): + pass + + +def test_fingerprint_payload_has_one_authoritative_constructor() -> None: + """Future mutating paths must delegate instead of reimplementing writes.""" + package_root = Path(__file__).parents[1] / "pdd" + constructor_owners = [] + serialized_write_owners = [] + for path in package_root.rglob("*.py"): + source = path.read_text(encoding="utf-8") + if "Fingerprint(" in source: + constructor_owners.append(path.relative_to(package_root).as_posix()) + if "json.dump(asdict(fingerprint)" in source: + serialized_write_owners.append(path.relative_to(package_root).as_posix()) + + # read_fingerprint reconstructs the persisted dataclass; only the + # transaction is allowed to construct a new write payload. + assert sorted(constructor_owners) == [ + "fingerprint_transaction.py", + "sync_determine_operation.py", + ] + assert serialized_write_owners == [] diff --git a/tests/test_issue_1714_sync_stall.py b/tests/test_issue_1714_sync_stall.py index 305665d4b6..5380608c0d 100644 --- a/tests/test_issue_1714_sync_stall.py +++ b/tests/test_issue_1714_sync_stall.py @@ -1523,7 +1523,7 @@ def test_prd_sync_run_agentic_task_must_pass_explicit_timeout( "model": "anthropic", "error": "", }), \ - patch("pdd.operation_log.infer_module_identity", return_value=(None, None)), \ + patch("pdd.update_main._finalize_single_file_fingerprint"), \ patch("pdd.architecture_registry.find_architecture_for_project", return_value=[arch_file]), \ patch("pdd.architecture_sync.update_architecture_from_prompt", @@ -1787,4 +1787,3 @@ def test_e2e_test_modules_importable(self): ] for mod_name in e2e_modules: importlib.import_module(mod_name) - diff --git a/tests/test_metadata_sync.py b/tests/test_metadata_sync.py index b0521f40d1..0f0238e13f 100644 --- a/tests/test_metadata_sync.py +++ b/tests/test_metadata_sync.py @@ -579,7 +579,7 @@ def test_run_report_skipped_when_identity_cannot_be_inferred(tmp_path: Path) -> result = run_metadata_sync(ws["prompt_path"], dry_run=False) assert result.stages["run_report"].status == "skipped" mock_clear.assert_not_called() - assert result.stages["fingerprint"].status == "skipped" + assert result.stages["fingerprint"].status == "failed" mock_save.assert_not_called() diff --git a/tests/test_operation_log.py b/tests/test_operation_log.py index 2dc9fa7942..d1d44768df 100644 --- a/tests/test_operation_log.py +++ b/tests/test_operation_log.py @@ -9,6 +9,7 @@ # Import the module under test from pdd import operation_log +from pdd.fingerprint_transaction import FingerprintFinalizeError # -------------------------------------------------------------------------------- # TEST PLAN @@ -235,7 +236,10 @@ def test_load_operation_log_compatibility(temp_pdd_env): def test_save_fingerprint(temp_pdd_env): """Test saving fingerprint state in Fingerprint dataclass format.""" basename, lang = "state", "go" - paths = {"prompt": Path("prompts/state_go.prompt")} + prompt_path = Path(temp_pdd_env).parents[1] / "prompts" / "state_go.prompt" + prompt_path.parent.mkdir() + prompt_path.write_text("% State\n", encoding="utf-8") + paths = {"prompt": prompt_path} operation_log.save_fingerprint(basename, lang, "op1", paths, 0.5, "gpt-4") @@ -299,7 +303,9 @@ def test_log_operation_decorator_success(temp_pdd_env): def my_command(prompt_file: str): return {"status": "ok"}, 0.15, "gpt-3.5" - prompt_path = "prompts/feat_logic_python.prompt" + prompt_path = Path(temp_pdd_env).parents[1] / "prompts" / "feat_logic_python.prompt" + prompt_path.parent.mkdir() + prompt_path.write_text("% Feature logic\n", encoding="utf-8") # Run result = my_command(prompt_file=prompt_path) @@ -398,7 +404,9 @@ def test_log_operation_decorator_failure_preserves_run_report(temp_pdd_env): def failing_example(prompt_file: str): raise RuntimeError("generation failed") - prompt_path = f"prompts/{basename}_{lang}.prompt" + prompt_path = Path(temp_pdd_env).parents[1] / "prompts" / f"{basename}_{lang}.prompt" + prompt_path.parent.mkdir() + prompt_path.write_text("% Example\n", encoding="utf-8") with pytest.raises(RuntimeError, match="generation failed"): failing_example(prompt_file=prompt_path) @@ -471,8 +479,10 @@ def test_log_operation_decorator_success_clears_run_report(temp_pdd_env): def ok_example(prompt_file: str): return "ok", 0.0, "mock" - prompt_path = f"prompts/{basename}_{lang}.prompt" - ok_example(prompt_file=prompt_path) + prompt_path = Path(temp_pdd_env).parents[1] / "prompts" / f"{basename}_{lang}.prompt" + prompt_path.parent.mkdir() + prompt_path.write_text("% Example\n", encoding="utf-8") + ok_example(prompt_file=str(prompt_path)) assert not rr_path.exists(), ( "clears_run_report must remove the stale run report on success" @@ -600,13 +610,16 @@ def test_fingerprint_path_extension_consistency(tmp_path): # Patch module to use temp directory with patch("pdd.operation_log.META_DIR", str(meta_dir)): + prompt_path = tmp_path / "prompts" / "test.prompt" + prompt_path.parent.mkdir() + prompt_path.write_text("% Test\n", encoding="utf-8") # Write a fingerprint using operation_log save_fingerprint( basename=basename, language=language, operation="test_operation", - paths={"prompt": Path("prompts/test.prompt")}, + paths={"prompt": prompt_path}, cost=0.123, model="test-model" ) @@ -643,11 +656,15 @@ def test_fingerprint_format_compatibility(tmp_path): with patch("pdd.operation_log.META_DIR", str(meta_dir)), \ patch("pdd.sync_determine_operation.get_meta_dir", return_value=meta_dir): + prompt_path = tmp_path / "prompts" / f"{basename}_{language}.prompt" + prompt_path.parent.mkdir() + prompt_path.write_text("% Format\n", encoding="utf-8") + save_fingerprint( basename=basename, language=language, operation="test_op", - paths={}, + paths={"prompt": prompt_path}, cost=0.1, model="test" ) @@ -848,7 +865,7 @@ def test_save_fingerprint_resolves_paths_when_none_issue_983(temp_pdd_env, tmp_p mock_paths = {"prompt": prompt_file, "code": code_file} with patch( - "pdd.sync_determine_operation.get_pdd_file_paths", return_value=mock_paths + "pdd.fingerprint_transaction.get_pdd_file_paths", return_value=mock_paths ), patch( "pdd.sync_determine_operation.get_meta_dir", return_value=Path(temp_pdd_env), @@ -890,7 +907,7 @@ def test_save_fingerprint_skips_resolution_when_paths_provided_issue_983(temp_pd explicit_paths = {"prompt": prompt_file} with patch( - "pdd.sync_determine_operation.get_pdd_file_paths" + "pdd.fingerprint_transaction.get_pdd_file_paths" ) as mock_get_paths: operation_log.save_fingerprint( "mymod", "python", operation="generate", paths=explicit_paths @@ -899,26 +916,23 @@ def test_save_fingerprint_skips_resolution_when_paths_provided_issue_983(temp_pd mock_get_paths.assert_not_called() -def test_save_fingerprint_warns_on_path_resolution_failure_issue_983(temp_pdd_env): +def test_save_fingerprint_fails_on_path_resolution_failure_issue_983(temp_pdd_env): """ - Issue #983: If get_pdd_file_paths raises a recoverable error during - path resolution, save_fingerprint should warn and produce null hashes - (graceful degradation) rather than crashing. + Issue #1926: path-resolution failure must not produce a null-hash + fingerprint or report command success. """ with patch( - "pdd.sync_determine_operation.get_pdd_file_paths", + "pdd.fingerprint_transaction.get_pdd_file_paths", side_effect=OSError("prompts dir not found"), - ), patch("pdd.operation_log.logger") as mock_logger: - # Should NOT raise - operation_log.save_fingerprint("badmod", "python", operation="generate") + ): + with pytest.raises(FingerprintFinalizeError) as raised: + operation_log.save_fingerprint("badmod", "python", operation="generate") - mock_logger.warning.assert_called_once() - warning_args = str(mock_logger.warning.call_args) - assert "badmod" in warning_args - assert "python" in warning_args + assert "path resolution failed" in str(raised.value) + assert "badmod_python.json" in str(raised.value) -def test_log_operation_decorator_skips_fingerprint_when_clear_silently_fails(temp_pdd_env): +def test_log_operation_decorator_fails_when_clear_silently_fails(temp_pdd_env): """ Regression for issue #1057: if a stale run report survives clear_run_report(), the decorator must not write a fresh fingerprint next to stale runtime state. @@ -935,10 +949,15 @@ def test_log_operation_decorator_skips_fingerprint_when_clear_silently_fails(tem def successful_command(prompt_file): return "ok", False, 0.0, "model" + prompt_path = Path(temp_pdd_env).parents[1] / "prompts" / "demo_python.prompt" + prompt_path.parent.mkdir() + prompt_path.write_text("% Demo\n", encoding="utf-8") + with patch("pdd.operation_log.os.remove", lambda _path: None), patch( "pdd.operation_log.save_fingerprint" ) as mock_save_fingerprint: - successful_command(prompt_file="prompts/demo_python.prompt") + with pytest.raises(FingerprintFinalizeError, match="run report not cleared"): + successful_command(prompt_file=str(prompt_path)) assert run_report_path.exists() mock_save_fingerprint.assert_not_called() @@ -1149,14 +1168,7 @@ def run_example(prompt_file): def test_log_operation_fallback_coerces_prompt_to_path_issue_1305(tmp_path): - """Issue #1305 (Test 5): when get_pdd_file_paths resolution fails, the - decorator must fall back to {"prompt": Path(prompt_file)} — a COERCED Path, - never a raw string — so prompt_hash is still real and the command does not - crash. - - On buggy code (or a naive raw-string fallback) the prompt value stays a str, - calculate_current_hashes skips it, and prompt_hash is null -> assertion fails. - """ + """A derivative/unregistered prompt still receives a real prompt hash.""" basename, language = "fallbackmod", "python" meta_dir, paths = _setup_pdd_module_files(tmp_path, basename, language) real_prompt_path = paths["prompt"] # absolute path to the real prompt file on disk @@ -1171,23 +1183,16 @@ def run_example(prompt_file): "pdd.sync_determine_operation.get_pdd_file_paths", side_effect=OSError("prompts dir not found"), ): - # Must not raise even though path resolution failed. result = run_example(prompt_file=str(real_prompt_path)) - assert result == ({"status": "ok"}, 0.1, "gpt-4"), "decorator must still return the result" + assert result == ({"status": "ok"}, 0.1, "gpt-4") fp_path = operation_log.get_fingerprint_path(basename, language, project_root=tmp_path) - assert fp_path.exists(), "fallback path must still write a fingerprint file" + assert fp_path.exists() with open(fp_path) as f: fp_data = json.load(f) - - # Fallback coerces the prompt string to a Path, so prompt_hash is real. - assert fp_data["prompt_hash"] is not None, ( - "fallback must pass Path(prompt_file) (not a raw str) so prompt_hash is non-null" - ) - assert _is_hex_sha256(fp_data["prompt_hash"]), ( - f"prompt_hash should be a 64-char hex SHA-256, got: {fp_data['prompt_hash']!r}" - ) + assert fp_data["prompt_hash"] is not None + assert _is_hex_sha256(fp_data["prompt_hash"]) def test_log_operation_example_writes_test_hash_issue_1305(tmp_path): diff --git a/tests/test_update_main.py b/tests/test_update_main.py index 99d8ceea6b..75eca9bcb3 100644 --- a/tests/test_update_main.py +++ b/tests/test_update_main.py @@ -8,6 +8,7 @@ import git from pdd import DEFAULT_STRENGTH +from pdd.fingerprint_transaction import FingerprintFinalizeError from pdd.update_main import ( _finalize_single_file_fingerprint, _included_docs_for_drift_report, @@ -71,7 +72,8 @@ def mock_open_file(): """ Patches the built-in open function so no real file I/O happens. """ - with patch("builtins.open", mock_open()) as mock_file: + with patch("builtins.open", mock_open()) as mock_file, \ + patch("pdd.operation_log.save_fingerprint"): yield mock_file @pytest.fixture @@ -725,8 +727,10 @@ def mock_update_logic( ctx = click.Context(click.Command('update')) ctx.obj = {"strength": 0.5, "temperature": 0.1, "verbose": False, "time": 0.25, "quiet": False} - # Run update_main in repo mode - result = update_main(ctx=ctx, input_prompt_file=None, modified_code_file=None, input_code_file=None, output=None, use_git=False, repo=True) + # The update primitive is mocked and does not actually write its prompt; + # isolate this orchestration test from the real finalizer. + with patch("pdd.operation_log.save_fingerprint"): + result = update_main(ctx=ctx, input_prompt_file=None, modified_code_file=None, input_code_file=None, output=None, use_git=False, repo=True) # Assert that the update function was called for each pair (all 3 marked as changed) assert mock_update_file_pair.call_count == 3 @@ -775,16 +779,17 @@ def mock_update_logic( ctx = click.Context(click.Command('update')) ctx.obj = {"strength": 0.5, "temperature": 0.1, "verbose": False, "time": 0.25, "quiet": False} - result = update_main( - ctx=ctx, - input_prompt_file=None, - modified_code_file=None, - input_code_file=None, - output=None, - use_git=False, - repo=True, - budget=1.0, - ) + with patch("pdd.operation_log.save_fingerprint"): + result = update_main( + ctx=ctx, + input_prompt_file=None, + modified_code_file=None, + input_code_file=None, + output=None, + use_git=False, + repo=True, + budget=1.0, + ) # First two updates run (0.6 + 0.6), then cap is reached and third is skipped. assert mock_update_file_pair.call_count == 2 @@ -2285,6 +2290,7 @@ def test_explicit_params_override_ctx(self): patch("pdd.update_main.resolve_prompt_code_pair", return_value=("/tmp/test.prompt", "/tmp/test.py")), \ patch("pdd.update_main._resolve_update_runtime_config", side_effect=self._fake_resolve_runtime_config), \ + patch("pdd.update_main._finalize_single_file_fingerprint"), \ patch("builtins.open", mock_open(read_data="def foo(): pass\n")): update_main( ctx=ctx, @@ -2324,6 +2330,7 @@ def test_ctx_values_used_when_params_none(self): patch("pdd.update_main.resolve_prompt_code_pair", return_value=("/tmp/test.prompt", "/tmp/test.py")), \ patch("pdd.update_main._resolve_update_runtime_config", side_effect=self._fake_resolve_runtime_config), \ + patch("pdd.update_main._finalize_single_file_fingerprint"), \ patch("builtins.open", mock_open(read_data="def foo(): pass\n")): update_main( ctx=ctx, @@ -3076,10 +3083,9 @@ def test_default_single_file_update_writes_fresh_fingerprint( kwargs = mock_save_fp.call_args.kwargs assert mock_save_fp.call_args.args == ("modified_code", "python") assert kwargs["operation"] == "update" - assert kwargs["paths"] == { - "prompt": Path(str(derived_prompt)), - "code": Path(str(code_file)), - } + assert kwargs["paths"]["prompt"] == Path(str(derived_prompt)) + assert kwargs["paths"]["code"] == Path(str(code_file)) + assert {"prompt", "code", "example", "test"}.issubset(kwargs["paths"]) @patch("pdd.update_main.resolve_prompt_code_pair") @@ -3218,7 +3224,7 @@ def test_default_single_file_update_clears_stale_run_report( assert (meta_dir / "foo_python.json").exists() -def test_finalize_single_file_fingerprint_skips_save_when_run_report_survives_clear( +def test_finalize_single_file_fingerprint_fails_when_run_report_survives_clear( tmp_path, monkeypatch, capsys, @@ -3254,15 +3260,16 @@ def test_finalize_single_file_fingerprint_skips_save_when_run_report_survives_cl import pdd.operation_log as ol monkeypatch.setattr(ol.os, "remove", lambda *a, **kw: None) - _finalize_single_file_fingerprint( - prompt_path=prompt_path, - code_path=code_path, - sync_metadata=False, - dry_run=False, - quiet=False, - cost=0.0, - model="test-model", - ) + with pytest.raises(FingerprintFinalizeError, match="run report not cleared"): + _finalize_single_file_fingerprint( + prompt_path=prompt_path, + code_path=code_path, + sync_metadata=False, + dry_run=False, + quiet=False, + cost=0.0, + model="test-model", + ) # Acceptance criteria: # - The stale run report still exists (because os.remove was nulled). @@ -3320,15 +3327,16 @@ def test_finalize_single_file_fingerprint_warns_about_stale_run_report_even_when import pdd.operation_log as ol monkeypatch.setattr(ol.os, "remove", lambda *a, **kw: None) - _finalize_single_file_fingerprint( - prompt_path=prompt_path, - code_path=code_path, - sync_metadata=False, - dry_run=False, - quiet=True, # explicit: warning must surface anyway - cost=0.0, - model="test-model", - ) + with pytest.raises(FingerprintFinalizeError, match="run report not cleared"): + _finalize_single_file_fingerprint( + prompt_path=prompt_path, + code_path=code_path, + sync_metadata=False, + dry_run=False, + quiet=True, # explicit: warning must surface anyway + cost=0.0, + model="test-model", + ) assert not (meta_dir / "foo_python.json").exists() # Rich's Console wraps long lines on narrow terminals; normalize @@ -3342,7 +3350,7 @@ def test_finalize_single_file_fingerprint_warns_about_stale_run_report_even_when ) -def test_finalize_single_file_fingerprint_swallows_import_error_for_helpers( +def test_finalize_single_file_fingerprint_wraps_import_error_for_helpers( tmp_path, monkeypatch, capsys, @@ -3384,7 +3392,10 @@ def test_finalize_single_file_fingerprint_swallows_import_error_for_helpers( # statement; the import machinery resolves that via `pdd.operation_log` # in sys.modules. Our stand-in lacks the needed names, so the # `from ... import (a, b, c)` raises ImportError at the `a` lookup. - try: + with pytest.raises( + FingerprintFinalizeError, + match="could not import finalization helpers", + ): _finalize_single_file_fingerprint( prompt_path=prompt_path, code_path=code_path, @@ -3394,12 +3405,6 @@ def test_finalize_single_file_fingerprint_swallows_import_error_for_helpers( cost=0.0, model="test-model", ) - except ImportError as exc: - pytest.fail( - f"_finalize_single_file_fingerprint must NOT propagate an " - f"ImportError out — best-effort metadata cleanup may not " - f"break the caller's successful update tuple. Got: {exc!r}" - ) finally: if real_module is not None: sys.modules["pdd.operation_log"] = real_module @@ -3408,16 +3413,10 @@ def test_finalize_single_file_fingerprint_swallows_import_error_for_helpers( # helpers, so writing a fingerprint without first clearing the stale # run report would defeat the issue #1106 invariant. save_fingerprint_mock.assert_not_called() - # A user-facing warning must surface so the operator knows finalization - # was skipped (it is informational, not status fluff). - captured = " ".join(capsys.readouterr().out.split()) - assert "Could not import finalization helpers" in captured, ( - f"Expected a 'Could not import finalization helpers' warning " - f"when the import wrapping fires; got stdout: {captured!r}" - ) + # The caller receives a typed failure and cannot report the update green. -def test_default_single_file_update_skips_fingerprint_when_identity_unknown( +def test_default_single_file_update_fails_when_identity_unknown( mock_ctx, minimal_input_files, mock_construct_paths, @@ -3431,20 +3430,21 @@ def test_default_single_file_update_skips_fingerprint_when_identity_unknown( with patch("pdd.update_main.get_available_agents", return_value=[]), \ patch("pdd.operation_log.infer_module_identity", return_value=(None, None)), \ patch("pdd.operation_log.save_fingerprint") as mock_save_fp: - result = update_main( - ctx=mock_ctx, - input_prompt_file="updated_prompt.prompt", - modified_code_file=minimal_input_files["modified_code_file"], - input_code_file=minimal_input_files["input_code_file"], - output="updated_prompt.prompt", - use_git=False, - ) + with pytest.raises(click.exceptions.Exit) as raised: + update_main( + ctx=mock_ctx, + input_prompt_file="updated_prompt.prompt", + modified_code_file=minimal_input_files["modified_code_file"], + input_code_file=minimal_input_files["input_code_file"], + output="updated_prompt.prompt", + use_git=False, + ) - assert result is not None + assert raised.value.exit_code == 1 mock_save_fp.assert_not_called() -def test_default_single_file_update_swallows_fingerprint_save_failure( +def test_default_single_file_update_fails_on_fingerprint_save_failure( mock_ctx, minimal_input_files, mock_construct_paths, @@ -3458,16 +3458,17 @@ def test_default_single_file_update_swallows_fingerprint_save_failure( with patch("pdd.update_main.get_available_agents", return_value=[]), \ patch("pdd.operation_log.infer_module_identity", return_value=("mod", "python")), \ patch("pdd.operation_log.save_fingerprint", side_effect=OSError("disk full")): - result = update_main( - ctx=mock_ctx, - input_prompt_file="updated_prompt.prompt", - modified_code_file=minimal_input_files["modified_code_file"], - input_code_file=minimal_input_files["input_code_file"], - output="updated_prompt.prompt", - use_git=False, - ) + with pytest.raises(click.exceptions.Exit) as raised: + update_main( + ctx=mock_ctx, + input_prompt_file="updated_prompt.prompt", + modified_code_file=minimal_input_files["modified_code_file"], + input_code_file=minimal_input_files["input_code_file"], + output="updated_prompt.prompt", + use_git=False, + ) - assert result == ("updated prompt text", 0.123456, "test-model") + assert raised.value.exit_code == 1 def test_default_single_file_update_skips_fingerprint_when_output_redirected( @@ -3846,7 +3847,7 @@ def _update(prompt_file, code_file, ctx, repo, simple=False, strength=None, temp @patch("pdd.update_main.get_git_changed_files", return_value=set()) @patch("pdd.update_main.update_file_pair") @patch("pdd.pddrc_initializer.ensure_pddrc_for_scan") -def test_repo_mode_clear_run_report_failure_warns_and_continues( +def test_repo_mode_clear_run_report_failure_is_hard_failure( mock_pddrc, mock_update_file_pair, mock_git_changed, @@ -3882,27 +3883,24 @@ def _update(prompt_file, code_file, ctx, repo, simple=False, strength=None, temp # quiet=False so the warning is emitted ctx.obj = {"strength": 0.5, "temperature": 0.1, "verbose": False, "time": 0.25, "quiet": False} - result = update_main( - ctx=ctx, - input_prompt_file=None, - modified_code_file=None, - input_code_file=None, - output=None, - use_git=False, - repo=True, - sync_metadata=False, - ) + with pytest.raises(click.exceptions.Exit) as raised: + update_main( + ctx=ctx, + input_prompt_file=None, + modified_code_file=None, + input_code_file=None, + output=None, + use_git=False, + repo=True, + sync_metadata=False, + ) - assert result is not None + assert raised.value.exit_code == 1 assert mock_sync.call_count == 0 - # clear_run_report attempted for each successful pair - assert mock_clear_rr.call_count == mock_update_file_pair.call_count - # save_fingerprint still called per pair despite clear failure - assert mock_save_fp.call_count == mock_update_file_pair.call_count - assert mock_save_fp.call_count >= 1 - # Warning surfaced to the user + assert mock_clear_rr.call_count == 1 + assert mock_save_fp.call_count == 0 out = capsys.readouterr().out - assert "Run report clear failed" in out + assert "Fingerprint finalization failed" in out assert "disk full" in out @@ -3911,7 +3909,7 @@ def _update(prompt_file, code_file, ctx, repo, simple=False, strength=None, temp @patch("pdd.update_main.get_git_changed_files", return_value=set()) @patch("pdd.update_main.update_file_pair") @patch("pdd.pddrc_initializer.ensure_pddrc_for_scan") -def test_repo_mode_clear_run_report_silent_unlink_failure_warns( +def test_repo_mode_clear_run_report_silent_unlink_failure_is_hard_failure( mock_pddrc, mock_update_file_pair, mock_git_changed, @@ -3958,22 +3956,23 @@ def _update(prompt_file, code_file, ctx, repo, simple=False, strength=None, temp # quiet=False so the defensive warning is emitted ctx.obj = {"strength": 0.5, "temperature": 0.1, "verbose": False, "time": 0.25, "quiet": False} - result = update_main( - ctx=ctx, - input_prompt_file=None, - modified_code_file=None, - input_code_file=None, - output=None, - use_git=False, - repo=True, - sync_metadata=False, - ) + with pytest.raises(click.exceptions.Exit) as raised: + update_main( + ctx=ctx, + input_prompt_file=None, + modified_code_file=None, + input_code_file=None, + output=None, + use_git=False, + repo=True, + sync_metadata=False, + ) - assert result is not None + assert raised.value.exit_code == 1 assert mock_sync.call_count == 0 # clear_run_report attempted for each successful pair, but it silently # did nothing (no exception raised, no file removed). - assert mock_clear_rr.call_count == mock_update_file_pair.call_count + assert mock_clear_rr.call_count == 1 # save_fingerprint must be SKIPPED when the stale run report remains, # so we don't claim finalized metadata while runtime verification still # describes the pre-update files (issue #1057). @@ -3981,8 +3980,9 @@ def _update(prompt_file, code_file, ctx, repo, simple=False, strength=None, temp # Defensive warning surfaced to the user because the report file still # exists after clear_run_report returned. out = capsys.readouterr().out - assert "Run report clear failed" in out - assert "still exists after" in out + normalized_out = " ".join(out.split()) + assert "still exists after clear" in normalized_out + assert "Fingerprint finalization failed" in normalized_out # The file is indeed still on disk (the whole point of this regression). assert stale_report.exists() @@ -4421,6 +4421,7 @@ def _run_prd_sync_update(repo_root, ctx, sync_metadata=False): patch("pdd.update_main.git.Repo") as mock_repo, patch("pdd.update_main.os.getcwd", return_value=str(repo_root)), patch("pdd.pddrc_initializer.ensure_pddrc_for_scan"), + patch("pdd.operation_log.save_fingerprint"), ): mock_repo.return_value.working_tree_dir = str(repo_root) return update_main( From c7fc6a6d514cc310dfc774f77fb8bf9584de1736 Mon Sep 17 00:00:00 2001 From: Serhan Date: Fri, 10 Jul 2026 18:13:54 -0700 Subject: [PATCH 4/5] Declare maintenance finalization dependency --- architecture.json | 3 ++- pdd/prompts/commands/maintenance_python.prompt | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/architecture.json b/architecture.json index ff5a09dcdb..4ecf37396d 100644 --- a/architecture.json +++ b/architecture.json @@ -8597,7 +8597,8 @@ "description": "Contains maintenance and setup commands for the PDD CLI, including single-module sync, no-argument project-wide sync, GitHub issue sync dispatch, prompt-to-architecture sync, auto-deps, and setup.", "dependencies": [ "auto_deps_main_python.prompt", - "architecture_sync_python.prompt" + "architecture_sync_python.prompt", + "fingerprint_transaction_python.prompt" ], "priority": 70, "filename": "commands/maintenance_python.prompt", diff --git a/pdd/prompts/commands/maintenance_python.prompt b/pdd/prompts/commands/maintenance_python.prompt index 6326e5a8b0..0157de6b94 100644 --- a/pdd/prompts/commands/maintenance_python.prompt +++ b/pdd/prompts/commands/maintenance_python.prompt @@ -16,6 +16,7 @@ auto_deps_main_python.prompt architecture_sync_python.prompt +fingerprint_transaction_python.prompt % You are an expert Python engineer. Your goal is to write `pdd/commands/maintenance.py`. From a34b7db88f67ebbfd062bcc28c1c589980849a1a Mon Sep 17 00:00:00 2001 From: Serhan Date: Fri, 10 Jul 2026 18:40:08 -0700 Subject: [PATCH 5/5] Reconcile transactional fingerprint metadata --- .pdd/meta/auto_deps_main_python.json | 23 ++++++++++++++ .pdd/meta/auto_deps_main_python_run.json | 11 +++++++ .pdd/meta/ci_drift_heal_python.json | 14 ++++----- .pdd/meta/ci_drift_heal_python_run.json | 10 +++--- .pdd/meta/fingerprint_transaction_python.json | 18 +++++++++++ .../fingerprint_transaction_python_run.json | 11 +++++++ .pdd/meta/metadata_sync_python.json | 25 ++++++++------- .pdd/meta/metadata_sync_python_run.json | 12 +++---- .pdd/meta/operation_log_python.json | 17 ++++++++++ .pdd/meta/operation_log_python_run.json | 12 +++++++ .pdd/meta/pin_example_hack_python.json | 17 +++++----- .pdd/meta/pin_example_hack_python_run.json | 8 ++--- .pdd/meta/sync_main_python.json | 24 ++++++++++++++ .pdd/meta/sync_main_python_run.json | 11 +++++++ .pdd/meta/sync_orchestration_python.json | 31 ++++++++++--------- .pdd/meta/sync_orchestration_python_run.json | 12 +++---- .pdd/meta/update_main_python.json | 28 +++++++++++++++++ .pdd/meta/update_main_python_run.json | 11 +++++++ architecture.json | 4 +-- pdd/prompts/pin_example_hack_python.prompt | 3 +- 20 files changed, 235 insertions(+), 67 deletions(-) create mode 100644 .pdd/meta/auto_deps_main_python.json create mode 100644 .pdd/meta/auto_deps_main_python_run.json create mode 100644 .pdd/meta/fingerprint_transaction_python.json create mode 100644 .pdd/meta/fingerprint_transaction_python_run.json create mode 100644 .pdd/meta/operation_log_python.json create mode 100644 .pdd/meta/operation_log_python_run.json create mode 100644 .pdd/meta/sync_main_python.json create mode 100644 .pdd/meta/sync_main_python_run.json create mode 100644 .pdd/meta/update_main_python.json create mode 100644 .pdd/meta/update_main_python_run.json diff --git a/.pdd/meta/auto_deps_main_python.json b/.pdd/meta/auto_deps_main_python.json new file mode 100644 index 0000000000..836bde50ad --- /dev/null +++ b/.pdd/meta/auto_deps_main_python.json @@ -0,0 +1,23 @@ +{ + "pdd_version": "0.0.301.dev0", + "timestamp": "2026-07-11T01:12:43.417595+00:00", + "command": "test", + "prompt_hash": "f1c52d413f859c1c765f1947e5b5ef70e5d3d19dffdaacb0b2a56759dff3550f", + "code_hash": "e927e3905403152639f890f0c0d8cf94e0bed766963dcfc49ce089d30c66b05d", + "example_hash": "9b9c36c18d39494b6e31b46f6e4758c25135b6fa3d1672fbb7365844f7d2de04", + "test_hash": "785d7e33ae60fa6a349a8f67dffb412c9c20645a0648e491266ef67841c3e91c", + "test_files": { + "test_auto_deps_main.py": "785d7e33ae60fa6a349a8f67dffb412c9c20645a0648e491266ef67841c3e91c" + }, + "include_deps": { + "README.md": "a01a0c12f49bb49b3e3db5f4a5453ad6621965ae814f357533aeead0160608a6", + "context/ctx_obj_params.prompt": "829e2ab327e37b75afcd0dcf3893dcaec9012a89a1f6e5198ec7cfc0395d2782", + "context/construct_paths_example.py": "412cb260650bcbd628fba647db917a8d32f8191df2e1c32cc22f22c0c11b1740", + "context/insert_includes_example.py": "c304bf4656f75fa26192c92b3eb0422a903d8a1bc82dda3158dc9f184b337873", + "context/operation_log_example.py": "2660def7012d9b79bad3db33cf399ac603cc1c35dff0b064cbdd2638a013de1d", + "context/python_preamble.prompt": "0388ed131bf986f8752e1bc4c81e4da0460cfe2908ec8c60b1314edbab768254", + "pdd/auto_deps_architecture.py": "d20e3525f49abb1c4d4d923ed619e9f0b26cb29cbca12df2f22479b8d3926a44", + "pdd/operation_log.py": "d9cf881b67d99c19b76be717e2d57b952ab008a469db97f65f5b6974b0424aef", + "pdd/validate_prompt_includes.py": "fa97b72a58534e255768634f928f4cfdd2130a7a4d8b296da3eefdf77d47855b" + } +} \ No newline at end of file diff --git a/.pdd/meta/auto_deps_main_python_run.json b/.pdd/meta/auto_deps_main_python_run.json new file mode 100644 index 0000000000..66599d123b --- /dev/null +++ b/.pdd/meta/auto_deps_main_python_run.json @@ -0,0 +1,11 @@ +{ + "timestamp": "2026-07-11T01:38:00.444353+00:00", + "exit_code": 0, + "tests_passed": 316, + "tests_failed": 0, + "coverage": 72.82608695652173, + "test_hash": "785d7e33ae60fa6a349a8f67dffb412c9c20645a0648e491266ef67841c3e91c", + "test_files": { + "test_auto_deps_main.py": "785d7e33ae60fa6a349a8f67dffb412c9c20645a0648e491266ef67841c3e91c" + } +} \ No newline at end of file diff --git a/.pdd/meta/ci_drift_heal_python.json b/.pdd/meta/ci_drift_heal_python.json index 40ec37cd07..1e31d7352c 100644 --- a/.pdd/meta/ci_drift_heal_python.json +++ b/.pdd/meta/ci_drift_heal_python.json @@ -1,13 +1,13 @@ { - "pdd_version": "0.0.296.dev0", - "timestamp": "2026-07-10T19:43:29.185846+00:00", - "command": "fix", - "prompt_hash": "26d9fef09114b1da971d431c3f7876b923c4086199f9d55522aedfa33b9392be", - "code_hash": "680193331b1f6e73d1cfe40b88f06989db27ca1d4324faa2f9e781db3b08fc36", + "pdd_version": "0.0.301.dev0", + "timestamp": "2026-07-11T01:12:43.462145+00:00", + "command": "test", + "prompt_hash": "637777d89005399e63b17f0279b4f67aeef4a02682a0775225c35840ae6601f9", + "code_hash": "f76ca4cd49fe69989f6c57890dfeadf869da0a64ca28e3d487e6576f9418a893", "example_hash": "550e750fb041fb73dc92462d4f266bc999102ddd8e4efd3b40dbab0bb1d0d879", - "test_hash": "03ea52b7e66a6a3ec720417bc19e05ff577d76dc8a381e9f8f83552f6d8ce6f2", + "test_hash": "d7370e26542c3c96d5b448bef9bb588016db28369572ceed551bf636c7a64ef4", "test_files": { - "test_ci_drift_heal.py": "03ea52b7e66a6a3ec720417bc19e05ff577d76dc8a381e9f8f83552f6d8ce6f2", + "test_ci_drift_heal.py": "d7370e26542c3c96d5b448bef9bb588016db28369572ceed551bf636c7a64ef4", "test_ci_drift_heal_e2e.py": "dfcd225cc5a6442ba11e9cd41b79568a57d97b7804ee6ea66f4191aa068478e3" }, "include_deps": { diff --git a/.pdd/meta/ci_drift_heal_python_run.json b/.pdd/meta/ci_drift_heal_python_run.json index 844d8196a5..9a7023963a 100644 --- a/.pdd/meta/ci_drift_heal_python_run.json +++ b/.pdd/meta/ci_drift_heal_python_run.json @@ -1,12 +1,12 @@ { - "timestamp": "2026-07-09T11:33:33.405832+00:00", + "timestamp": "2026-07-11T01:38:00.501020+00:00", "exit_code": 0, - "tests_passed": 195, + "tests_passed": 249, "tests_failed": 0, - "coverage": 83.0, - "test_hash": "04bbe525d057462cb9e669037e210629f850dce9a42b5c08032cbbbb642abdf1", + "coverage": 74.79820627802691, + "test_hash": "d7370e26542c3c96d5b448bef9bb588016db28369572ceed551bf636c7a64ef4", "test_files": { - "test_ci_drift_heal.py": "04bbe525d057462cb9e669037e210629f850dce9a42b5c08032cbbbb642abdf1", + "test_ci_drift_heal.py": "d7370e26542c3c96d5b448bef9bb588016db28369572ceed551bf636c7a64ef4", "test_ci_drift_heal_e2e.py": "dfcd225cc5a6442ba11e9cd41b79568a57d97b7804ee6ea66f4191aa068478e3" } } \ No newline at end of file diff --git a/.pdd/meta/fingerprint_transaction_python.json b/.pdd/meta/fingerprint_transaction_python.json new file mode 100644 index 0000000000..f998585a30 --- /dev/null +++ b/.pdd/meta/fingerprint_transaction_python.json @@ -0,0 +1,18 @@ +{ + "pdd_version": "0.0.301.dev0", + "timestamp": "2026-07-11T01:12:43.506970+00:00", + "command": "test", + "prompt_hash": "f30859a96c97d0b5c370784784b1df7e6957d6254655fce6c49b9e062f79cb64", + "code_hash": "36ab6833444e1775a967d2183315c0244e23815b2cc26ff682a70a627d6aa799", + "example_hash": "fa64f0b0d58a128e5df176a561026ce73cd5a853ea671cbd1863c103e2033ba3", + "test_hash": "d373f17cdf581c39bfe95fd61b3023410b50bcb0738845c2d1b0278c2993d8d9", + "test_files": { + "test_fingerprint_transaction.py": "d373f17cdf581c39bfe95fd61b3023410b50bcb0738845c2d1b0278c2993d8d9" + }, + "include_deps": { + "context/operation_log_example.py": "2660def7012d9b79bad3db33cf399ac603cc1c35dff0b064cbdd2638a013de1d", + "context/python_preamble.prompt": "0388ed131bf986f8752e1bc4c81e4da0460cfe2908ec8c60b1314edbab768254", + "pdd/operation_log.py": "d9cf881b67d99c19b76be717e2d57b952ab008a469db97f65f5b6974b0424aef", + "pdd/sync_determine_operation.py": "e616ac746674a9f1e8be84bf3093d5e1f45ff0175bc3767b758f2d17b2c54132" + } +} \ No newline at end of file diff --git a/.pdd/meta/fingerprint_transaction_python_run.json b/.pdd/meta/fingerprint_transaction_python_run.json new file mode 100644 index 0000000000..0f64ea00c7 --- /dev/null +++ b/.pdd/meta/fingerprint_transaction_python_run.json @@ -0,0 +1,11 @@ +{ + "timestamp": "2026-07-11T01:38:00.554427+00:00", + "exit_code": 0, + "tests_passed": 316, + "tests_failed": 0, + "coverage": 59.30232558139535, + "test_hash": "d373f17cdf581c39bfe95fd61b3023410b50bcb0738845c2d1b0278c2993d8d9", + "test_files": { + "test_fingerprint_transaction.py": "d373f17cdf581c39bfe95fd61b3023410b50bcb0738845c2d1b0278c2993d8d9" + } +} \ No newline at end of file diff --git a/.pdd/meta/metadata_sync_python.json b/.pdd/meta/metadata_sync_python.json index bba992e110..6d54686b1b 100644 --- a/.pdd/meta/metadata_sync_python.json +++ b/.pdd/meta/metadata_sync_python.json @@ -1,19 +1,20 @@ { - "pdd_version": "0.0.237", - "timestamp": "2026-05-16T20:50:00.397700+00:00", - "command": "fix", - "prompt_hash": "18e8e073748217aa9c224bea7c7e8425ea36de28f769b30684cd21033018bfab", - "code_hash": "739ace489ae1bf00a9aa696332dabce993b934047228fe0d10a1ef9145130fb2", + "pdd_version": "0.0.301.dev0", + "timestamp": "2026-07-11T01:12:43.549811+00:00", + "command": "test", + "prompt_hash": "c0d96267e86c7706bcc3f19b25b5e0724ff56b6efb3158c81c8d3c753128422b", + "code_hash": "5d81376096aafa315060d05acd94bd355a391cb5aa3bb73d5622e38b8309fda9", "example_hash": "704453b3386724eef225b61c48f2fe922e1a973ac211c494e1d955dd44bcca11", - "test_hash": "a1bf7b5a8ce6c671aee77fca08edd29eea86e93cce2d02871d0bd25b31591dc3", + "test_hash": "5171a3985d843ae8bf54d35b964f1ae7300974d76de05b9518dbccac8947d9a5", "test_files": { - "test_metadata_sync.py": "a1bf7b5a8ce6c671aee77fca08edd29eea86e93cce2d02871d0bd25b31591dc3" + "test_metadata_sync.py": "5171a3985d843ae8bf54d35b964f1ae7300974d76de05b9518dbccac8947d9a5" }, "include_deps": { "context/python_preamble.prompt": "0388ed131bf986f8752e1bc4c81e4da0460cfe2908ec8c60b1314edbab768254", - "pdd/architecture_registry.py": "f822c4ef9d66da1dc6e6104ad3ca99d822c277423e71bb807966e4fc80f779ed", - "pdd/architecture_sync.py": "89329e32dd4d13e7b095f1b96843e383936bbc5f329e50daf006dbb12af0d54b", - "pdd/operation_log.py": "78b05aadd7656dbabb7cdd113fa84d39d32c64e4cb4322aa2ac2570d390d94ef", - "pdd/sync_determine_operation.py": "dff5e2b0628122c49305106d255e91d57446d7c768d4ae78fa4a09a619383e23" + "pdd/architecture_registry.py": "5722225846b9cc8189b410a78e5714b4f2916d5748271d27f6f738692429c309", + "pdd/architecture_sync.py": "0e0ff74978f49d629f92365723ebbea0e7c70e6d5234ccfdafb1c8e09c110c11", + "pdd/fingerprint_transaction.py": "36ab6833444e1775a967d2183315c0244e23815b2cc26ff682a70a627d6aa799", + "pdd/operation_log.py": "d9cf881b67d99c19b76be717e2d57b952ab008a469db97f65f5b6974b0424aef", + "pdd/sync_determine_operation.py": "e616ac746674a9f1e8be84bf3093d5e1f45ff0175bc3767b758f2d17b2c54132" } -} +} \ No newline at end of file diff --git a/.pdd/meta/metadata_sync_python_run.json b/.pdd/meta/metadata_sync_python_run.json index d7d4e6e49b..fdd635f083 100644 --- a/.pdd/meta/metadata_sync_python_run.json +++ b/.pdd/meta/metadata_sync_python_run.json @@ -1,11 +1,11 @@ { - "timestamp": "2026-05-15T18:09:53.425787+00:00", + "timestamp": "2026-07-11T01:38:00.607673+00:00", "exit_code": 0, - "tests_passed": 45, + "tests_passed": 249, "tests_failed": 0, - "coverage": 100.0, - "test_hash": "a1bf7b5a8ce6c671aee77fca08edd29eea86e93cce2d02871d0bd25b31591dc3", + "coverage": 73.24414715719064, + "test_hash": "5171a3985d843ae8bf54d35b964f1ae7300974d76de05b9518dbccac8947d9a5", "test_files": { - "test_metadata_sync.py": "a1bf7b5a8ce6c671aee77fca08edd29eea86e93cce2d02871d0bd25b31591dc3" + "test_metadata_sync.py": "5171a3985d843ae8bf54d35b964f1ae7300974d76de05b9518dbccac8947d9a5" } -} +} \ No newline at end of file diff --git a/.pdd/meta/operation_log_python.json b/.pdd/meta/operation_log_python.json new file mode 100644 index 0000000000..1d79c61bce --- /dev/null +++ b/.pdd/meta/operation_log_python.json @@ -0,0 +1,17 @@ +{ + "pdd_version": "0.0.301.dev0", + "timestamp": "2026-07-11T01:12:43.592727+00:00", + "command": "test", + "prompt_hash": "1ab7c992671a61321b8492726ddb8d288d320204fca45c472ab84ca64639d265", + "code_hash": "d9cf881b67d99c19b76be717e2d57b952ab008a469db97f65f5b6974b0424aef", + "example_hash": "2660def7012d9b79bad3db33cf399ac603cc1c35dff0b064cbdd2638a013de1d", + "test_hash": "18935609569e46ffec5f864ad77dd3d3ddbaf0927c3ffb31af8d263e4901b871", + "test_files": { + "test_operation_log.py": "18935609569e46ffec5f864ad77dd3d3ddbaf0927c3ffb31af8d263e4901b871", + "test_operation_logging_e2e.py": "1fc74b2e4db145dfe21150f751d4b71a81ab0aaf1cc72e44bbe8b037f671556e" + }, + "include_deps": { + "context/sync_determine_operation_example.py": "225d20ea0210bbf52650a9f424879ff495e574dd2411eb4b86592a6d06cf59a5", + "pdd/fingerprint_transaction.py": "36ab6833444e1775a967d2183315c0244e23815b2cc26ff682a70a627d6aa799" + } +} \ No newline at end of file diff --git a/.pdd/meta/operation_log_python_run.json b/.pdd/meta/operation_log_python_run.json new file mode 100644 index 0000000000..19da354f00 --- /dev/null +++ b/.pdd/meta/operation_log_python_run.json @@ -0,0 +1,12 @@ +{ + "timestamp": "2026-07-11T01:38:00.658906+00:00", + "exit_code": 0, + "tests_passed": 316, + "tests_failed": 0, + "coverage": 77.41935483870968, + "test_hash": "18935609569e46ffec5f864ad77dd3d3ddbaf0927c3ffb31af8d263e4901b871", + "test_files": { + "test_operation_log.py": "18935609569e46ffec5f864ad77dd3d3ddbaf0927c3ffb31af8d263e4901b871", + "test_operation_logging_e2e.py": "1fc74b2e4db145dfe21150f751d4b71a81ab0aaf1cc72e44bbe8b037f671556e" + } +} \ No newline at end of file diff --git a/.pdd/meta/pin_example_hack_python.json b/.pdd/meta/pin_example_hack_python.json index 8f159bc131..9ab177ac02 100644 --- a/.pdd/meta/pin_example_hack_python.json +++ b/.pdd/meta/pin_example_hack_python.json @@ -1,9 +1,9 @@ { - "pdd_version": "0.0.234", - "timestamp": "2026-05-12T02:53:56.323298+00:00", - "command": "fix", - "prompt_hash": "77e8c17f1eb7ed4e1347312a702129bcfd8af95a985ee1ccd718291f83c93fae", - "code_hash": "c90df80316bc8339fcf9d4690ac29e4dc8ae3391e25c26c82e496b87dcd4a788", + "pdd_version": "0.0.301.dev0", + "timestamp": "2026-07-11T01:39:17.476843+00:00", + "command": "test", + "prompt_hash": "9ec4eabd87cb0df043e3549ac3afc2d08345bacfdf88ae0b49e3c32d6a4b8648", + "code_hash": "fd390a78ee241bed49502716076afb07d4081becfca64c2241b0981d91665f55", "example_hash": "34e285dedea6ad0d3f3a5539401cc7c0bf5b2181d954915963fd66cd86e54e3c", "test_hash": "7fbffec72f9941a52b4ba7efd6468330dd2a3235dc53d0bb81168ea6eb6c3a72", "test_files": { @@ -12,13 +12,14 @@ "include_deps": { "context/auto_deps_main_example.py": "9b9c36c18d39494b6e31b46f6e4758c25135b6fa3d1672fbb7365844f7d2de04", "context/cmd_test_main_example.py": "0774984ee29f70407429f1eff4a1b76287715b7ddd6adf2bc149e874c9b844bb", - "context/code_generator_main_example.py": "679590f0913600886d99469d8ece23fbbfe6bea2e626b0d5d2ab5fdcdc079b80", + "context/code_generator_main_example.py": "4b66f24a1e2ac4195255738566b0cccdac0971523a245d3a8b78ad45d9e1a872", "context/context_generator_main_example.py": "77c3de786f31392540ec609d4b3942e98e996ae14c4014f5018cf2b05a560df4", "context/crash_main_example.py": "5c9b909107f74f773ccf7013d556a5812643d2a3b086e79fe07da7ea7877458c", + "context/fingerprint_transaction_example.py": "fa64f0b0d58a128e5df176a561026ce73cd5a853ea671cbd1863c103e2033ba3", "context/fix_main_example.py": "29f88d8bfcc2256ebebe6efcb611f9487eb5ca6d8781286a6dd98689f92bca7a", "context/fix_verification_main_example.py": "6490f398aca6df47b1a95e46a4ecfc0aabe299b5022de5c31ebc436e1bc3e459", "context/get_test_command_example.py": "33b16476536f8c9424d3222ffc4a5cd5ecee6ad0589b2c66a4f3c7f1e265106c", - "context/sync_determine_operation_example.py": "94a333b37d9875bcc9b39e22d4d00de1b196ad2f7e0fa997e6f638e153dbdfad", - "context/update_main_example.py": "b318834c648f25e6141d81f3ecac1d9da592a6bb4740500983f2b07b8a10d902" + "context/sync_determine_operation_example.py": "225d20ea0210bbf52650a9f424879ff495e574dd2411eb4b86592a6d06cf59a5", + "context/update_main_example.py": "9fe01fd19196c2604c91d101e49db849e0e1c9fabafeb2ad831a9c91b2f26416" } } \ No newline at end of file diff --git a/.pdd/meta/pin_example_hack_python_run.json b/.pdd/meta/pin_example_hack_python_run.json index 7234a78c54..95a95080ca 100644 --- a/.pdd/meta/pin_example_hack_python_run.json +++ b/.pdd/meta/pin_example_hack_python_run.json @@ -1,11 +1,11 @@ { - "timestamp": "2026-05-12T02:53:56.323642+00:00", + "timestamp": "2026-07-11T01:39:17.479330+00:00", "exit_code": 0, - "tests_passed": 1, + "tests_passed": 249, "tests_failed": 0, - "coverage": 100.0, + "coverage": 4.582843713278496, "test_hash": "7fbffec72f9941a52b4ba7efd6468330dd2a3235dc53d0bb81168ea6eb6c3a72", "test_files": { "test_pin_example_hack.py": "7fbffec72f9941a52b4ba7efd6468330dd2a3235dc53d0bb81168ea6eb6c3a72" } -} +} \ No newline at end of file diff --git a/.pdd/meta/sync_main_python.json b/.pdd/meta/sync_main_python.json new file mode 100644 index 0000000000..e2b1c84ba2 --- /dev/null +++ b/.pdd/meta/sync_main_python.json @@ -0,0 +1,24 @@ +{ + "pdd_version": "0.0.301.dev0", + "timestamp": "2026-07-11T01:12:43.719224+00:00", + "command": "test", + "prompt_hash": "7c19904ac8c14d783b776b3cde3c02742651e1140c34fa31162a14dc0312d448", + "code_hash": "22ce1c2e258b9a9f87aaa1e8f87a53ee60fa48485650f4e7bbb83f70e45d8b60", + "example_hash": "9a7961a4044a54c46ce6c43c8adbbb83da46f233d9c6a9eaa28302440c0052c0", + "test_hash": "9f617d403a3ba62d0f46451bf63d81485f6fb7c4ec0a56a9b342b4c244a8d330", + "test_files": { + "test_sync_main.py": "9f617d403a3ba62d0f46451bf63d81485f6fb7c4ec0a56a9b342b4c244a8d330" + }, + "include_deps": { + "README.md": "a01a0c12f49bb49b3e3db5f4a5453ad6621965ae814f357533aeead0160608a6", + "context/auth_service_example.py": "4b6d2c8c9e13d1edc0d9805ee0c07657494cbf31dffe2bb0029b72975674a8cf", + "context/construct_paths_example.py": "412cb260650bcbd628fba647db917a8d32f8191df2e1c32cc22f22c0c11b1740", + "context/ctx_obj_params.prompt": "829e2ab327e37b75afcd0dcf3893dcaec9012a89a1f6e5198ec7cfc0395d2782", + "context/python_preamble.prompt": "0388ed131bf986f8752e1bc4c81e4da0460cfe2908ec8c60b1314edbab768254", + "context/sync_main_example.py": "9a7961a4044a54c46ce6c43c8adbbb83da46f233d9c6a9eaa28302440c0052c0", + "context/sync_orchestration_example.py": "cc1313a948946026d418c85a07e5bf0eedeb5915c44d630440e489979755e1f7", + "pdd/agentic_sync_runner.py": "572f865d1bdbddf4827d5f308eb72030c3ed085c4f570284cbda8c961d01af2d", + "pdd/code_generator_main.py": "e83cc9bd8890d032fcf63001b83feb766d632ab08d00e20110e489e0386962b4", + "pdd/compressed_sync_context.py": "661fde0fcc42ce39c1004e47a774a8fd799b789065519cb5688949c35fed4859" + } +} \ No newline at end of file diff --git a/.pdd/meta/sync_main_python_run.json b/.pdd/meta/sync_main_python_run.json new file mode 100644 index 0000000000..1b1b8ff658 --- /dev/null +++ b/.pdd/meta/sync_main_python_run.json @@ -0,0 +1,11 @@ +{ + "timestamp": "2026-07-11T01:38:00.808827+00:00", + "exit_code": 0, + "tests_passed": 316, + "tests_failed": 0, + "coverage": 66.66666666666667, + "test_hash": "9f617d403a3ba62d0f46451bf63d81485f6fb7c4ec0a56a9b342b4c244a8d330", + "test_files": { + "test_sync_main.py": "9f617d403a3ba62d0f46451bf63d81485f6fb7c4ec0a56a9b342b4c244a8d330" + } +} \ No newline at end of file diff --git a/.pdd/meta/sync_orchestration_python.json b/.pdd/meta/sync_orchestration_python.json index 5b52c7dd61..d9bb7c91a7 100644 --- a/.pdd/meta/sync_orchestration_python.json +++ b/.pdd/meta/sync_orchestration_python.json @@ -1,31 +1,32 @@ { - "pdd_version": "0.0.234", - "timestamp": "2026-05-12T02:53:56.362536+00:00", - "command": "fix", - "prompt_hash": "b9b900ce1b0cf9e50b73c4aad750e2231f632beda3c286c0469f21c0dc3a8475", - "code_hash": "7ef19b3e7eafd7abd0f54c9de74b7c4d8be5559d84b2673f1c60cb9313d4842e", + "pdd_version": "0.0.301.dev0", + "timestamp": "2026-07-11T01:12:43.766895+00:00", + "command": "test", + "prompt_hash": "e60633f86a80d67660c9c8f74c0940620c750f67967340193b314283494514c9", + "code_hash": "c480e811aa42454d3cf553148a73ed840dfac54ec13aad239d987e07c1359413", "example_hash": "cc1313a948946026d418c85a07e5bf0eedeb5915c44d630440e489979755e1f7", - "test_hash": "3fce96bc02b851cc3e2d00001b0de3a2436f687c65a53d5e423212ac5c6d2517", + "test_hash": "7a4a23ed7c00408887d4ca9b604cb9715e0ad6e0a9c3e9e1b4df78603b396f41", "test_files": { - "test_sync_orchestration.py": "3fce96bc02b851cc3e2d00001b0de3a2436f687c65a53d5e423212ac5c6d2517" + "test_sync_orchestration.py": "7a4a23ed7c00408887d4ca9b604cb9715e0ad6e0a9c3e9e1b4df78603b396f41" }, "include_deps": { - "docs/whitepaper.md": "25b87d4af53ba35e7fa80b12f6bf09d12e16b8b952341df94cf8b47160886adf", - "README.md": "b572bb7bf64f3d9473845a45b8c942fc66b527bc516c960059bc538e574ae4bc", + "docs/whitepaper.md": "3347db03fa35376024b59d8546efdf0858655746490561b88b710fa6b85017cc", + "README.md": "a01a0c12f49bb49b3e3db5f4a5453ad6621965ae814f357533aeead0160608a6", "context/auto_deps_main_example.py": "9b9c36c18d39494b6e31b46f6e4758c25135b6fa3d1672fbb7365844f7d2de04", "context/cmd_test_main_example.py": "0774984ee29f70407429f1eff4a1b76287715b7ddd6adf2bc149e874c9b844bb", - "context/code_generator_main_example.py": "679590f0913600886d99469d8ece23fbbfe6bea2e626b0d5d2ab5fdcdc079b80", + "context/code_generator_main_example.py": "4b66f24a1e2ac4195255738566b0cccdac0971523a245d3a8b78ad45d9e1a872", "context/context_generator_main_example.py": "77c3de786f31392540ec609d4b3942e98e996ae14c4014f5018cf2b05a560df4", "context/crash_main_example.py": "5c9b909107f74f773ccf7013d556a5812643d2a3b086e79fe07da7ea7877458c", "context/fix_main_example.py": "29f88d8bfcc2256ebebe6efcb611f9487eb5ca6d8781286a6dd98689f92bca7a", "context/fix_verification_main_example.py": "6490f398aca6df47b1a95e46a4ecfc0aabe299b5022de5c31ebc436e1bc3e459", "context/get_extension_example.py": "def9c34c1416259a0a031a996b96aeff5e73a7012adff6af1cc557e74ba97cc8", - "context/get_run_command_example.py": "4d187e844adb255c2738274556f4828728d0d73d84ebf8bda885d51d4be9414f", "context/get_test_command_example.py": "33b16476536f8c9424d3222ffc4a5cd5ecee6ad0589b2c66a4f3c7f1e265106c", - "context/operation_log_example.py": "6cd7962ab255d6080aeb97897175250f987ce936cf7333e7601f10834405a940", - "context/pytest_output_example.py": "6f0cd68e3c4440081267c716651caec4fbdd7d9c4516f967517f02ae2a853920", + "context/operation_log_example.py": "2660def7012d9b79bad3db33cf399ac603cc1c35dff0b064cbdd2638a013de1d", "context/python_env_detector_example.py": "65619b3e8bad0509f1ccf54d81b5c3760f423b6fc1925c932fc7968da78a172e", - "context/sync_determine_operation_example.py": "94a333b37d9875bcc9b39e22d4d00de1b196ad2f7e0fa997e6f638e153dbdfad", - "context/update_main_example.py": "b318834c648f25e6141d81f3ecac1d9da592a6bb4740500983f2b07b8a10d902" + "context/sync_determine_operation_example.py": "225d20ea0210bbf52650a9f424879ff495e574dd2411eb4b86592a6d06cf59a5", + "context/update_main_example.py": "9fe01fd19196c2604c91d101e49db849e0e1c9fabafeb2ad831a9c91b2f26416", + "pdd/compressed_sync_context.py": "661fde0fcc42ce39c1004e47a774a8fd799b789065519cb5688949c35fed4859", + "pdd/get_run_command.py": "5eec778a234abc4ace4166d71a68f6e49f1b81491962867de81f61be882a5319", + "pdd/pytest_output.py": "652a27dd1e08769611abe3cb6a83f372c450279d6bf1b676741b9b7186665975" } } \ No newline at end of file diff --git a/.pdd/meta/sync_orchestration_python_run.json b/.pdd/meta/sync_orchestration_python_run.json index 21a1df2d31..eb0a9463fb 100644 --- a/.pdd/meta/sync_orchestration_python_run.json +++ b/.pdd/meta/sync_orchestration_python_run.json @@ -1,11 +1,11 @@ { - "timestamp": "2026-05-12T02:53:56.363281+00:00", + "timestamp": "2026-07-11T01:38:00.868634+00:00", "exit_code": 0, - "tests_passed": 1, + "tests_passed": 249, "tests_failed": 0, - "coverage": 100.0, - "test_hash": "3fce96bc02b851cc3e2d00001b0de3a2436f687c65a53d5e423212ac5c6d2517", + "coverage": 21.680376028202115, + "test_hash": "7a4a23ed7c00408887d4ca9b604cb9715e0ad6e0a9c3e9e1b4df78603b396f41", "test_files": { - "test_sync_orchestration.py": "3fce96bc02b851cc3e2d00001b0de3a2436f687c65a53d5e423212ac5c6d2517" + "test_sync_orchestration.py": "7a4a23ed7c00408887d4ca9b604cb9715e0ad6e0a9c3e9e1b4df78603b396f41" } -} +} \ No newline at end of file diff --git a/.pdd/meta/update_main_python.json b/.pdd/meta/update_main_python.json new file mode 100644 index 0000000000..5e43dd853d --- /dev/null +++ b/.pdd/meta/update_main_python.json @@ -0,0 +1,28 @@ +{ + "pdd_version": "0.0.301.dev0", + "timestamp": "2026-07-11T01:12:43.814380+00:00", + "command": "test", + "prompt_hash": "cd1caff9862284ca219d4ed196b85c9aea1e65ffb4940ec389dea1d7c81b75b7", + "code_hash": "7d53f775228a444ddced72041bbbfd28a4b9d3d2d527844c78d0f5b6d3c90f4c", + "example_hash": "9fe01fd19196c2604c91d101e49db849e0e1c9fabafeb2ad831a9c91b2f26416", + "test_hash": "3a069a794e3e0b8831830aa2bb46d8e201ddd6575318d037326914b92b9a4c1a", + "test_files": { + "test_update_main.py": "3a069a794e3e0b8831830aa2bb46d8e201ddd6575318d037326914b92b9a4c1a" + }, + "include_deps": { + "context/agentic_common_example.py": "c1538a115a3a33da228e7f6cc048f2a99023cd2c83a0efef382e49c477b24ad3", + "context/agentic_update_example.py": "967254cd1c636ac5ba09afa0446312ffc1f28d3041ed453a4c6588c931f722c4", + "context/api_key_scanner_example.py": "4b9f880a30445f25edffc394857ae0c218097726c41ab996a03fcbc4350b5482", + "context/architecture_sync_example.py": "b6158dbc7ca3fd32665528562225f6739ec521a071d845acab0f7caaea221ae0", + "context/auto_include_example.py": "0c278ed716298149777bc7082827eb8aa6d80dac264c66b6cded6871232f2df7", + "context/construct_paths_example.py": "412cb260650bcbd628fba647db917a8d32f8191df2e1c32cc22f22c0c11b1740", + "context/ctx_obj_params.prompt": "829e2ab327e37b75afcd0dcf3893dcaec9012a89a1f6e5198ec7cfc0395d2782", + "context/git_update_example.py": "811304cee2cb890879ee68a816aed49dff2e7b50cfdca3125e16e0ae99819574", + "context/metadata_sync_example.py": "704453b3386724eef225b61c48f2fe922e1a973ac211c494e1d955dd44bcca11", + "context/operation_log_example.py": "2660def7012d9b79bad3db33cf399ac603cc1c35dff0b064cbdd2638a013de1d", + "context/python_preamble.prompt": "0388ed131bf986f8752e1bc4c81e4da0460cfe2908ec8c60b1314edbab768254", + "context/sync_determine_operation_example.py": "225d20ea0210bbf52650a9f424879ff495e574dd2411eb4b86592a6d06cf59a5", + "context/update_prompt_example.py": "56b9d8c7aa823c2996a3dc5741a47844e223bfd088366d74e48b6a099b5a11c4", + "pdd/validate_prompt_includes.py": "fa97b72a58534e255768634f928f4cfdd2130a7a4d8b296da3eefdf77d47855b" + } +} \ No newline at end of file diff --git a/.pdd/meta/update_main_python_run.json b/.pdd/meta/update_main_python_run.json new file mode 100644 index 0000000000..2a50a39853 --- /dev/null +++ b/.pdd/meta/update_main_python_run.json @@ -0,0 +1,11 @@ +{ + "timestamp": "2026-07-11T01:38:00.923700+00:00", + "exit_code": 0, + "tests_passed": 316, + "tests_failed": 0, + "coverage": 77.03619909502262, + "test_hash": "3a069a794e3e0b8831830aa2bb46d8e201ddd6575318d037326914b92b9a4c1a", + "test_files": { + "test_update_main.py": "3a069a794e3e0b8831830aa2bb46d8e201ddd6575318d037326914b92b9a4c1a" + } +} \ No newline at end of file diff --git a/architecture.json b/architecture.json index 50e2710dd0..0a00b36c6d 100644 --- a/architecture.json +++ b/architecture.json @@ -5556,9 +5556,7 @@ { "reason": "CLI entry point for the change command.", "description": "Command-line interface for change. Parses arguments, orchestrates the workflow, and formats output.", - "dependencies": [ - "fingerprint_transaction_python.prompt" - ], + "dependencies": [], "priority": 150, "filename": "change_main_python.prompt", "filepath": "pdd/change_main.py", diff --git a/pdd/prompts/pin_example_hack_python.prompt b/pdd/prompts/pin_example_hack_python.prompt index 7e29dd7e69..dda1aea9f1 100644 --- a/pdd/prompts/pin_example_hack_python.prompt +++ b/pdd/prompts/pin_example_hack_python.prompt @@ -12,6 +12,7 @@ Create a Python module that orchestrates the full PDD sync workflow for one base context/fix_main_example.py context/update_main_example.py context/get_test_command_example.py +context/fingerprint_transaction_example.py ## Public API @@ -66,7 +67,7 @@ At function start, coerce `None` for `target_coverage`, `budget`, `max_attempts` Implement: - `load_sync_log`, `create_sync_log_entry`, `update_sync_log_entry`, `append_sync_log`, `log_sync_event` - `save_run_report(report, basename, language, atomic_state=None)` writing `META_DIR / "{safe_basename}_{language.lower()}_run.json"` -- `_save_operation_fingerprint(..., atomic_state=None)` using `calculate_current_hashes` + `Fingerprint`, writing `META_DIR / "{safe_basename}_{language.lower()}.json"` +- `_save_operation_fingerprint(..., atomic_state=None)` using `FingerprintTransaction(basename, language, operation, paths=paths, cost=cost, model=model)`. Without an outer `AtomicStateUpdate`, successful context exit commits through the shared transaction. With `atomic_state`, call `transaction.prepare()`, buffer that payload at `transaction.fingerprint_path`, then call `transaction.skip("deferred to AtomicStateUpdate")` so only the outer atomic state owns the final write. - `_python_cov_target_for_code_file`, `_python_cov_target_for_test_and_code` - `_parse_test_output` for Python, JS/TS, Go, Rust, plus fallback - `_detect_example_errors` (traceback + ERROR-log patterns only)