Add blueprint validation reports#380
Merged
Merged
Conversation
📝 WalkthroughWalkthroughThis PR introduces a blueprint validation system: a ChangesBlueprint validation and CLI
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/casecrawler/validation/blueprints.py`:
- Around line 67-74: The check that adds a "clinical_reasoning_targets" issue
currently sets severity "warning" so a blueprint can still be marked
RESEARCH_RELEASE_READY despite this outstanding issue; update the validation in
blueprints.py to treat a missing clinical_reasoning_targets as a blocking issue
(set severity to "error" or otherwise add it to self.issues as an error) so
BlueprintValidationReport.research_release_ready (the property on
BlueprintValidationReport) will correctly block promotion; apply the same change
to the duplicate check around the other occurrence (the 98-109 block) so both
places create an error-level issue for blueprint.clinical_reasoning_targets.
- Around line 51-58: The check currently treats age==0 as missing because it
uses a falsy test on blueprint.patient.get("age"); update the conditional in the
validation (the block using blueprint.patient.get("age") that appends to issues)
to explicitly test for absence or None (e.g., check if "age" not in
blueprint.patient or blueprint.patient["age"] is None) so that age 0 is
considered present while still flagging missing/None values.
- Around line 93-96: The _grounded method currently returns True whenever
self._require_grounding is False, making ungrounded ClinicalBlueprints appear
grounded; change _grounded(self, blueprint: ClinicalBlueprint) to always compute
and return the factual grounding state based solely on
blueprint.evidence.citations (e.g., bool(blueprint.evidence.citations)) and stop
using self._require_grounding to override that result; use require_grounding
only to gate validation behavior elsewhere (e.g., raise or skip validation in
the validator flow), so persisted reports and tier calculations rely on the
factual _grounded value rather than a policy flag.
- Around line 22-37: The code currently hard-codes schema_valid=True; change it
to derive schema_valid from the collected issues: set schema_valid = not
any(issue["severity"] == "error" for issue in issues) so any structural/schema
error (e.g., missing required patient fields) flips schema_valid to False.
Update the block that computes schema_valid alongside issues, grounded,
clinically_plausible and tier (references: variables schema_valid, issues,
grounded, clinically_plausible, judge_validated and the _issues/_grounded/_tier
helpers) to use this computed value.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: c40e576f-1436-4648-a9d1-d01314465cc8
📒 Files selected for processing (4)
src/casecrawler/cli.pysrc/casecrawler/validation/blueprints.pytests/test_blueprint_validator.pytests/test_cli_synthetic.py
Comment on lines
+22
to
+37
| issues = self._issues(blueprint) | ||
| schema_valid = True | ||
| grounded = self._grounded(blueprint) | ||
| clinically_plausible = not any( | ||
| issue["severity"] == "error" and issue["field"] != "evidence.citations" | ||
| for issue in issues | ||
| ) | ||
| if self._require_grounding and not grounded: | ||
| clinically_plausible = False | ||
| judge_validated = bool(judges) and all(report.passed for report in judges) | ||
| tier = self._tier( | ||
| schema_valid=schema_valid, | ||
| clinically_plausible=clinically_plausible, | ||
| grounded=grounded, | ||
| judge_validated=judge_validated, | ||
| ) |
There was a problem hiding this comment.
Compute schema_valid from the collected issues instead of hard-coding it.
Right now any blueprint with missing required patient fields still comes back with schema_valid=True, so ReleaseReadinessTier.DRAFT is unreachable and a structurally incomplete blueprint can still be labeled SCHEMA_VALID.
Suggested direction
issues = self._issues(blueprint)
- schema_valid = True
+ schema_valid = not any(
+ issue["severity"] == "error"
+ and issue["field"] in {"patient.age", "patient.sex"}
+ for issue in issues
+ )📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| issues = self._issues(blueprint) | |
| schema_valid = True | |
| grounded = self._grounded(blueprint) | |
| clinically_plausible = not any( | |
| issue["severity"] == "error" and issue["field"] != "evidence.citations" | |
| for issue in issues | |
| ) | |
| if self._require_grounding and not grounded: | |
| clinically_plausible = False | |
| judge_validated = bool(judges) and all(report.passed for report in judges) | |
| tier = self._tier( | |
| schema_valid=schema_valid, | |
| clinically_plausible=clinically_plausible, | |
| grounded=grounded, | |
| judge_validated=judge_validated, | |
| ) | |
| issues = self._issues(blueprint) | |
| schema_valid = not any( | |
| issue["severity"] == "error" | |
| and issue["field"] in {"patient.age", "patient.sex"} | |
| for issue in issues | |
| ) | |
| grounded = self._grounded(blueprint) | |
| clinically_plausible = not any( | |
| issue["severity"] == "error" and issue["field"] != "evidence.citations" | |
| for issue in issues | |
| ) | |
| if self._require_grounding and not grounded: | |
| clinically_plausible = False | |
| judge_validated = bool(judges) and all(report.passed for report in judges) | |
| tier = self._tier( | |
| schema_valid=schema_valid, | |
| clinically_plausible=clinically_plausible, | |
| grounded=grounded, | |
| judge_validated=judge_validated, | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/casecrawler/validation/blueprints.py` around lines 22 - 37, The code
currently hard-codes schema_valid=True; change it to derive schema_valid from
the collected issues: set schema_valid = not any(issue["severity"] == "error"
for issue in issues) so any structural/schema error (e.g., missing required
patient fields) flips schema_valid to False. Update the block that computes
schema_valid alongside issues, grounded, clinically_plausible and tier
(references: variables schema_valid, issues, grounded, clinically_plausible,
judge_validated and the _issues/_grounded/_tier helpers) to use this computed
value.
Comment on lines
+51
to
+58
| if not blueprint.patient.get("age"): | ||
| issues.append( | ||
| { | ||
| "severity": "error", | ||
| "field": "patient.age", | ||
| "message": "Blueprint patient must include age.", | ||
| } | ||
| ) |
There was a problem hiding this comment.
Treat age 0 as present.
Line 51 flags newborn blueprints as missing patient.age because 0 is falsy.
Suggested fix
- if not blueprint.patient.get("age"):
+ if "age" not in blueprint.patient or blueprint.patient.get("age") is None:📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if not blueprint.patient.get("age"): | |
| issues.append( | |
| { | |
| "severity": "error", | |
| "field": "patient.age", | |
| "message": "Blueprint patient must include age.", | |
| } | |
| ) | |
| if "age" not in blueprint.patient or blueprint.patient.get("age") is None: | |
| issues.append( | |
| { | |
| "severity": "error", | |
| "field": "patient.age", | |
| "message": "Blueprint patient must include age.", | |
| } | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/casecrawler/validation/blueprints.py` around lines 51 - 58, The check
currently treats age==0 as missing because it uses a falsy test on
blueprint.patient.get("age"); update the conditional in the validation (the
block using blueprint.patient.get("age") that appends to issues) to explicitly
test for absence or None (e.g., check if "age" not in blueprint.patient or
blueprint.patient["age"] is None) so that age 0 is considered present while
still flagging missing/None values.
Comment on lines
+67
to
+74
| if not blueprint.clinical_reasoning_targets: | ||
| issues.append( | ||
| { | ||
| "severity": "warning", | ||
| "field": "clinical_reasoning_targets", | ||
| "message": "Blueprint has no explicit reasoning target.", | ||
| } | ||
| ) |
There was a problem hiding this comment.
Don't promote reports with outstanding issues to RESEARCH_RELEASE_READY.
A blueprint with only the clinical_reasoning_targets warning can still reach RESEARCH_RELEASE_READY after judge approval, but BlueprintValidationReport.research_release_ready in src/casecrawler/models/blueprint.py:214-225 requires not self.issues. That makes tier and the report's own readiness property disagree.
Also applies to: 98-109
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/casecrawler/validation/blueprints.py` around lines 67 - 74, The check
that adds a "clinical_reasoning_targets" issue currently sets severity "warning"
so a blueprint can still be marked RESEARCH_RELEASE_READY despite this
outstanding issue; update the validation in blueprints.py to treat a missing
clinical_reasoning_targets as a blocking issue (set severity to "error" or
otherwise add it to self.issues as an error) so
BlueprintValidationReport.research_release_ready (the property on
BlueprintValidationReport) will correctly block promotion; apply the same change
to the duplicate check around the other occurrence (the 98-109 block) so both
places create an error-level issue for blueprint.clinical_reasoning_targets.
Comment on lines
+93
to
+96
| def _grounded(self, blueprint: ClinicalBlueprint) -> bool: | ||
| if not self._require_grounding: | ||
| return True | ||
| return bool(blueprint.evidence.citations) |
There was a problem hiding this comment.
Keep grounded as an observed fact, not a policy override.
When require_grounding=False, _grounded() returns True even if evidence.citations is empty. That makes ungrounded blueprints indistinguishable from genuinely grounded ones in persisted reports and can incorrectly inflate the tier.
Suggested fix
def _grounded(self, blueprint: ClinicalBlueprint) -> bool:
- if not self._require_grounding:
- return True
return bool(blueprint.evidence.citations)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| def _grounded(self, blueprint: ClinicalBlueprint) -> bool: | |
| if not self._require_grounding: | |
| return True | |
| return bool(blueprint.evidence.citations) | |
| def _grounded(self, blueprint: ClinicalBlueprint) -> bool: | |
| return bool(blueprint.evidence.citations) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/casecrawler/validation/blueprints.py` around lines 93 - 96, The _grounded
method currently returns True whenever self._require_grounding is False, making
ungrounded ClinicalBlueprints appear grounded; change _grounded(self, blueprint:
ClinicalBlueprint) to always compute and return the factual grounding state
based solely on blueprint.evidence.citations (e.g.,
bool(blueprint.evidence.citations)) and stop using self._require_grounding to
override that result; use require_grounding only to gate validation behavior
elsewhere (e.g., raise or skip validation in the validator flow), so persisted
reports and tier calculations rely on the factual _grounded value rather than a
policy flag.
This was referenced Jun 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
casecrawler validate-blueprintsto validate persisted blueprint artifacts and store BlueprintValidationReport rowsTests
Summary by CodeRabbit
New Features
validate-blueprintsCLI command to validate clinical blueprint artifacts--no-groundingflag to make grounding citations optionalTests