🧪 [Add evaluation logic and corresponding tests]#4
Conversation
Adds the evaluate_predictions function to compute accuracy metrics from CSV files. Includes test_evaluation.py with missing file, empty file, and accuracy correctness scenarios. Co-authored-by: NITISH-R-G <225521762+NITISH-R-G@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
Reviewer's GuideImplements a CSV-based prediction evaluation function that computes per-field correctness counts and accuracies, along with a focused pytest suite covering missing files, empty files, and correctness of the computed metrics. Flow diagram for evaluate_predictions evaluation logicflowchart TD
A["evaluate_predictions(predictions_path, truth_path)"] --> B{predictions_path.exists and truth_path.exists}
B -- No --> Z[Return empty dict]
B -- Yes --> C[Read predictions CSV with csv.DictReader]
C --> D[Read truth CSV with csv.DictReader into dict keyed by user_id]
D --> E[Initialize metrics counters]
E --> F[For each pred in preds]
F --> G{user_id present and in truth}
G -- No --> F
G -- Yes --> H[Compare claim_status, issue_type, object_part, severity]
H --> I[Increment correct_* counters on match]
I --> F
F --> J{After loop: metrics total > 0}
J -- Yes --> K[Compute per field accuracies as correct_*/total]
J -- No --> L[Set per field accuracies to 0.0]
K --> M[Return metrics]
L --> M[Return metrics]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
📝 WalkthroughSummary by CodeRabbit
WalkthroughA new ChangesPrediction Evaluation
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- Consider basing the accuracy denominators on the number of predictions that actually have a matching
user_idin the truth set rather thanlen(preds), to avoid skewed metrics when there are many unmatched predictions. - You might want to factor the repeated field names (
claim_status,issue_type,object_part,severity) into a single list and loop over it to build and update the metrics, which will reduce duplication and make it easier to add/remove fields consistently.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- Consider basing the accuracy denominators on the number of predictions that actually have a matching `user_id` in the truth set rather than `len(preds)`, to avoid skewed metrics when there are many unmatched predictions.
- You might want to factor the repeated field names (`claim_status`, `issue_type`, `object_part`, `severity`) into a single list and loop over it to build and update the metrics, which will reduce duplication and make it easier to add/remove fields consistently.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@code/evaluation/main.py`:
- Around line 29-36: Add schema validation before the comparison block that
checks for the presence of required columns (user_id, claim_status, issue_type,
object_part, and severity) in both the pred and t dictionaries. If any required
field is missing from either input, fail fast or return a structured error
instead of allowing the .get() comparisons to silently treat missing fields as
None. This prevents the incorrect increment of correct_claim_status,
correct_issue_type, correct_object_part, and correct_severity metrics when
fields are absent in both files.
In `@code/tests/test_evaluation.py`:
- Line 27: Replace direct floating-point equality assertions with
pytest.approx() to handle potential floating-point precision issues. For the
assertion checking metrics["accuracy_claim_status"] == 0.0 at line 27 and the
similar assertions at lines 65-68, wrap the expected float values with
pytest.approx() so that the comparisons allow for acceptable floating-point
tolerances instead of requiring exact equality.
- Around line 21-23: The CSV writer calls in the test file are not explicitly
setting encoding and line terminator settings, which can result in
platform-dependent line endings and encoding. Locate all instances where
csv.writer is instantiated in the file (including the open call at lines 21-23
and the similar pattern at lines 47-55), and modify each to set encoding="utf-8"
in the open() function call and lineterminator="\n" in the csv.writer() call to
ensure consistent UTF-8 encoding with LF-only newlines across all platforms.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 1eaadda4-4ed2-4943-b66d-7047faafddba
📒 Files selected for processing (2)
code/evaluation/main.pycode/tests/test_evaluation.py
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Sourcery review
🧰 Additional context used
📓 Path-based instructions (1)
**/*.{py,js,ts}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{py,js,ts}: Always resolve log file path using platform's home dir (os.homedir() / pathlib.Path.home() / $HOME / %USERPROFILE%); never hardcode paths like /Users/... or C:\Users...
Write log file in UTF-8 with LF line endings (\n); do not emit CRLF (\r\n) even on Windows
Prefer language-native APIs over shelling out for file and path operations; when shelling out is necessary, provide both Unix and Windows forms
Read secrets from environment variables only (OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.); never hardcode
Files:
code/tests/test_evaluation.pycode/evaluation/main.py
🪛 GitHub Check: SonarCloud Code Analysis
code/tests/test_evaluation.py
[warning] 66-66: Correct one of the identical sub-expressions on both sides of operator "/".
[warning] 27-27: Do not perform equality checks with floating point values.
code/evaluation/main.py
[failure] 5-5: Refactor this function to reduce its Cognitive Complexity from 16 to the 15 allowed.
🪛 Ruff (0.15.17)
code/evaluation/main.py
[warning] 10-10: Unnecessary mode argument
Remove mode argument
(UP015)
[warning] 12-12: Unnecessary mode argument
Remove mode argument
(UP015)
🔇 Additional comments (1)
code/evaluation/main.py (1)
16-26: Current behavior confirmed: unmatched predictions are penalized (included in denominator, never counted as correct).The test suite (lines 59–68) explicitly validates this contract: with 3 predictions all matched to truth,
total=3andaccuracy_claim_status=2/3. The accuracy calculation (lines 39–42) divides bytotal(all predictions), not by matched predictions. This design penalizes unmatched rows as automatic failures.The evaluation context supports this: the spec (evaluation_spec.md) describes evaluation on a labeled sample dataset where all predictions are expected to have ground truth entries. Code at lines 25–26 silently skips unmatched predictions (neither increments correct counts nor adjusts denominator), making this behavior implicit but correct as implemented.
No test covers the case of predictions with missing truth entries, and the docstring does not document this denominator choice. If unmatched predictions may occur in practice, add a test case and clarify the metric semantics in a docstring.
| if pred.get('claim_status') == t.get('claim_status'): | ||
| metrics['correct_claim_status'] += 1 | ||
| if pred.get('issue_type') == t.get('issue_type'): | ||
| metrics['correct_issue_type'] += 1 | ||
| if pred.get('object_part') == t.get('object_part'): | ||
| metrics['correct_object_part'] += 1 | ||
| if pred.get('severity') == t.get('severity'): | ||
| metrics['correct_severity'] += 1 |
There was a problem hiding this comment.
Validate required CSV columns before scoring.
If a required field is absent in both files, pred.get(field) == t.get(field) becomes None == None, which incorrectly increments “correct_*”. Add an upfront schema check for user_id, claim_status, issue_type, object_part, and severity in both inputs, and fail fast (or return a structured error) when missing.
Proposed fix
def evaluate_predictions(predictions_path: Path, truth_path: Path) -> dict:
@@
with open(predictions_path, encoding='utf-8') as f:
- preds = list(csv.DictReader(f))
+ pred_reader = csv.DictReader(f)
+ preds = list(pred_reader)
with open(truth_path, encoding='utf-8') as f:
- truth = {row['user_id']: row for row in csv.DictReader(f)}
+ truth_reader = csv.DictReader(f)
+ truth_rows = list(truth_reader)
+
+ required = {"user_id", "claim_status", "issue_type", "object_part", "severity"}
+ pred_cols = set(pred_reader.fieldnames or [])
+ truth_cols = set(truth_reader.fieldnames or [])
+ if not required.issubset(pred_cols) or not required.issubset(truth_cols):
+ return {}
+
+ truth = {row["user_id"]: row for row in truth_rows}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@code/evaluation/main.py` around lines 29 - 36, Add schema validation before
the comparison block that checks for the presence of required columns (user_id,
claim_status, issue_type, object_part, and severity) in both the pred and t
dictionaries. If any required field is missing from either input, fail fast or
return a structured error instead of allowing the .get() comparisons to silently
treat missing fields as None. This prevents the incorrect increment of
correct_claim_status, correct_issue_type, correct_object_part, and
correct_severity metrics when fields are absent in both files.
| with open(path, "w", newline="") as f: | ||
| writer = csv.writer(f) | ||
| writer.writerow(headers) |
There was a problem hiding this comment.
Write test CSVs as UTF-8 with LF-only newlines.
Current writers can emit CRLF (\r\n) and rely on platform-default encoding. Set encoding="utf-8" and lineterminator="\n" in csv.writer to satisfy cross-platform determinism and repository policy.
As per coding guidelines, “Write log file in UTF-8 with LF line endings (\n); do not emit CRLF (\r\n) even on Windows”.
Proposed fix
- with open(path, "w", newline="") as f:
- writer = csv.writer(f)
+ with open(path, "w", newline="", encoding="utf-8") as f:
+ writer = csv.writer(f, lineterminator="\n")
writer.writerow(headers)
@@
- with open(truth_path, "w", newline="") as f:
- writer = csv.writer(f)
+ with open(truth_path, "w", newline="", encoding="utf-8") as f:
+ writer = csv.writer(f, lineterminator="\n")
writer.writerow(headers)
writer.writerows(truth_data)
@@
- with open(pred_path, "w", newline="") as f:
- writer = csv.writer(f)
+ with open(pred_path, "w", newline="", encoding="utf-8") as f:
+ writer = csv.writer(f, lineterminator="\n")
writer.writerow(headers)
writer.writerows(pred_data)Also applies to: 47-55
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@code/tests/test_evaluation.py` around lines 21 - 23, The CSV writer calls in
the test file are not explicitly setting encoding and line terminator settings,
which can result in platform-dependent line endings and encoding. Locate all
instances where csv.writer is instantiated in the file (including the open call
at lines 21-23 and the similar pattern at lines 47-55), and modify each to set
encoding="utf-8" in the open() function call and lineterminator="\n" in the
csv.writer() call to ensure consistent UTF-8 encoding with LF-only newlines
across all platforms.
Source: Coding guidelines
Resolves flake8 and pylint errors found in SonarCloud Code Analysis. Removes trailing whitespaces, unused imports, line too long errors, and adds missing docstrings for code/evaluation/main.py and code/tests/test_evaluation.py. Also fixes missing encoding in file opening calls. Co-authored-by: NITISH-R-G <225521762+NITISH-R-G@users.noreply.github.com>
There was a problem hiding this comment.
♻️ Duplicate comments (2)
code/tests/test_evaluation.py (1)
32-32:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winUse tolerant float assertions for accuracy metrics.
Line 32 and Line 77–Line 80 use direct float equality; switch to
pytest.approx(...)to avoid brittle precision failures.Proposed fix
+import pytest import csv from evaluation.main import evaluate_predictions @@ - assert metrics["accuracy_claim_status"] == 0.0 + assert metrics["accuracy_claim_status"] == pytest.approx(0.0) @@ - assert metrics["accuracy_claim_status"] == 2 / 3 - assert metrics["accuracy_issue_type"] == 3 / 3 - assert metrics["accuracy_object_part"] == 2 / 3 - assert metrics["accuracy_severity"] == 2 / 3 + assert metrics["accuracy_claim_status"] == pytest.approx(2 / 3) + assert metrics["accuracy_issue_type"] == pytest.approx(1.0) + assert metrics["accuracy_object_part"] == pytest.approx(2 / 3) + assert metrics["accuracy_severity"] == pytest.approx(2 / 3)Also applies to: 77-80
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@code/tests/test_evaluation.py` at line 32, Replace direct float equality assertions with pytest.approx() to handle floating point precision tolerantly. In the test_evaluation.py file, locate the assertion at line 32 that compares metrics["accuracy_claim_status"] == 0.0 and modify it to use pytest.approx() wrapper around the expected float value. Apply the same fix to the accuracy metric assertions at lines 77-80. This ensures that floating point comparisons allow for small precision differences rather than requiring exact equality, preventing brittle test failures.Source: Linters/SAST tools
code/evaluation/main.py (1)
11-37:⚠️ Potential issue | 🟠 Major | ⚡ Quick winValidate required headers before scoring comparisons.
On Line 30–Line 37, missing required columns in both inputs can be counted as correct (
None == None), inflating metrics. Add an upfront required-schema check foruser_id,claim_status,issue_type,object_part, andseverityin both CSVs before building/using row dicts.Proposed fix
- with open(predictions_path, 'r', encoding='utf-8') as f: - preds = list(csv.DictReader(f)) - with open(truth_path, 'r', encoding='utf-8') as f: - truth = {row['user_id']: row for row in csv.DictReader(f)} + required = {"user_id", "claim_status", "issue_type", "object_part", "severity"} + with open(predictions_path, 'r', encoding='utf-8') as f: + pred_reader = csv.DictReader(f) + preds = list(pred_reader) + with open(truth_path, 'r', encoding='utf-8') as f: + truth_reader = csv.DictReader(f) + truth_rows = list(truth_reader) + + pred_cols = set(pred_reader.fieldnames or []) + truth_cols = set(truth_reader.fieldnames or []) + if not required.issubset(pred_cols) or not required.issubset(truth_cols): + return {} + + truth = {row['user_id']: row for row in truth_rows}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@code/evaluation/main.py` around lines 11 - 37, Missing CSV columns can cause None values to match incorrectly, inflating metrics. After reading both CSV files (after the truth dictionary is constructed at line 14), add validation to ensure the required headers user_id, claim_status, issue_type, object_part, and severity are present in both the predictions and truth data. This validation should check the fieldnames from the DictReader objects and raise an appropriate error if any required columns are missing before entering the comparison loop that starts at line 24.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@code/evaluation/main.py`:
- Around line 11-37: Missing CSV columns can cause None values to match
incorrectly, inflating metrics. After reading both CSV files (after the truth
dictionary is constructed at line 14), add validation to ensure the required
headers user_id, claim_status, issue_type, object_part, and severity are present
in both the predictions and truth data. This validation should check the
fieldnames from the DictReader objects and raise an appropriate error if any
required columns are missing before entering the comparison loop that starts at
line 24.
In `@code/tests/test_evaluation.py`:
- Line 32: Replace direct float equality assertions with pytest.approx() to
handle floating point precision tolerantly. In the test_evaluation.py file,
locate the assertion at line 32 that compares metrics["accuracy_claim_status"]
== 0.0 and modify it to use pytest.approx() wrapper around the expected float
value. Apply the same fix to the accuracy metric assertions at lines 77-80. This
ensures that floating point comparisons allow for small precision differences
rather than requiring exact equality, preventing brittle test failures.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: f41f572d-3877-4d88-8a34-4da6f216c3a0
📒 Files selected for processing (2)
code/evaluation/main.pycode/tests/test_evaluation.py
📜 Review details
🧰 Additional context used
📓 Path-based instructions (1)
**/*.{py,js,ts}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{py,js,ts}: Always resolve log file path using platform's home dir (os.homedir() / pathlib.Path.home() / $HOME / %USERPROFILE%); never hardcode paths like /Users/... or C:\Users...
Write log file in UTF-8 with LF line endings (\n); do not emit CRLF (\r\n) even on Windows
Prefer language-native APIs over shelling out for file and path operations; when shelling out is necessary, provide both Unix and Windows forms
Read secrets from environment variables only (OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.); never hardcode
Files:
code/tests/test_evaluation.pycode/evaluation/main.py
Enhances `evaluate_predictions` in `code/evaluation/main.py` with proper type hints (`Dict[str, Union[int, float]]`), removes duplicate code by using a field loop for calculating metrics, and adds a `main` execution block for local script runs. Refactors `code/tests/test_evaluation.py` to use `pytest.approx` for floating point assertions to avoid precision issues reported by SonarCloud. Co-authored-by: NITISH-R-G <225521762+NITISH-R-G@users.noreply.github.com>
|
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
code/tests/test_evaluation.py (1)
17-33: 🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick winAdd regression tests for skipped rows and missing-column schema cases.
Current tests miss two critical branches: (1) predictions with missing/unknown
user_id(denominator behavior), and (2) inputs missing required columns (schema validation). Add explicit cases so these correctness rules stay protected.Also applies to: 36-81
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@code/tests/test_evaluation.py` around lines 17 - 33, The test suite for evaluate_predictions is missing coverage for two critical branches: handling predictions with missing or unknown user_id values (which affects denominator behavior), and validating that input files contain all required columns. Add two new regression test functions: one that calls evaluate_predictions with a CSV containing rows where user_id is missing or unknown to verify the function handles denominator logic correctly, and another that calls evaluate_predictions with CSV files missing required columns (like claim_status or issue_type) to verify schema validation error handling. These tests should explicitly verify the correctness rules for these edge cases.
♻️ Duplicate comments (1)
code/evaluation/main.py (1)
16-18:⚠️ Potential issue | 🟠 Major | ⚡ Quick winValidate required CSV schema before scoring to avoid false “correct” counts.
If required fields are absent in both rows,
pred.get(field) == t_row.get(field)becomesNone == Noneand incorrectly incrementscorrect_*. Add upfront required-column validation (user_id,claim_status,issue_type,object_part,severity) for both files and fail fast or return{}.Also applies to: 35-37
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@code/evaluation/main.py` around lines 16 - 18, Add upfront validation of required CSV columns immediately after loading both the truth CSV file (when reading from truth_path with csv.DictReader) and the prediction CSV file (around lines 35-37). Validate that both files contain all required fields: user_id, claim_status, issue_type, object_part, and severity. If any required field is missing from either file, fail fast by either raising an error or returning an empty dict to prevent incorrect scoring where missing fields in both files would result in None == None comparisons that falsely increment correct counts.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@code/evaluation/main.py`:
- Around line 20-21: The accuracy calculation uses total row count as the
denominator, but rows with missing or unknown user_ids are skipped during
evaluation, causing accuracy to be underreported. Create a separate
evaluated_total counter that only increments when a row is actually evaluated
(not skipped), and use evaluated_total as the denominator for accuracy
calculations instead of total. Apply this same pattern to all accuracy
calculations in the file (the sections around lines 29-33 and 39-43 that also
have similar skip logic for rows).
---
Outside diff comments:
In `@code/tests/test_evaluation.py`:
- Around line 17-33: The test suite for evaluate_predictions is missing coverage
for two critical branches: handling predictions with missing or unknown user_id
values (which affects denominator behavior), and validating that input files
contain all required columns. Add two new regression test functions: one that
calls evaluate_predictions with a CSV containing rows where user_id is missing
or unknown to verify the function handles denominator logic correctly, and
another that calls evaluate_predictions with CSV files missing required columns
(like claim_status or issue_type) to verify schema validation error handling.
These tests should explicitly verify the correctness rules for these edge cases.
---
Duplicate comments:
In `@code/evaluation/main.py`:
- Around line 16-18: Add upfront validation of required CSV columns immediately
after loading both the truth CSV file (when reading from truth_path with
csv.DictReader) and the prediction CSV file (around lines 35-37). Validate that
both files contain all required fields: user_id, claim_status, issue_type,
object_part, and severity. If any required field is missing from either file,
fail fast by either raising an error or returning an empty dict to prevent
incorrect scoring where missing fields in both files would result in None ==
None comparisons that falsely increment correct counts.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: feba1d4d-d549-4f2d-b498-7837b7a31d38
📒 Files selected for processing (2)
code/evaluation/main.pycode/tests/test_evaluation.py
📜 Review details
🧰 Additional context used
📓 Path-based instructions (1)
**/*.{py,js,ts}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{py,js,ts}: Always resolve log file path using platform's home dir (os.homedir() / pathlib.Path.home() / $HOME / %USERPROFILE%); never hardcode paths like /Users/... or C:\Users...
Write log file in UTF-8 with LF line endings (\n); do not emit CRLF (\r\n) even on Windows
Prefer language-native APIs over shelling out for file and path operations; when shelling out is necessary, provide both Unix and Windows forms
Read secrets from environment variables only (OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.); never hardcode
Files:
code/tests/test_evaluation.pycode/evaluation/main.py
🪛 Ruff (0.15.17)
code/evaluation/main.py
[warning] 4-4: typing.Dict is deprecated, use dict instead
(UP035)
[warning] 9-9: Use X | Y for type annotations
Convert to X | Y
(UP007)
[warning] 19-19: Use X | Y for type annotations
Convert to X | Y
(UP007)
| 'total': len(preds), | ||
| 'correct_claim_status': 0, |
There was a problem hiding this comment.
Use the evaluated-row count as the accuracy denominator.
total (Line 20) counts all prediction rows, but Lines 31-32 skip rows that are not actually evaluated. This underreports accuracy whenever predictions contain missing/unknown user_ids. Track a separate evaluated_total and divide by that instead.
Proposed fix
- metrics: Dict[str, Union[int, float]] = {
- 'total': len(preds),
+ metrics: Dict[str, Union[int, float]] = {
+ 'total': 0,
'correct_claim_status': 0,
'correct_issue_type': 0,
'correct_object_part': 0,
'correct_severity': 0
}
@@
for pred in preds:
user_id = pred.get('user_id')
if not user_id or user_id not in truth:
continue
+ metrics['total'] += 1
t_row = truth[user_id]Also applies to: 29-33, 39-43
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@code/evaluation/main.py` around lines 20 - 21, The accuracy calculation uses
total row count as the denominator, but rows with missing or unknown user_ids
are skipped during evaluation, causing accuracy to be underreported. Create a
separate evaluated_total counter that only increments when a row is actually
evaluated (not skipped), and use evaluated_total as the denominator for accuracy
calculations instead of total. Apply this same pattern to all accuracy
calculations in the file (the sections around lines 29-33 and 39-43 that also
have similar skip logic for rows).
1 participant
🎯 What: The testing gap addressed is the missing evaluation code and its corresponding tests in
code/evaluation/main.py. This ensures accuracy in metrics which is critical for the pipeline.📊 Coverage: The scenarios now tested cover handling missing prediction/ground truth files, handling empty files to avoid division-by-zero, and correctly computing counts and accuracy percentages for the
claim_status,issue_type,object_part, andseverityfields.✨ Result: The system now has a reliable, thoroughly tested function to evaluate the quality of predictions against ground truth labels, increasing confidence in metric reporting.
PR created automatically by Jules for task 2116392416809784904 started by @NITISH-R-G
Summary by Sourcery
Add evaluation logic for comparing prediction CSVs against ground truth and validate it with unit tests.
New Features:
Tests: