feat: enhance Civic Intelligence Engine delta and emerging concern logic#491
feat: enhance Civic Intelligence Engine delta and emerging concern logic#491RohanExploit wants to merge 2 commits intomainfrom
Conversation
- Updated `backend/civic_intelligence.py` to calculate the Civic Intelligence Index delta and properly assign `top_emerging_concern` based on the highest category percentage growth. - Added a new suite of unit tests in `backend/tests/test_civic_intelligence_delta.py` that fully cover logic around index deltas and emerging concerns.
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
✅ Deploy Preview for fixmybharat canceled.
|
🙏 Thank you for your contribution, @RohanExploit!PR Details:
Quality Checklist:
Review Process:
Note: The maintainers will monitor code quality and ensure the overall project flow isn't broken. |
📝 WalkthroughWalkthroughThis PR enhances the Civic Intelligence Index with day-over-day delta tracking and improves emerging concern detection by computing category growth metrics during spike detection, complemented by comprehensive unit tests for the new functionality. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested labels
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
Enhances the Daily Civic Intelligence Refinement Engine to (a) identify top_emerging_concern using growth-based comparisons and (b) enrich the generated snapshot’s civic_index with day-over-day delta and delta_str, with accompanying tests covering the new behaviors.
Changes:
- Add growth-based emerging-concern selection and persist
trends.top_emerging_concernin the daily snapshot. - Extend
civic_indexgeneration to compute and emitdeltaanddelta_strfrom the previous snapshot score. - Add a new pytest suite to validate emerging concern growth ranking and civic index delta output.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| backend/civic_intelligence.py | Implements growth-based spike/emerging concern logic and adds civic index delta fields derived from the prior snapshot. |
| backend/tests/test_civic_intelligence_delta.py | Adds tests that mock filesystem/DB dependencies to validate emerging concern growth logic and civic index delta formatting. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| category_growth[category] = increase | ||
| elif prev_count == 0 and count > 5: | ||
| spikes.append(category) # New surge | ||
| # Represent "infinite" growth for new surges by assigning a high placeholder | ||
| # or based on pure volume, for ranking emerging concerns | ||
| category_growth[category] = float(count) | ||
|
|
||
| trends['spikes'] = spikes | ||
| # Find the top emerging concern based on highest percentage growth | ||
| top_emerging = "None" | ||
| if category_growth: | ||
| top_emerging = max(category_growth, key=category_growth.get) |
There was a problem hiding this comment.
category_growth stores percent-growth ratios for existing categories (increase) but uses float(count) for new surges (prev_count == 0). That mixes units (ratio vs absolute count), so max(category_growth, key=...) can pick a “top emerging concern” based on incomparable values. Use a consistent metric (e.g., treat new surges as float('inf')/a very large sentinel, or store a tuple like (is_new, increase, count) and compare deterministically).
| category_growth[category] = increase | |
| elif prev_count == 0 and count > 5: | |
| spikes.append(category) # New surge | |
| # Represent "infinite" growth for new surges by assigning a high placeholder | |
| # or based on pure volume, for ranking emerging concerns | |
| category_growth[category] = float(count) | |
| trends['spikes'] = spikes | |
| # Find the top emerging concern based on highest percentage growth | |
| top_emerging = "None" | |
| if category_growth: | |
| top_emerging = max(category_growth, key=category_growth.get) | |
| # Store as (is_new, percent_increase, current_count) for consistent comparison | |
| category_growth[category] = (0, increase, count) | |
| elif prev_count == 0 and count > 5: | |
| spikes.append(category) # New surge | |
| # Represent new surges explicitly; they outrank existing spikes via is_new flag | |
| category_growth[category] = (1, 0.0, count) | |
| trends['spikes'] = spikes | |
| # Find the top emerging concern based on highest percentage growth | |
| top_emerging = "None" | |
| if category_growth: | |
| # Select category with highest (is_new, percent_increase, count) tuple | |
| top_emerging = max(category_growth.items(), key=lambda item: item[1])[0] |
| if delta is not None: | ||
| result["delta"] = delta | ||
| # Format string delta like "+3.1 from yesterday" | ||
| sign = "+" if delta >= 0 else "" | ||
| result["delta_str"] = f"{sign}{delta} from yesterday" | ||
|
|
There was a problem hiding this comment.
delta_str can become malformed for small negative deltas that round to -0.0 (Python keeps the sign), because the sign is computed from delta >= 0 and then interpolated with the negative-zero value (resulting in "+-0.0 from yesterday"). Normalize delta to 0.0 when it rounds to zero (or format using an explicit sign formatter like f"{delta:+.1f}" after normalization).
| import os | ||
| from unittest.mock import MagicMock, patch, mock_open | ||
| from datetime import datetime, timezone |
There was a problem hiding this comment.
This test module includes unused imports (os, datetime, timezone), which adds noise and can trip linting if enabled. Remove them or use them in the tests.
| import os | |
| from unittest.mock import MagicMock, patch, mock_open | |
| from datetime import datetime, timezone | |
| from unittest.mock import MagicMock, patch, mock_open |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (3)
backend/civic_intelligence.py (2)
189-189: Use explicitOptionaltype hint.Per PEP 484, a parameter with a
Nonedefault should have an explicit optional type. The static analysis tool correctly flagged this.- def _calculate_index(self, db: Session, issues_24h: List[Issue], trends: Dict[str, Any], previous_snapshot: Dict[str, Any] = None) -> Dict[str, Any]: + def _calculate_index(self, db: Session, issues_24h: List[Issue], trends: Dict[str, Any], previous_snapshot: Dict[str, Any] | None = None) -> Dict[str, Any]:Or if targeting Python <3.10:
from typing import Optional # ... previous_snapshot: Optional[Dict[str, Any]] = None🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@backend/civic_intelligence.py` at line 189, The parameter previous_snapshot in the _calculate_index method has a default of None but lacks an explicit Optional type; update the signature of _calculate_index to annotate previous_snapshot as Optional[Dict[str, Any]] (and add "from typing import Optional" if not already imported) so the type correctly reflects a nullable dictionary.
75-79: Clarify the growth placeholder heuristic for new surges.Using
float(count)as a growth proxy for new categories (whereprev_count == 0) creates an implicit weighting scheme: a new surge with 6+ issues will always rank higher than an existing category with <500% growth. If this prioritization is intentional (new categories are more alarming), consider documenting it. Otherwise, a consistent approach could use a fixed high constant (e.g.,float('inf')or1000.0) to clearly signal "new surge always wins."🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@backend/civic_intelligence.py` around lines 75 - 79, The code path handling new surges (elif prev_count == 0 and count > 5) currently sets category_growth[category] = float(count), which implicitly weights new categories by raw volume and can unfairly outrank large-percentage growths; change this to use a clear, consistent placeholder (for example float('inf') or a defined constant like NEW_SURGE_WEIGHT = 1000.0) and add a one-line comment explaining the chosen heuristic so the behavior is explicit; update the assignment in the branch that appends to spikes and set category_growth[category] to the new constant instead of float(count).backend/tests/test_civic_intelligence_delta.py (1)
1-12: Unused imports.
datetimeandtimezone(line 5) andos(line 3) are imported but not directly used in the test code. The engine uses its own internal datetime. Consider removing unused imports.import pytest import json -import os from unittest.mock import MagicMock, patch, mock_open -from datetime import datetime, timezone from backend.models import Issue from backend.civic_intelligence import CivicIntelligenceEngine🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@backend/tests/test_civic_intelligence_delta.py` around lines 1 - 12, Remove the unused imports in the test module: delete the os import and the datetime, timezone imports from the top of backend/tests/test_civic_intelligence_delta.py so only required imports remain (keep pytest, json, and the unittest.mock imports as needed) and ensure the CivicIntelligenceEngine fixture function name engine() and referenced symbols (CivicIntelligenceEngine, Issue) remain unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@backend/civic_intelligence.py`:
- Line 189: The parameter previous_snapshot in the _calculate_index method has a
default of None but lacks an explicit Optional type; update the signature of
_calculate_index to annotate previous_snapshot as Optional[Dict[str, Any]] (and
add "from typing import Optional" if not already imported) so the type correctly
reflects a nullable dictionary.
- Around line 75-79: The code path handling new surges (elif prev_count == 0 and
count > 5) currently sets category_growth[category] = float(count), which
implicitly weights new categories by raw volume and can unfairly outrank
large-percentage growths; change this to use a clear, consistent placeholder
(for example float('inf') or a defined constant like NEW_SURGE_WEIGHT = 1000.0)
and add a one-line comment explaining the chosen heuristic so the behavior is
explicit; update the assignment in the branch that appends to spikes and set
category_growth[category] to the new constant instead of float(count).
In `@backend/tests/test_civic_intelligence_delta.py`:
- Around line 1-12: Remove the unused imports in the test module: delete the os
import and the datetime, timezone imports from the top of
backend/tests/test_civic_intelligence_delta.py so only required imports remain
(keep pytest, json, and the unittest.mock imports as needed) and ensure the
CivicIntelligenceEngine fixture function name engine() and referenced symbols
(CivicIntelligenceEngine, Issue) remain unchanged.
This submission improves the Daily Civic Intelligence Refinement Engine by allowing the system to accurately highlight the
top_emerging_concernusing mathematically calculated volume growth comparisons (instead of just pure volume). It also enriches the generated snapshot'scivic_indexwithdeltaanddelta_strattributes (e.g.+3.1 from yesterday) to help track system performance and user behavior trends effectively.Also included is a newly created test suite,
test_civic_intelligence_delta.pywhich mocks out filesystem, caching, and DB components to confidently test the new mathematical growth calculations over simulated consecutive daily operations. All tests execute successfully.PR created automatically by Jules for task 3734117019519320683 started by @RohanExploit
Summary by cubic
Improve the Daily Civic Intelligence Engine to rank emerging concerns by percentage growth and to show day-over-day index changes. Adds tests for growth logic and delta formatting.
Written for commit 8ed2d12. Summary will update on new commits.
Summary by CodeRabbit