Test: Validate Telemetry in PR Flow (Phase 1 Evaluation) #9
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…hPro - Enhanced workflow to use single-step run-ci command - Added new functions with style and correctness issues - Test integrated Denis analyzer + LLM pipeline - Focus analysis on changed files only for efficiency
- Added patchpro-artifacts/ to .gitignore - Removed tracked artifact files that should be generated at runtime - Artifacts will be properly ignored going forward
- Added fallback strategies for git diff when no merge base exists - Better handling of PR file detection across different branch histories - Should now properly detect example.py changes for PatchPro analysis
- Set fetch-depth: 0 to get full git history in GitHub Actions - Simplified changed files detection logic - Should now properly detect example.py changes - Added debug output to show number of changed files
- Added test comment to example.py - Should now run complete E2E pipeline with LLM patch generation - Test the full integrated Denis analyzer + LLM system
- Only add source code files (*.py, *.js, etc.) not artifacts - Explicitly exclude artifact/ and changed_files_only/ directories - This prevents pollution of git history with generated files
- Removed artifact/ and changed_files_only/ directories from git tracking - Added changed_files_only/ to .gitignore - These files should only exist during workflow execution, not in git history - Prevents git repository pollution with generated files
- Should now run complete pipeline with LLM patch generation - Workflow will not commit any artifacts to git - Test the full integrated Denis analyzer + LLM + patch application
- Use --tools ruff --tools semgrep instead of --tools ruff semgrep - Fix CLI argument parsing error that was causing workflow failure
- This should trigger a pull_request event (not workflow_dispatch) - Should post complete comment with generated patches visible - Test the full E2E with comment posting
- Added back unused imports (json, os) - Reverted to == None instead of is None - Reverted to bare except - This commit will trigger post-commit analysis
- Multiple imports on one line (I001) - SQL injection vulnerabilities (semgrep) - Hardcoded secrets (S105, S106) - Missing docstrings (D) - Unsafe file operations (PTH123)
Add comment header to test file to trigger new workflow run. This run should now: - Load .patchpro.toml successfully - Enable agentic mode from config - Create telemetry traces - Upload artifacts with traces.db Testing Phase 1 - Config-driven agentic mode enablement
Add debug step to list .patchpro directory contents before upload. This will help diagnose why telemetry traces are created but artifacts upload reports 'No files were found'. PatchTracer logs show it initialized 4 times and created traces at: /home/runner/work/patchpro-demo-repo/patchpro-demo-repo/.patchpro/traces But upload-artifact action can't find files. This debug step will show exactly what exists in the directory before upload.
The workflow failed because github.base_ref is empty when triggered via
workflow_dispatch (manual trigger), causing git diff command to fail:
git diff --name-only 'origin/...HEAD' # ❌ Invalid
Add fallback to 'demo/patchpro-ci-test' for workflow_dispatch events:
BASE_REF="${{ github.base_ref || 'demo/patchpro-ci-test' }}"
This ensures the workflow works for both:
- pull_request events: Uses actual PR base branch
- workflow_dispatch events: Uses hardcoded fallback branch
This was blocking telemetry testing because analyze-pr exited early
with no files to analyze, so AgenticPatchGeneratorV2 was never invoked.
PatchPro Bot Enhanced ReportGenerated on: /home/runner/work/patchpro-demo-repo/patchpro-demo-repo/.patchpro Summary
Performance MetricsProcessing Statistics
Cache Performance
Scalability Features Used
Findings BreakdownBy Severity
By Tool
By Category
Generated Patches
Combined Patch
Affected Files
|
ezeanyicollins
requested review from
waigisteve
and removed request for
waigisteve
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Testing Phase 1 of Evaluation Framework - Telemetry in CI/CD environment
What This Tests
Changes
New workflow:
.github/workflows/patchpro-agent-dev-test.ymlpatchpro-bot@agent-dev(not main)analyze-prcommand (replaces run_ci.py).patchpro.tomlTest file:
test_code_quality.pyExpected Behavior
traces.dband JSON trace filesSuccess Criteria
traces.dbcontains patch attempt dataRelated
docs/PATH_TO_MVP.md(Week 1, Phase 1)TELEMETRY_PR_TEST_PLAN.md