Skip to content

feat: Migrate E2E test runner to Python with HTML report generation#5

Merged
abhraina-aws merged 53 commits intoabhraina-aws:testsfrom
abhishekanne:tests
Sep 16, 2025
Merged

feat: Migrate E2E test runner to Python with HTML report generation#5
abhraina-aws merged 53 commits intoabhraina-aws:testsfrom
abhishekanne:tests

Conversation

@abhishekanne
Copy link

@abhishekanne abhishekanne commented Aug 26, 2025

🚀 Overview

Migrated the E2E test framework from shell scripts to a comprehensive Python-based test runner with advanced reporting capabilities.

✨ Key Features

Python Test Runner (run_tests.py)

  • Feature-based test execution with configurable categories
  • Real-time progress tracking with per-test feedback
  • Parallel test suite support (sanity/regression)
  • Individual test result parsing from cargo output
  • System information collection (OS, Q CLI version)

HTML Report Generation

  • Interactive HTML reports with collapsible sections
  • Visual success rate charts and statistics
  • Detailed test output capture with syntax highlighting
  • Test categorization by feature groups
  • Timestamped reports in dedicated reports/ folder

Enhanced Test Structure

  • Test descriptions added to few test cases for better reporting
  • Improved output formatting with emojis and status indicators
  • **Project files restructuring ** moved to tests to respective mods

📊 Reporting Capabilities

  • Success rate visualization with pie charts
  • Feature-wise breakdown of test results
  • Individual test output with full cargo logs
  • System environment details for debugging
  • JSON and HTML export formats

📁 Files Changed

  • Added: run_tests.py - New Python test runner
  • Enhanced: All test files with descriptions and improved formatting
  • Updated: Cargo.toml with feature organization
  • Added: .gitignore for reports and build artifacts

🧪 Testing

  • ✅ All existing tests pass with new runner
  • ✅ HTML reports generate successfully
  • ✅ Feature categorization group (directory) wise and sanity ,regression works correctly

shreyabhagat1 and others added 16 commits August 21, 2025 19:08
@shreyabhagat1
Refactor: Merge save/load into one file, remove verbose from MCP context
4b990d5
@shreyabhagat1
Remove failing MCP test case
c1c6756
@abhishekanne
Created folder structure for e2e test
5f3e986
@abhishekanne
Fixed the warnings in code
ceb7a6e
@abhishekanne
Modified the tests features to support multiple features
a192763
@sayemaanjum0412
QCLI Automation for core_session,integration,mcp,model,session_mgmt t…
27131f1 
…ests
@sayemaanjum0412
Reduce read_response timeout from 20s to 4s
d65314c
@shreyabhagat1
Refactor: Remove verbose from MCP context and save/load and fixed the…
6da0041 
… failing test case for adding multiple file as context.
@abhishekanne
Migrated to python script with HTML report generation
571ce1d
@shreyabhagat1
Add new tests cases test_add_and_remove_mcp_command and test_q_mcp_st…
6931157 
…atus_command
@abhishekanne
added html template
021762a
@abhishekanne
Test wise console output
0c281dc
@abhishekanne
Test wise console output for single thread
1ca00cd
@abhishekanne
Move reports to reports folder
4018f06
@abhishekanne
Script updated to support test description
f2b936a
@abhishekanne
Added description to agent and usage command tests , add .gitignore
ebf882b
@abhishekanne
Copy link
Author

abhishekanne commented Aug 26, 2025

@abhraina-aws The code is under testing for stability.

shreyabhagat1 and others added 24 commits September 1, 2025 18:36
@shreyabhagat1
QCLI Automation for tools
a4e3f2f
@sayemaanjum0412
move common functions from lib to individual test files to fix interm…
0936edd 
…ittent /quit issue, fix MCP failures, and add compact feature tests
@sayemaanjum0412
add compact new tests and fix chat function for help,clear,ai_prompts…
ae4d39d 
… tests
@shreyabhagat1
Add new test cases related to tools and fix the warning
9ee973a
@abhishekanne
Updated help section of script
d7fdcf5
@abhishekanne
Added executng animation while executing cargo command
4b8722a
@abhishekanne
Improved quiet mode
f87ff99
@shreyabhagat1
Add detail descriptions to 5 test cases related to tools.
24035aa
@sayemaanjum0412
add new tests for editor feature
b4b71af
@abhishekanne
Fixed quiet mode
b1ff5f1
@abhishekanne
Fixed quiet mode with logs
9552cee
@sayemaanjum0412
add new tests for editor,compact feature and description for editor ,…
55af456 
… comoact feature test cases
@abhishekanne
Fixed order of features
9b9ae7f
@shreyabhagat1
Add Q subcommand support and corresponding tests
8f81460 
- Introduced `q_subcommand` feature in Cargo.toml
- Implemented functions for starting and executing Q subcommands in lib.rs
- Added tests for `q chat`, `q doctor`, and `q translate` commands
- Updated all_tests.rs to include the new q_subcommand tests
@sayemaanjum0412
added new test case for compact feature
ff47f0d
@abhishekanne
Improved animation
4bfd154
@sayemaanjum0412
Add regression tests for MCP feature
7387c6e
@abhishekanne
Modified the report css for code
f265c84
@shreyabhagat1
Refactor Q CLI subcommand execution and enhance tests
e1d7f33 
- Replaced `start_q_subcommand` and `read_session_response` with `execute_q_subcommand` and `execute_q_subcommand_with_stdin` for improved command execution and response handling.
- Added new tests for `q mcp` subcommands, verifying help commands and functionality.
- Enhanced existing test descriptions for clarity and consistency.
@sayemaanjum0412
Add tests for q settings and q whoami subcommands
8e90c22
@shreyabhagat1
Add tests for q debug and q inline subcommands
aeb975b
@shreyabhagat1
Refactor test descriptions to use HTML code tags for command formatting
6e0903e 
- Updated test output messages in various test files to wrap command names in <code> tags for better readability.
@sayemaanjum0412
Enhance MCP subcommand tests and add q debug build related tests
265366e
@shreyabhagat1
Add help subcommand tests for q inline commands and Add todos command…
cd74a72 
… support and enhance related tests
@abhraina-aws abhraina-aws merged commit ec5d631 into abhraina-aws:tests Sep 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@abhishekanne @abhraina-aws @shreyabhagat1 @sayemaanjum0412