feat(pyats): add tag-based filtering for pyATS tests by oboehmer · Pull Request #657 · netascode/nac-test

oboehmer · 2026-03-17T21:19:40Z

Description

This PR adds tag-based filtering support for PyATS tests using the --include and --exclude CLI options, bringing PyATS test selection in line with Robot Framework's tag filtering capabilities. PyATS tests are filtered based on their groups class attribute using Robot Framework's TagPatterns API for consistent pattern matching across both frameworks.

Closes

Fixes [feat] support filtering pyats tests using tags #436

Related Issue(s)

N/A

Type of Change

Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Refactoring / Technical debt (internal improvements with no user-facing changes)
Documentation update
Chore (build process, CI, tooling, dependencies)
Other (please describe):

Test Framework Affected

PyATS
Robot Framework
Both
N/A (not test-framework specific)

Network as Code (NaC) Architecture Affected

Platform Tested

nac-test supports macOS and Linux only

macOS (version tested: )
Linux (distro/version tested: )

Key Changes

Core Tag Filtering Implementation

New TagMatcher class (tag_matcher.py): Wraps Robot Framework's TagPatterns API to provide consistent tag matching behavior for both Robot and PyATS tests
- Supports all Robot Framework tag pattern syntax: simple tags, wildcards (bgp*), boolean operators (healthANDbgp, bgpORospf, bgpNOTnrfu)
- Case-insensitive matching with underscore normalization
- Include/exclude patterns are private; no public instance state
Refactored test discovery architecture (test_discovery.py, test_type_resolver.py):
- Renamed TestTypeResolver to TestMetadataResolver to reflect expanded responsibilities
- AST-based extraction of groups attribute from PyATS test classes without importing
- Pre-filtering during discovery phase eliminates subprocess overhead
- Combined discovery and categorization into single pass returning PyatsDiscoveryResult
- TestMetadataResolver methods converted to @staticmethod; no instantiation required. Module-level logger replaces self.logger, consistent with the rest of the codebase
Updated dataclasses (types.py):
- TestFileMetadata: Structured representation of test file metadata (path, type, groups)
- PyatsDiscoveryResult: Discovery results container with computed properties (total_count, all_tests, api_paths, d2d_paths). The former test_type_by_path derived dict and get_test_type() method have been removed -- type information is now carried forward through the execution pipeline rather than reconstructed after the fact (see Execution Pipeline below)
Execution pipeline (orchestrator.py, output_processor.py):
- OutputProcessor receives a test_type_by_path map at construction time and stamps test_type directly into each test_info entry at task_start time
- The post-execution split loop reads test_info["test_type"] directly -- no external path lookup needed
- This removes the design smell where type information was known at discovery but dropped and re-derived after execution
- A comment in the orchestrator documents the deeper future fix: separate OutputProcessor/test_status instances per execution type
Orchestrator integration (orchestrator.py):
- Improved error messaging: "No pyATS tests matching tag filter (exclude: 'bgp OR ospf')" when all tests filtered out
- Distinguishes between "no test files" vs "no tests matching filter"
- Discovery print restored to show breakdown: "Discovered N PyATS test files (M api, K d2d)"
Documentation (README.md, PRD_AND_ARCHITECTURE.md):
- Updated --include/--exclude CLI option descriptions
- Added PyATS groups attribute usage examples
- Tag pattern syntax reference

Testing

New Tests for Tag Filtering

E2E tests (tests/e2e/):
- 4 new scenarios covering include/exclude combinations
- Fixtures with both Robot and PyATS tests tagged for filtering validation
- Tests verify correct exit codes and test counts for various filter patterns
- Clarified scenario descriptions: exit 252 ("no tests found") is only raised when the combined total across all frameworks is zero
Unit tests (tests/unit/pyats_core/discovery/):
- test_tag_matcher.py: Tag pattern matching, boolean logic, wildcards, edge cases
- test_metadata_resolver.py: AST-based test type detection, directory fallback, error handling
- test_groups_extraction.py: Groups attribute extraction for tag filtering
- Updated orchestrator tests to use new PyatsDiscoveryResult structure
New unit tests for previously untested code:
- tests/unit/pyats_core/common/test_types.py: PyatsDiscoveryResult computed properties (total_count, all_tests, api_paths, d2d_paths)
- tests/unit/pyats_core/execution/test_output_processor.py: test_type stamping for known api/d2d paths, unknown paths, absent map, and missing test_file in event

Test Suite Refactoring (Independent of Feature)

Consolidation: Integration to Unit Tests

Moved 42 non-integration tests from test_discovery_integration.py to new tests/unit/pyats_core/discovery/test_test_discovery.py
Retained only TestDiscoveryPerformance as the true integration test (validates timing with 100+ files)
Result: Integration tests reduced from 43 to 1

Parametrization

Consolidated 39 individual test methods in test_tag_matcher.py into 15 parametrized tests (86 total cases)
Applied pytest parametrization per project testing guidelines

Code Quality

Added proper type annotations for all test parameters
Fixed mypy errors for generic types (list to list[str])
Added MockerFixture type hints for pytest-mock fixtures
Flattened nested with patch.object(...): blocks using Python 3.10+ parenthesised multi-target with statements

Note: This refactoring improves test maintainability and follows project testing guidelines but is not strictly required for the tag filtering feature.

Testing Done

Unit tests added/updated
- tests/unit/pyats_core/discovery/: tag matcher, metadata resolver, groups extraction
- tests/unit/pyats_core/common/test_types.py: PyatsDiscoveryResult computed properties
- tests/unit/pyats_core/execution/test_output_processor.py: test_type stamping behaviour
- Updated orchestrator tests to use new PyatsDiscoveryResult structure
E2E tests added
- 4 tag filtering scenarios validating include/exclude combinations
Manual testing performed:
- PyATS tests executed successfully
- Robot Framework tests executed successfully
- D2D/SSH tests executed successfully (if applicable)
- HTML reports generated correctly
All existing tests pass (pytest / pre-commit run -a)

Test Commands Used

# Run all unit and integration tests
uv run pytest -n auto --dist loadscope

# Test tag filtering - include specific tag
nac-test -d ./tests/e2e/fixtures/tag_filtering -t ./tests/e2e/fixtures/tag_filtering/templates -o ./output --include bgp

# Test tag filtering - exclude specific tag
nac-test -d ./tests/e2e/fixtures/tag_filtering -t ./tests/e2e/fixtures/tag_filtering/templates -o ./output --exclude ospf

# Test tag filtering - boolean pattern
nac-test -d ./data -t ./tests -o ./output --include "healthORbgp" --exclude nrfu

# Test no matches (exits 0 when Robot tests still run)
nac-test -d ./tests/e2e/fixtures/tag_filtering -t ./tests/e2e/fixtures/tag_filtering/templates -o ./output --exclude "bgpORospf"

Checklist

Code follows project style guidelines (pre-commit run -a passes)
Self-review of code completed
Code is commented where necessary (especially complex logic)
Documentation updated (if applicable)
No new warnings introduced
Changes work on both macOS and Linux
CHANGELOG.md updated (if applicable)

Screenshots (if applicable)

N/A

Additional Notes

Implementation Approach

The tag filtering implementation leverages Robot Framework's TagPatterns API to ensure consistent behavior between Robot and PyATS test selection. Key design decisions:

PyATS groups attribute: Uses the native pyATS mechanism (simple list of strings on test classes) for tagging

class VerifyBgpNeighbors(SDWANTestBase):
    groups = ["bgp", "routing"]  # Matches --include bgp or --include routing

Discovery-phase filtering: Tests are filtered during the discovery phase via AST parsing, avoiding subprocess overhead and enabling early validation
Single-pass architecture: Discovery and categorization combined into one pass that returns a PyatsDiscoveryResult with pre-computed properties for efficient access
Type information carried forward: test_type is stamped into each result entry at the moment it is first observed (task_start), rather than being re-derived from path lookups after execution completes
Consistent pattern formatting: Uses Robot Framework's TagPatterns string representation for user-facing messages ('bgpORospf' to 'bgp OR ospf')

Breaking Changes

None. The changes are backward-compatible:

PyATS tests without groups attribute remain unaffected and run as before
Existing --include/--exclude behavior for Robot Framework unchanged
All existing test repositories continue to work without modification

Migration Notes

PyATS tests can optionally add a groups class attribute to enable tag-based filtering
The groups attribute should be a list of strings (native pyATS format)
Tests without groups are treated as having no tags and are included by default (unless an --include pattern is specified)

Extend --include and --exclude CLI options to filter pyATS tests based on their `groups` class attribute using Robot Framework tag pattern semantics. Changes: - Add TagMatcher class wrapping Robot's TagPatterns API for consistent tag matching between Robot and pyATS tests - Add TestMetadataResolver for AST-based extraction of test type and groups from pyATS test files without importing them - Add TestFileMetadata and TestExecutionPlan dataclasses to types.py for structured test discovery results - Integrate tag filtering into test discovery pipeline Refactor test discovery architecture: - Remove caching from TestMetadataResolver (previously TestTypeResolver) - Combine discovery and categorization into single pass returning TestExecutionPlan with pre-computed test_type_by_path mapping - Orchestrator now uses execution plan for O(1) type lookups during post-execution result splitting instead of re-instantiating resolver Tag pattern syntax (from Robot Framework): - Simple tags: 'health', 'bgp', 'ospf' - Wildcards: 'bgp*', '?est' - Boolean: 'healthANDbgp', 'healthORbgp', 'healthNOTnrfu' - Case-insensitive, underscores ignored Example usage: nac-test -d data/ -t tests/ -o out/ --include health --exclude nrfu --- feat(pyats): improve "no tests found" message when filtered by tags When all pyATS tests are filtered out by tag patterns, the previous message "No PyATS test files (*.py) found" was misleading — files existed but were excluded. Now displays a descriptive message like: No pyATS tests matching tag filter (exclude: 'bgp OR ospf') --- test(e2e): add tag filtering e2e scenarios for Robot and pyATS Add 4 e2e scenarios verifying --include/--exclude tag filtering works correctly for both Robot and pyATS tests. - TAG_FILTER_INCLUDE: --include bgp → 1 Robot + 1 PyATS pass - TAG_FILTER_EXCLUDE: --exclude ospf → 1 Robot + 1 PyATS pass - TAG_FILTER_COMBINED: --include api-only → 0 Robot (exit 252) + 1 PyATS pass - TAG_FILTER_NO_MATCH: --exclude bgpORospf → 0 Robot + 0 PyATS (exit 252) --- docs: update --include/--exclude documentation for pyATS tag filtering --- fix(pyats): use absolute() instead of resolve() for symlink-safe paths Replace Path.resolve() with Path.absolute() to preserve symlinks and prevent ValueError when using relative_to() on paths that resolve differently than their absolute form.

Refactor discovery tests for better maintainability: - Move test_test_type_resolver.py to tests/unit/pyats_core/discovery/ - Extract helpers.py with shared FIXTURES_DIR and create_mock_path - Split groups extraction tests into separate test_groups_extraction.py - Parametrize tests to reduce duplication (475→301 lines in main test file) - Use _UNUSED_TEST_ROOT for tests using mock paths vs FIXTURES_DIR for real fixtures Rename TestExecutionPlan to PyatsDiscoveryResult for clarity: - Better reflects its purpose as discovery output, not execution context - Update all references in discovery module and tests

…ports - Add frozen=True and slots=True to PyatsDiscoveryResult for immutability - Convert properties to cached_property for better performance - Clarify skipped_files vs filtered_by_tags semantics in docstring - Remove stale re-exports from discovery __init__.py and test_type_resolver - Update test imports to use canonical locations (common.types) - Rename execution_plan → discovery_result in orchestrator for consistency

…ization Refactor the PyATS discovery test suite to eliminate redundant coverage, consolidate overlapping tests, and apply pytest parametrization for improved maintainability. ## Changes ### Phase 1: Consolidate integration tests into unit tests - Move non-integration tests from test_discovery_integration.py to new tests/unit/pyats_core/discovery/test_test_discovery.py: - TestDiscoveryFiltering (5 parametrized tests) - TestRelaxedPathRequirements (4 parametrized tests) - TestExcludePaths (1 parametrized test) - TestErrorHandling (2 tests for unreadable file handling) - Retain only TestDiscoveryPerformance in test_discovery_integration.py as it is the only true integration test (validates discovery timing with 100+ files) ### Phase 2: Parametrize test_tag_matcher.py Consolidate 39 individual test methods into 15 parametrized tests: - TestBasicMatching: Consolidate has_filters tests (4 → 1) - TestIncludePatterns: Consolidate include tests (4 → 1, 13 cases) - TestExcludePatterns: Consolidate exclude tests (4 → 1, 14 cases) - TestCombinedPatterns: Consolidate combined tests (2 → 1, 8 cases) - TestRobotPatternSemantics: Split into boolean + wildcard (7 → 2) - TestEdgeCases: Consolidate edge case tests (9 → 4) - TestFormatFilterDescription: Consolidate formatting tests (9 → 3) ### Code quality - Add proper type annotations for all test parameters - Fix mypy errors for generic types (list → list[str]) - Add MockerFixture type hints for pytest-mock fixtures ## Test Coverage Metrics ### Before refactor | File | Tests | Coverage | |--------------------------------|-------|----------| | test_discovery_integration.py | 43 | - | | test_tag_matcher.py | 39 | 100% | | test_discovery.py (source) | - | 90% | ### After refactor | File | Tests | Coverage | |--------------------------------|-------|----------| | test_discovery_integration.py | 1 | - | | test_test_discovery.py (new) | 12 | - | | test_tag_matcher.py | 15 | 100% | | test_discovery.py (source) | - | 95% | | tag_matcher.py (source) | - | 100% | ### Summary - Integration tests: 43 → 1 (moved 42 to unit tests) - test_tag_matcher.py methods: 39 → 15 (86 parametrized cases) - Total discovery test cases: 145 passed - Overall discovery module coverage: 85% ## Validation - All pre-commit hooks pass (ruff, mypy, license headers) - All 145 tests pass - No coverage regression

Add pytest configuration to eliminate collection warnings for source classes in nac_test/ that start with "Test" prefix (e.g., TestDiscovery, TestResults, TestFileMetadata). Changes: - Add nac_test to norecursedirs to prevent pytest from recursing into source directory during test collection - Add filterwarnings with explicit regex matching the 10 Test* classes to suppress warnings when they are imported by test files The regex explicitly names each class rather than using a wildcard to avoid accidentally suppressing warnings for actual test issues. Ref: #650

Replace relative imports with absolute imports in regular modules for consistency with the rest of the codebase. Relative imports are kept only in __init__.py files for re-exports. Changed files: - broker/connection_broker.py - discovery/test_discovery.py - execution/device/device_executor.py

…ltering

… patterns private has_filters leaked an implementation detail to callers, which needed to guard should_include() even though that method already handles the no-filter case correctly. Remove has_filters entirely and make include_patterns/exclude_patterns private, so should_include() is the single unconditional API callers need. Simplify the call site in TestDiscovery accordingly.

…nario config The robot_results/ directory is always created when pabot is invoked, even when tag filters match zero tests. The previous has_robot_tests (expected_robot_total > 0) conflated "pabot ran" with "robot produced test output", causing directory-existence assertions to fail for tag-filtering scenarios with 0 robot results. Introduce robot_invoked on E2EScenario (derived from .robot files in the fixture, no manual field) and robot_dir_exists on E2EResults, parallel to the existing is_dry_run/has_pyats_*_results pattern. Use robot_dir_exists for directory-presence checks only; all content assertions (output.xml, log.html, xunit) keep has_robot_results. The expected_exit_code change in TAG_FILTER_COMBINED_SCENARIO is a consequence of fixing #643

…alls

@staticmethod

…aResolver to static methods PyatsDiscoveryResult now builds test_type_by_path automatically in __post_init__ from api_tests + d2d_tests, eliminating the redundant manual dict construction in callers. TestMetadataResolver no longer holds instance state: self.logger is replaced with a module-level logger (consistent with the rest of the codebase), __init__ is removed, and all methods are @staticmethod. The instantiation in discover_pyats_tests and all test files is gone.

skipped_files was never read by any caller — on main the orchestrator unpacked it from the tuple return and immediately discarded it. The logging of skipped files already happens inside discover_pyats_tests itself, which is the right place for that information. Remove the field from PyatsDiscoveryResult, the local accumulator from discover_pyats_tests, and the associated unit test assertions.

…ltering

…ted patch blocks - Restore "(N api, M d2d)" breakdown in the "Discovered N PyATS test files" print that was dropped when the line was moved before the dry-run block - Convert PyatsDiscoveryResult.test_type_by_path from __post_init__ eager dict to @cached_property; add staleness note documenting why it is safe - Add DEFAULT_TEST_TYPE type annotation; use it in TestMetadataResolver fallback warning instead of a bare "api" string literal - Flatten nested with patch.object(...) blocks in test_orchestrator_dry_run using Python 3.10+ parenthesised multi-target with statements - Fix typo "conistency" → "consistency" in E2EScenario.validate docstring - Clarify TAG_FILTER_COMBINED_SCENARIO description: Robot=0 but PyATS=1 means combined exit is 0, not 252; add explanatory comments - Rename test_categorization_performance → test_discovery_performance to match what the test actually exercises after the discovery refactor

…f post-hoc path lookup Previously, OutputProcessor stored test_file in each test_info entry and the orchestrator re-derived test_type after execution by looking up each path in PyatsDiscoveryResult.test_type_by_path. This was a design smell: the type was known at discovery time but dropped and reconstructed later. Now the orchestrator builds a path→type map from discovery_result.all_tests and passes it to OutputProcessor, which stamps test_type directly into each test_info entry at task_start time. The split loop reads test_info["test_type"] with no external lookup needed. - Remove test_type_by_path cached_property and get_test_type() from PyatsDiscoveryResult; remove now-unused cached_property import - Add test_type_by_path parameter to OutputProcessor; stamp test_type into test_info on task_start using DEFAULT_TEST_TYPE as fallback - Use DEFAULT_TEST_TYPE constant instead of bare "api" literal in fallback - Add comment in orchestrator noting the deeper fix (separate OutputProcessor/test_status per execution type) as a future path forward - Fix stale comment: "no tests after categorization" → "after device inventory check" (categorization now happens during discovery) - Fix cross-file line number reference in split-loop comment; reference method name instead - Add tests/unit/pyats_core/common/test_types.py covering PyatsDiscoveryResult computed properties (total_count, all_tests, api_paths, d2d_paths) - Add tests/unit/pyats_core/execution/test_output_processor.py covering test_type stamping for known api/d2d paths, unknown paths, absent map, and missing test_file

CombinedOrchestrator.run_tests() calls has_pyats_tests() to check for PyATS files, not discover_pyats_tests(). The mock was using the old tuple-return API from before the TestMetadataResolver refactor, making it dead code — the test passed accidentally via MagicMock's truthy default return value.

…list All callers (Typer CLI, orchestrator, test_discovery) pass list[str] | None. The Sequence type and defensive list() copies were unnecessarily broad. Replace with list[str] | None and `or []` throughout; remove the now-unused collections.abc.Sequence import.

oboehmer added enhancement New feature or request pyats PyATS framework related labels Mar 19, 2026

oboehmer force-pushed the feat/436-pyats-tag-filtering branch from 58b8ea2 to c944b81 Compare March 19, 2026 08:44

oboehmer force-pushed the feat/436-pyats-tag-filtering branch from c944b81 to 7ee38fb Compare March 19, 2026 08:47

oboehmer added 5 commits March 19, 2026 10:48

oboehmer self-assigned this Mar 19, 2026

oboehmer changed the base branch from release/pyats-integration-v1.1-beta to release/pyats-integration-v2.0 March 19, 2026 16:25

oboehmer changed the base branch from release/pyats-integration-v2.0 to release/pyats-integration-v1.1-beta March 20, 2026 21:56

oboehmer mentioned this pull request Mar 22, 2026

New pyats support #459

Merged

oboehmer changed the base branch from release/pyats-integration-v1.1-beta to main March 22, 2026 07:58

oboehmer added 8 commits March 25, 2026 08:04

Merge remote-tracking branch 'origin/main' into feat/436-pyats-tag-fi…

556ca28

…ltering

refactor(e2e): cache robot_invoked property to avoid repeated rglob c…

4e024ef

…alls

remove redundant helper function

69fe04e

Merge remote-tracking branch 'origin/main' into feat/436-pyats-tag-fi…

96af416

…ltering

oboehmer marked this pull request as ready for review March 25, 2026 14:03

oboehmer requested a review from aitestino March 25, 2026 16:50

oboehmer marked this pull request as draft March 25, 2026 19:36

oboehmer added 2 commits March 25, 2026 23:25

oboehmer marked this pull request as ready for review March 25, 2026 22:47

oboehmer added 2 commits March 25, 2026 23:54

remove tags which validated flexible inputs

243ac61

oboehmer added the prio: high label Mar 31, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(pyats): add tag-based filtering for pyATS tests#657

feat(pyats): add tag-based filtering for pyATS tests#657
oboehmer wants to merge 19 commits intomainfrom
feat/436-pyats-tag-filtering

oboehmer commented Mar 17, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

oboehmer commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Closes

Related Issue(s)

Type of Change

Test Framework Affected

Network as Code (NaC) Architecture Affected

Platform Tested

Key Changes

Core Tag Filtering Implementation

Testing

New Tests for Tag Filtering

Test Suite Refactoring (Independent of Feature)

Testing Done

Test Commands Used

Checklist

Screenshots (if applicable)

Additional Notes

Implementation Approach

Breaking Changes

Migration Notes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

oboehmer commented Mar 17, 2026 •

edited

Loading