Set up CI/CD pipeline and automated test suite for shell scripts

[jellydn]

Problem Statement

The cmdk project currently relies on manual testing across different shells (bash, zsh, fish), which is:

• Time-consuming and error-prone
• Not scalable as the project grows
• Lacks coverage validation
• Difficult to ensure compatibility across different OS environments

Proposed Solution

1. Testing Framework Selection

After researching 2025 shell testing frameworks, I recommend ShellSpec as the primary testing framework because:

• ✅ Supports all POSIX shells (bash, zsh, dash, ksh, mksh, etc.)
• ✅ BDD-style testing with describe/it blocks
• ✅ Code coverage support for bash/zsh/ksh
• ✅ Mocking capabilities for external commands
• ✅ Parallel test execution
• ✅ Docker support for isolated testing
• ✅ CI-friendly TAP output

Alternative: BATS (Bash Automated Testing System) as a fallback for bash-specific tests.

2. GitHub Actions Matrix Strategy

Set up CI pipeline that tests across:

• Shells: bash, zsh, fish
• OS: Ubuntu (latest), macOS (latest)
• Dependencies: With/without optional tools (bat, tiv, poppler)

3. Test Structure

tests/
├── unit/
│   ├── list-files_spec.sh       # Test list-files.sh logic
│   ├── preview_spec.sh          # Test preview.sh functionality
│   ├── cmdk-core_spec.sh        # Test core functionality
│   └── shell-wrappers_spec.sh   # Test shell-specific wrappers
├── integration/
│   ├── end-to-end_spec.sh       # Full workflow tests
│   ├── flag-behavior_spec.sh    # Test -o, -s, -O flags
│   └── file-handling_spec.sh    # Test different file types
├── fixtures/
│   ├── test-directories/        # Sample directory structures
│   ├── test-files/             # Sample files for testing
│   └── expected-outputs/       # Expected command outputs
└── helpers/
    ├── test-helpers.sh         # Common test utilities
    └── mock-commands.sh        # Mock external dependencies

4. CI Workflow Features

• Pre-commit hooks: ShellCheck linting, basic syntax validation
• Multi-shell testing: Matrix strategy for bash/zsh/fish
• Dependency testing: Test with/without optional dependencies
• Code coverage: Generate coverage reports for bash/zsh
• Performance testing: Benchmark file listing performance
• Regression testing: Test against previous versions

5. Quality Gates

• All tests must pass across all shell/OS combinations
• Minimum 80% code coverage where measurable
• ShellCheck must pass with no errors
• Performance benchmarks must not regress by >20%

6. Test Categories to Implement

Unit Tests

• Flag parsing logic (-o, -s, -O)
• Directory exclusion/inclusion logic
• File filtering algorithms
• Preview generation for different file types
• Temporary file handling
• Error handling and edge cases

Integration Tests

• End-to-end workflow: search → select → open
• Multi-file selection behavior
• Directory navigation and cd functionality
• File type detection and appropriate opening
• Home vs non-home directory behavior
• Performance with large directory structures

Shell Compatibility Tests

• Function behavior across bash/zsh/fish
• Argument passing and variable handling
• Exit codes and error propagation
• Environment variable handling
• Path resolution differences

7. Implementation Timeline

1. Week 1: Set up basic ShellSpec structure and simple unit tests
2. Week 2: Implement GitHub Actions workflow with matrix strategy
3. Week 3: Add comprehensive test coverage for core functionality
4. Week 4: Integration tests and performance benchmarking
5. Week 5: Documentation and refinement

8. Files to Create/Modify

• .github/workflows/ci.yml - Main CI pipeline
• .github/workflows/pre-commit.yml - Pre-commit checks
• tests/ directory structure - All test files
• .shellspec - ShellSpec configuration
• Makefile or scripts/test.sh - Local testing convenience
• Update README.md with testing instructions
• Update CLAUDE.md with testing commands

9. Benefits

• Quality Assurance: Catch regressions early
• Multi-shell Compatibility: Automated testing across shells
• Documentation: Tests serve as executable documentation
• Contributor Confidence: Safe refactoring and feature additions
• Performance Monitoring: Track performance regressions
• Release Automation: Automated testing before releases

10. Success Metrics

• 100% of commits automatically tested
• Zero shell compatibility issues in releases
• Test suite runs in <5 minutes
• Coverage reports available for each PR
• Contributors can run tests locally with single command

This comprehensive testing strategy will significantly improve the reliability and maintainability of cmdk while ensuring compatibility across all supported shells and environments.

References

• ShellSpec - BDD testing framework
• BATS - Bash Automated Testing
• GitHub Actions Matrix Strategy
• Fish Shell Testing

🤖 Generated with Claude Code