kci-dev roadmap 2025/2026

[aliceinwire]

kci-dev roadmap 2025/2026

kci-dev is envisioned as a stand-alone KernelCI CLI for both kernel developers and maintainers.

---

1. Enhanced CI Pipeline Workflow & Automation

Custom pipeline triggers and .kernelci.yml support

• Let maintainers define custom test plans via a .kernelci.yml file in their tree (similar to GitLab CI).
• Provide parsing/validation so the file reliably selects the right branches, configs, and platforms.
• Enable developers to control how KernelCI builds and tests their changes through a clear, versioned plan.

Pre-submit testing and CI integration

• Gate merges on KernelCI results (e.g., for merge requests and pull requests).
• Integrate with existing CI systems so critical build/boot failures prevent a merge.
• Catch regressions before patches land, helping both subsystem maintainers and contributors.

Local KernelCI instance support

• Support using a local or self-hosted KernelCI instance (including private hardware labs).
• Make it practical to do “local sanity tests” on key hardware before sending patches upstream.

Deeper git integration

• Automatically detect repository URL, branch, and HEAD commit from the user’s current git tree.
• Reduce manual arguments and make repeated testing across many trees simpler.

Scheduled and batch testing

• Schedule tests (e.g., nightly) and test multiple branches/trees in one action.
• Produce consolidated reports for maintainers tracking multiple branches (linux-next, stable, subsystem trees).
• Reduce manual monitoring by summarizing failures and trends over time.

---

2. Expanded Test Coverage & Reliability Improvements

Broader test suite integration

• Expand beyond build/boot by exposing additional test suites and signals, such as:
  • KUnit (kernel unit tests)
  • static analysis outputs
  • fuzzing-style workloads
  • driver-specific test suites
  • coverage-related artifacts (where supported)

Targeted configurations and fragments

• Allow focused test variants using config fragments (e.g., KASAN, lockdep, debug-heavy configs).
• Support selecting smaller “smoke” subsets vs. full-matrix runs, depending on intent.

Improved flake detection and auto-retry

• Identify flaky jobs and differentiate infrastructure issues from real regressions.
• Support smart retries and stability tracking (e.g., a “flakiness score” per job/test).

Automated regression bisect and triage

• When a new failure appears, help guide maintainers toward a bisect workflow.
• Tight coupling of KernelCI testing with bisection can reduce time-to-culprit for regressions.

Issue tracking integration

• Use KCIDB issue data to surface known build/boot problems.
• Allow tagging known issues, suppressing noise, and linking failures to external bug IDs.
• Help maintainers prioritize new regressions over ongoing known failures.

---

3. Improved Results Visualization & Analysis

Rich CLI output and optional TUI mode

• Improve terminal presentation of results beyond tables and colors.
• Consider a text-based UI for interactive browsing of trees, commits, platforms, and results.

Enhanced diffing of test runs

• Provide more detailed comparisons between runs:
  • which tests/platforms regressed
  • which ones improved
  • what stayed the same
• Make it easy to answer: “Did my patch series introduce new failures?”

Dashboard navigation from the CLI

• Provide convenient navigation from CLI output to the corresponding dashboard details for deep investigation.
• Useful for sharing results and for quick jumps to logs/artifacts.

Performance and caching of results

• Address slow fetching caused by large result payloads.
• Add caching and incremental refresh so repeated queries become fast and practical for frequent use.

Customizable reports and notifications

• Generate reports in formats suitable for email and meetings (e.g., Markdown or HTML).
• Integrate with notification systems (email or chat) to alert on new failures or regressions.
• Support targeted alerts, such as per-subsystem or per-platform summaries.

---

4. Integration with Patchwork, b4, and Git Hosting CI

Patch series injection (no public git required)

• Allow developers to submit a patch series directly for KernelCI testing without pushing to a remote repository.
• Treat a patch series as a first-class test input (alongside git trees), with clear provenance:
  • base commit and base tree selection
  • series identity (revision, hashes)
  • reproducible re-runs of the same exact series
• Produce a reply-ready summary that maintainers can copy into mailing list threads or merge request discussions.

Patchwork integration for mailing-list patches

• Fetch a patch series from Patchwork and trigger KernelCI testing on it.
• Optionally report status back into Patchwork (as checks or comments), if policy and auth allow.
• Bridge the gap for maintainers who work primarily via mailing lists.

b4 workflow support

• Support workflows where a series is fetched from public mailing list archives and tested.
• Reduce manual “git am” steps and shorten the path from email → test results.

GitLab CI / GitHub integration

• Provide standard integration patterns for:
  • triggering KernelCI from pipelines
  • waiting for results
  • reporting pass/fail back into merge request / pull request context (status checks, comments)

Bi-directional tool integration

• Minimize context switching for maintainers by integrating common tools:
  • patch tracking (Patchwork)
  • email-based series fetching (b4)
  • bug trackers (optional linking and evidence export)

---

5. Catering to Maintainers and Patch Submitters

Maintainer-focused features

• Multi-tree / multi-branch dashboards and summaries to avoid missing failures.
• Subsystem-focused filtering to highlight failures relevant to a maintainer’s area.
• Release readiness tools (e.g., unresolved failures during an RC cycle) as practical checklists.

Developer-focused features

• Quick pre-submit testing to catch regressions before sending patches.
• Lightweight setup for read-only usage.
• “Dry-run” style previews to estimate what a test plan would run and why.

Collaborative workflows

• Easy export of results (structured data plus human-readable summaries) for sharing.
• “Evidence bundles” to share logs and metadata with developers, maintainers, and infra teams.

---

6. Library, Packaging, and Extensibility

Library mode (SDK-style)

• Provide a stable, documented Python library interface that exposes the same core capabilities as the CLI.
• Make it straightforward for other tools and services to reuse kci-dev functionality:
  • CI integrations
  • dashboards
  • patch workflow automation
  • custom triage tooling
• Keep the CLI as a thin wrapper over the library to reduce duplication and improve testability.

Packaging and distribution

• Make installation easy and reproducible across environments.
• Ensure compatibility with common distro workflows and secure update practices.

Plugin-friendly architecture

• Consider a plugin mechanism so third parties can extend kci-dev without forking:
  • org-specific workflows
  • lab-specific metadata mapping
  • custom result views and reports
  • alternative backends (where appropriate)

---

7. Addressing Pain Points & Ongoing Improvements

Patchwork integration as a high priority

• A commonly requested gap in mailing-list-centric workflows.
• Closing this loop makes KernelCI substantially more useful to maintainers.

Performance and usability

• Caching and incremental updates reduce friction for frequent triage.
• Better error messages, shell completions, and documentation reduce onboarding time.

Roadmap alignment

• Prioritize foundational improvements like git context awareness and regression diffing.
• Expand toward richer automation once the “fast path” experience is solid.

Community and extensibility

• Encourage community contributions around org-specific workflows and lab integrations.
• Maintain clear boundaries between core features and optional extensions.

---

Conclusion

These ideas aim to make kci-dev a practical day-to-day tool for kernel work: fast access to results, strong regression signal, smooth integration with mailing lists and Git-based workflows, and scalable triage for maintainers responsible for multiple trees and platforms.

[padovan]

There is a lot of great stuff here. Thank you for creating this.

Some of task though, they seem to be related to other pieces of the architecture, and not kci-dev as the central place of work. We need to step out a bit and look at our entire architecture and how pieces connect.

Eg. Under 1. Enhanced CI Pipeline Workflow & Automation, many of these tasks are actually Maestro, Dashboard or regression tracking tasks. So the work is falls more on offering APIs that kci-dev can use to show that data to the user, rather this being a kci-dev roadmap.

Same under "Customizable reports and notifications" for example, the bulk of the work is on the Dashboard side. As a rule of thumb, we aim at offering the same data/features for most things both in the Dashboard frontend and in kci-dev, which means that a lot of the work happens in Dashboard backend. Same is true for Maestro.