Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ This is useful for:
- [Installation](docs/installation.md)
- [Usage guide](docs/usage.md)
- [JSON contract](docs/json-contract.md)
- [Confidence model](docs/confidence.md)
- [JSON schema](schema/prmaven-report.schema.json)
- [Examples](examples/README.md)
- [Fixture notes](docs/fixtures.md)
Expand Down
69 changes: 69 additions & 0 deletions docs/confidence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Confidence

PR Maven CLI uses confidence to describe how strong the local evidence is for a finding.

Confidence is about evidence quality. It is not a probability score, severity score, or guarantee that a developer made a mistake. A high-confidence finding means the CLI found deterministic Maven evidence on disk and can explain why it trusts that evidence.

## Current Stage 1 Behavior

Stage 1 emits `high` confidence for every finding because all current findings come from supported Maven report or log artifacts:

- Surefire JUnit XML reports;
- Failsafe JUnit XML reports;
- Checkstyle XML reports;
- SpotBugs XML reports;
- Maven Enforcer log artifacts;
- JaCoCo threshold log artifacts.

The analyzer does not infer failures from remote provider state, pull request metadata, free-form CI logs, or model output. It reads local artifacts, maps them back to Maven modules, and emits a finding only when the source artifact contains a deterministic failure, violation, bug, rule failure, or threshold failure.

## Levels

### `high`

Use `high` when the finding is backed by a supported local artifact and the CLI can explain the evidence path.

Current examples:

- a test failure found in a Surefire or Failsafe JUnit XML report;
- a Checkstyle violation found in a Checkstyle XML report;
- a SpotBugs bug found in a SpotBugs XML report;
- a Maven Enforcer rule failure found in a Maven log artifact;
- a JaCoCo threshold failure found in a Maven log artifact.

High-confidence findings should include confidence reasons such as:

- the source report or log format that contained the finding;
- the Maven module mapped from the report path;
- the test class, source file, rule, or threshold metric used to build the reproduction context.

### `medium`

`medium` is reserved for future releases.

It should be used when a finding is supported by useful evidence but one part of the chain is incomplete or inferred. Examples could include partial CI artifact layouts, provider metadata that points to a failing module without the full report payload, or log excerpts that identify the plugin but not every source detail.

Stage 1 does not emit `medium`.

### `low`

`low` is reserved for future releases.

It should be used only for weak or heuristic hints that may help triage but are not strong enough to claim a deterministic Maven failure. Examples could include changed-file relevance, broad log keywords, or provider-level failure summaries without a parseable Maven artifact.

Stage 1 does not emit `low`.

## JSON Contract

The current JSON schema allows `high` because the current implementation only emits report-backed deterministic findings. Future releases may add `medium` and `low` through an explicit compatibility note.

Consumers should:

- read `confidence` as a trust level for the evidence;
- read `confidenceReasons` before automating decisions;
- ignore unknown future fields;
- treat unknown future confidence values conservatively until their workflow explicitly supports them.

## Future Model

The Stage 3 confidence model may combine local Maven evidence with optional provider context, changed-file relevance, annotations, or PR metadata. Local report-backed evidence should remain the strongest signal, and every confidence level should keep human-readable reasons in JSON output.
3 changes: 2 additions & 1 deletion docs/implementation-status.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,8 @@ The CLI emits:
- JSON;
- optional file output.

Current findings emit `high` confidence with human-readable confidence reasons because Stage 1 findings are backed by deterministic Maven report or log artifacts. The `medium` and `low` levels are documented as reserved future vocabulary.

Validated by:

- CLI tests;
Expand Down Expand Up @@ -199,7 +201,6 @@ It does not currently fetch artifacts from CI providers, inspect remote pull req
Near-term priorities:

- keep parser fixtures easy for contributors to extend;
- document confidence levels;
- document fixture contribution rules;
- harden output compatibility expectations;
- keep issue labels and backlog aligned with the roadmap.
Expand Down
4 changes: 2 additions & 2 deletions docs/json-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,8 @@ Compatibility expectations:
| `failureType` | string | Optional source-specific type, rule, category, or threshold metric. |
| `message` | string | Optional one-line finding message from the source artifact. |
| `reproduceCommand` | string | Minimal Maven command for reproducing or re-running the relevant check locally. |
| `confidence` | string | Confidence level for the finding. Current values use `high` for fixture-backed deterministic evidence. |
| `confidenceReasons` | array of string | Human-readable evidence explaining why the finding is trusted. |
| `confidence` | string | Confidence level for the finding. See [Confidence](confidence.md). Current schema values use `high` for report-backed deterministic evidence. |
| `confidenceReasons` | array of string | Human-readable evidence explaining why the finding is trusted. See [Confidence](confidence.md). |
| `sourceReportFormat` | string | Original source format, such as `junit-xml`, `checkstyle-xml`, `spotbugs-xml`, or `maven-log`. |

## Demo JSON
Expand Down
Loading