Trt 2389 test summaries

[neisw]

Adds tests data type

Produces test_summaries.json

Need to review the importance of this if we are pivoting to openshift/origin#30604

[openshift-ci-robot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

[coderabbitai]

Walkthrough

Adds TestSummaryByPeriodRow model and BigQuery-backed ListTestSummaryByPeriod client (with retry and mock), a new "tests" data type path in the historical analyzer that fetches/formats/writes test summaries, related CLI validation, and helper utilities for formatting and reading test summary files.

Changes

Cohort / File(s)	Summary
Core data model pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go	New exported struct TestSummaryByPeriodRow with bigquery tags and civil.Date fields: Release, TestName, TotalTestCount, TotalFailureCount, TotalFlakeCount, FailureRate, AvgDurationMs, PeriodStart, PeriodEnd, DaysWithData.
CI data client & retry + misc const pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go, pkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.go, pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client_mock.go, pkg/jobrunaggregator/jobrunaggregatorlib/util.go	Added interface method ListTestSummaryByPeriod(ctx, suiteName, releaseName string, daysBack, minTestCount int) ([]jobrunaggregatorapi.TestSummaryByPeriodRow, error); implemented BigQuery aggregation query in ciDataClient; added retry wrapper in retryingCIDataClient; extended gomock mock with the new method and recorder; added constant bigQueryLabelValueTestSummaryByPeriod.
Historical analyzer: tests path, CLI, and helpers pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/analyzer.go, pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/cmd.go, pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go	Added "tests" to supported data types and adjusted validation around --current/release flags; introduced runTestsDataType to call ListTestSummaryByPeriod (hardcoded suite/params), format results via formatTestOutput, and write output file; added helpers formatTestOutput, readTestSummaryFile, and hasSufficientDaysOfData.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

✨ Finishing touches

• 📝 Generate docstrings

---

📜 Recent review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Cache: Disabled due to data retention organization setting

Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting

📥 Commits

Reviewing files that changed from the base of the PR and between d87305e and 6533fee.

📒 Files selected for processing (2)

• pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/analyzer.go
• pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go

🚧 Files skipped from review as they are similar to previous changes (1)

• pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/analyzer.go

🧰 Additional context used

📓 Path-based instructions (1)

**

⚙️ CodeRabbit configuration file

-Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity.

Files:

• pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go

🧬 Code graph analysis (1)

pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go (1)

pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)

• TestSummaryByPeriodRow (9-20)

🔇 Additional comments (3)

pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go (3)

170-185: LGTM! Clean multi-level sorting implementation.

The function correctly handles empty input, applies a stable three-level sort (Release ascending, TotalFailureCount descending, TestName ascending), and formats the output consistently with the existing formatOutput function.

---

187-200: LGTM! Consistent with existing file reading patterns.

The function follows the same pattern as readHistoricalDataFile, with appropriate error wrapping and clear error messages.

---

202-217: LGTM! Clear validation logic.

The function correctly validates data sufficiency by checking if any test summary meets the minimum days threshold. The early return optimization and empty-slice handling are appropriate.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[neisw]

Likely need more validation for when we change releases. Make sure we have at least 2 weeks of data, etc.

[neisw]

/hold

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: neisw

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• pkg/jobrunaggregator/OWNERS [neisw]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/analyzer.go (1)

25-52: New data-type=tests flow is cleanly separated; consider future configurability

The early branch in Run and runTestsDataType cleanly bypass the comparison logic for data-type=tests, fetching summaries once and writing them directly via formatTestOutput, without disturbing the alerts/disruptions path. The hardcoded suiteName, daysBack, and minTestCount are reasonable defaults, but if this grows beyond a single use case, promoting them to flags or config would improve long‑term flexibility.

Also applies to: 100-132

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

Cache: Disabled due to data retention organization setting

Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting

📥 Commits

Reviewing files that changed from the base of the PR and between f0f4b3b and c5366f4.

📒 Files selected for processing (7)

• pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1 hunks)
• pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (2 hunks)
• pkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.go (1 hunks)
• pkg/jobrunaggregator/jobrunaggregatorlib/util.go (1 hunks)
• pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/analyzer.go (2 hunks)
• pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/cmd.go (2 hunks)
• pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go (1 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**

⚙️ CodeRabbit configuration file

-Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity.

Files:

• pkg/jobrunaggregator/jobrunaggregatorlib/util.go
• pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go
• pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go
• pkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.go
• pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/analyzer.go
• pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/cmd.go
• pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go

🧬 Code graph analysis (3)

pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (1)

pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)

• TestSummaryByPeriodRow (9-20)

pkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.go (1)

pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)

• TestSummaryByPeriodRow (9-20)

pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go (2)

pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)

• TestSummaryByPeriodRow (9-20)

pkg/api/types.go (1)

• Release (452-466)

🔇 Additional comments (6)

pkg/jobrunaggregator/jobrunaggregatorlib/util.go (1)

39-58: BigQuery label addition is consistent and low‑risk

The new bigQueryLabelValueTestSummaryByPeriod constant matches existing naming/usage patterns for query labels; no issues from a maintainability or behavior standpoint.

pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/cmd.go (1)

29-49: data-type=tests wiring and validation look coherent

Adding "tests" to supportedDataTypes and relaxing only the --current / --previous-release requirements (while still enforcing BigQuery auth/coords when --new is empty) matches the new tests path, which fetches directly from BigQuery without doing comparisons. No obvious correctness or UX regressions here.

Also applies to: 51-77

pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go (1)

170-185: Test-summary formatting helper is straightforward and consistent

The in-place sort by release, failure count (desc), then test name mirrors the query’s ordering and existing formatOutput patterns, and json.MarshalIndent keeps output stable for diffs. Behavior for empty input is defensive and safe.

pkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.go (1)

188-196: Retry wrapper mirrors existing client behavior

ListTestSummaryByPeriod on the retrying client follows the same retry.OnError(slowBackoff, isReadQuotaError, …) pattern as the other methods, so retry semantics stay consistent and localized to quota-style failures.

If you haven’t already, run a quick integration test hitting the real BigQuery-backed implementation to confirm retry behavior and error wrapping are as expected.

pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)

1-20: Struct shape correctly matches the BigQuery query schema

TestSummaryByPeriodRow’s field names, types, and bigquery tags line up with the ListTestSummaryByPeriod SELECT clause (SUMs as int64, ratios/durations as float64, dates as civil.Date), so decoding should be straightforward.

Please confirm via a small end‑to‑end query (or unit test with a fake RowIterator) that BigQuery scanning into this struct works without field/tag mismatches.

pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (1)

70-76: Test-summary query is well‑structured and consistent with existing CIDataClient patterns

The new ListTestSummaryByPeriod method is documented clearly on the interface and the implementation’s SQL matches the TestSummaryByPeriodRow schema (field aliases ↔ struct tags). The aggregation logic (SUM/SAFE_DIVIDE/AVG, window via CURRENT_DATE() and daysBack, HAVING SUM(test_count) > @min_test_count) and labeling (bigQueryLabelValueTestSummaryByPeriod) align with how other queries in this client are written, so it should be straightforward to operate and debug.

If possible, run this against a representative dataset to confirm:

• Parameter binding for days_back / min_test_count is accepted by BigQuery.
• The DATE_SUB(CURRENT_DATE(), INTERVAL @days_back DAY) filter yields the expected window.
• The decoded TestSummaryByPeriodRow values (including FailureRate and PeriodStart/End) match manual query results.

Also applies to: 1107-1173

[neisw]

/test images

[neisw]

/test images

[openshift-ci]

@neisw: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/images	6533fee	link	true	/test images
ci/prow/breaking-changes	6533fee	link	false	/test breaking-changes

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.