Use Cases
Several integration/e2e test.yaml configs pin third-party dependency versions (e.g. tests/e2e/datadog-metrics/config/test.yaml's agent_version matrix) to avoid floating tags like latest silently picking up upstream breaking changes (see #25790, where an unpinned latest Datadog Agent tag broke datadog-metrics e2e tests when the Agent changed its default metrics intake API).
Pinning avoids the surprise breakage, but pinned versions go stale and don't get periodically revisited, so we lose test coverage against current upstream releases over time unless someone manually notices and bumps them.
Attempted Solutions
Currently version pinning/bumping across these matrices is manual and ad hoc, done reactively after a CI break (as in #25790) rather than on a schedule.
Proposal
Add automation that periodically checks pinned dependency versions used in test matrices (e.g. agent_version in tests/e2e/datadog-metrics/config/test.yaml, and similar matrix entries in other test.yaml files under tests/integration/** and tests/e2e/**), and opens a PR bumping them to the latest available version so CI can surface whether the bump passes or fails.
For this to be generic rather than per-test bespoke logic, the test.yaml config format could be extended to declare the Docker Hub (or other registry) image base associated with each pinned matrix value, so the automation can resolve current tags without hardcoding per-integration knowledge.
References
Use Cases
Several integration/e2e
test.yamlconfigs pin third-party dependency versions (e.g.tests/e2e/datadog-metrics/config/test.yaml'sagent_versionmatrix) to avoid floating tags likelatestsilently picking up upstream breaking changes (see #25790, where an unpinnedlatestDatadog Agent tag brokedatadog-metricse2e tests when the Agent changed its default metrics intake API).Pinning avoids the surprise breakage, but pinned versions go stale and don't get periodically revisited, so we lose test coverage against current upstream releases over time unless someone manually notices and bumps them.
Attempted Solutions
Currently version pinning/bumping across these matrices is manual and ad hoc, done reactively after a CI break (as in #25790) rather than on a schedule.
Proposal
Add automation that periodically checks pinned dependency versions used in test matrices (e.g.
agent_versionintests/e2e/datadog-metrics/config/test.yaml, and similar matrix entries in othertest.yamlfiles undertests/integration/**andtests/e2e/**), and opens a PR bumping them to the latest available version so CI can surface whether the bump passes or fails.For this to be generic rather than per-test bespoke logic, the
test.yamlconfig format could be extended to declare the Docker Hub (or other registry) image base associated with each pinned matrix value, so the automation can resolve current tags without hardcoding per-integration knowledge.References