fix(ci): retry fakeintake fetches in datadog-metrics e2e test#25784
fix(ci): retry fakeintake fetches in datadog-metrics e2e test#25784thomasqueirozb wants to merge 5 commits into
Conversation
thomasqueirozb
added
the
no-changelog
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a9f65d73a2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…nt fakeintake fetch failures
thomasqueirozb
added
the
preview: website
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3e3d46b47f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // whether the series intake looks complete, i.e. every expected metric type | ||
| // has arrived. Used to decide whether it's worth continuing to poll fakeintake. | ||
| fn series_intake_is_complete(series: &SeriesIntake) -> bool { | ||
| !series.is_empty() && found_metric_types(series).iter().all(|(found, _)| *found) |
There was a problem hiding this comment.
Is this the case because of our test setup? It feels like a brittle predicate.
There was a problem hiding this comment.
It's tied to the test's dogstatsd emitter (tests/e2e/datadog-metrics/dogstatsd_client/client.py), which sends exactly these four metric types (rate, gauge, set, histogram) once per run, deterministically.
If changes to client.py are made this would also need to keep track/in sync
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0d90cbb371
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // "wait until this test's data is complete" API of its own, so we have to infer readiness | ||
| // from what the emitter is known to send. | ||
| fn series_intake_is_complete(series: &SeriesIntake) -> bool { | ||
| !series.is_empty() && found_metric_types(series).iter().all(|(found, _)| *found) |
There was a problem hiding this comment.
Fresh evidence: this new completeness predicate still only waits for the four metric-name prefixes, but the DogStatsD client emits those same four metrics on every one of its 50 iterations (tests/e2e/datadog-metrics/dogstatsd_client/client.py:38-54). If the Agent flushes while that loop is still running, the first partial flush already satisfies this predicate; the agent snapshot can be taken before later flushes while the vector snapshot is fetched afterward and includes additional buckets, causing the exact context/sum comparisons below to fail intermittently. Wait for the intake to stabilize or for the expected sample/payload count instead of only checking prefixes.
Useful? React with 👍 / 👎.
| // test's data is complete" API of its own, so we have to infer readiness from what the emitter | ||
| // is known to send. | ||
| fn sketch_intake_is_complete(sketches: &SketchIntake) -> bool { | ||
| !sketches.is_empty() && has_distribution(sketches) |
There was a problem hiding this comment.
Fresh evidence: the new readiness check still treats the first distribution context as complete, while the emitter sends 50 distribution samples (tests/e2e/datadog-metrics/dogstatsd_client/client.py:38-54). In runs where the Agent produces an early sketch flush before the emitter finishes, agent_sketches can return a partial snapshot; the vector path is fetched later and may include subsequent fakeintake payloads, so the exact sketch equality check fails even though the pipeline is correct. Poll until the sketch data is stable or the expected amount of data has arrived.
Useful? React with 👍 / 👎.
thomasqueirozb
removed
the
preview: website
thomasqueirozb
added
preview: website
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: be1e95623d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| } | ||
| } | ||
|
|
||
| panic!("{last_error} after {MAX_FETCH_ATTEMPTS} attempts"); |
There was a problem hiding this comment.
Fresh evidence: this helper now retries connection/JSON errors internally, but it still panics after 10 attempts, so the 45-attempt metrics polling budget in tests/e2e/datadog/metrics/mod.rs is never reached when fakeintake is simply unavailable for more than ~10s. I checked tests/e2e/datadog-metrics/config/compose.yaml and the fakeintake services have no healthcheck/readiness gate, so a slow container startup can still abort the test before the data-arrival retry loop has a chance to continue.
Useful? React with 👍 / 👎.
2 participants
Summary
The
datadog-metricse2e test used a single fixed 2-second sleep before fetching metric payloads from fakeintake, with no retry on the fetch itself. Under load this window isn't always enough for events to flow through the agent -> fakeintake pipeline, causing intermittentassertion failed: !series.is_empty()failures.This applies the same retry pattern already used by the
datadog-logse2e test: poll fakeintake up to 10 times (1s apart) until data shows up, instead of a single fixed-delay fetch.Vector configuration
NA
How did you test this PR?
Ran
cargo check/cargo clippy --test e2e --no-default-features --features e2e-tests-datadoglocally to verify the change compiles cleanly. The e2e suite itself requires Docker-based fakeintake/agent services in CI.Change Type
Is this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.References
NA