fix(vector source): surface underlying error when gRPC request decompression fails#25796
Open
Stunned1 wants to merge 3 commits into
Open
fix(vector source): surface underlying error when gRPC request decompression fails#25796Stunned1 wants to merge 3 commits into
Stunned1 wants to merge 3 commits into
Conversation
Contributor
|
All contributors have signed the CLA ✍️ ✅ |
This comment has been minimized.
This comment has been minimized.
Contributor
Author
|
I have read the CLA Document and I hereby sign the CLA |
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a compressed gRPC request fails to decompress, the shared decompression layer (
sources/util/grpc/decompression.rs, used by thevectorandopentelemetrysources) discards the underlying error and returns:The error is not impossible: finalization is where the payload's integrity is actually validated (gzip CRC32/length trailer, zstd frame decode), so any corrupt or truncated payload ends up here, and reports like #25723 are undiagnosable without the real cause.
This PR includes the underlying error in the returned
Status, for example:It also corrects the misleading comment and adds the layer's first unit tests (identity/gzip/zstd round trips, chunked delivery, corrupted payloads, and rejection of compressed frames without a negotiated encoding).
What I cut and why it's safe: the GzDecoder::finish() mechanics (now condensed to one clause; the full explanation lives in the corrected code comment reviewers will read in the diff), the "three failure sites" enumeration (visible in the diff), and the itemized test list (collapsed to categories). Nothing a reviewer needs to evaluate the change is gone; the before/after messages still tell the whole story at a glance.
Vector configuration
How did you test this PR?
drive_body_decompressioncovering the happy paths (identity/gzip/zstd, including frames delivered in 3-byte chunks to exercise chunk boundaries inside the gRPC header and gzip trailer) and the error paths (corrupt gzip CRC32 trailer, corrupt zstd frame), asserting the underlying error text is surfaced.sources::vector::tests(receive_gzip_compressed_message,receive_zstd_compressed_message,receive_message, health checks).make check-clippy(workspace, all features),make check-fmt, andmake check-markdownpass locally.timberio/vectorcontainers (host_metrics to a vector sink with gzip and mTLS, into a vector source): millions of events delivered without error, confirming this change only affects the message attached to already-failing requests.Change Type
Is this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.References
Notes
This does not close #25723 on its own! The root cause of the arm64 corruption reported there has not been reproduced (tested: official 0.56.0 linux/arm64 images end-to-end with gzip, mTLS, and large batches, plus cross-backend fuzzing of the pinned zlib-rs backend on aarch64, all clean). With this change, affected senders will see the actual decompression failure in the sink's error logs.