Skip to content

fix(vector source): surface underlying error when gRPC request decompression fails#25796

Open
Stunned1 wants to merge 3 commits into
vectordotdev:masterfrom
Stunned1:fix-grpc-decompression-error-detail
Open

fix(vector source): surface underlying error when gRPC request decompression fails#25796
Stunned1 wants to merge 3 commits into
vectordotdev:masterfrom
Stunned1:fix-grpc-decompression-error-detail

Conversation

@Stunned1

@Stunned1 Stunned1 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary

When a compressed gRPC request fails to decompress, the shared decompression layer (sources/util/grpc/decompression.rs, used by the vector and opentelemetry sources) discards the underlying error and returns:

status: Internal, message: "reached impossible error during decompressor finalization"

The error is not impossible: finalization is where the payload's integrity is actually validated (gzip CRC32/length trailer, zstd frame decode), so any corrupt or truncated payload ends up here, and reports like #25723 are undiagnosable without the real cause.

This PR includes the underlying error in the returned Status, for example:

failed to finalize decompressor: corrupt gzip stream does not have a matching checksum

It also corrects the misleading comment and adds the layer's first unit tests (identity/gzip/zstd round trips, chunked delivery, corrupted payloads, and rejection of compressed frames without a negotiated encoding).

What I cut and why it's safe: the GzDecoder::finish() mechanics (now condensed to one clause; the full explanation lives in the corrected code comment reviewers will read in the diff), the "three failure sites" enumeration (visible in the diff), and the itemized test list (collapsed to categories). Nothing a reviewer needs to evaluate the change is gone; the before/after messages still tell the whole story at a glance.

Vector configuration

# aggregator
sources:
  vec:
    type: vector
    address: "0.0.0.0:6001"
    version: "2"
sinks:
  bh:
    type: blackhole
    inputs: [vec]

# agent
sources:
  hm:
    type: host_metrics
    scrape_interval_secs: 1
sinks:
  out:
    type: vector
    inputs: [hm]
    address: "aggregator:6001"
    compression: true

How did you test this PR?

  • Added 7 unit tests for drive_body_decompression covering the happy paths (identity/gzip/zstd, including frames delivered in 3-byte chunks to exercise chunk boundaries inside the gRPC header and gzip trailer) and the error paths (corrupt gzip CRC32 trailer, corrupt zstd frame), asserting the underlying error text is surfaced.
  • Existing end-to-end tests pass: sources::vector::tests (receive_gzip_compressed_message, receive_zstd_compressed_message, receive_message, health checks).
  • make check-clippy (workspace, all features), make check-fmt, and make check-markdown pass locally.
  • Separately validated the surrounding behavior with two timberio/vector containers (host_metrics to a vector sink with gzip and mTLS, into a vector source): millions of events delivered without error, confirming this change only affects the message attached to already-failing requests.

Change Type

  • Bug fix
  • New feature
  • Dependencies
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

Notes

This does not close #25723 on its own! The root cause of the arm64 corruption reported there has not been reproduced (tested: official 0.56.0 linux/arm64 images end-to-end with gzip, mTLS, and large batches, plus cross-backend fuzzing of the pinned zlib-rs backend on aarch64, all clean). With this change, affected senders will see the actual decompression failure in the sink's error logs.

@Stunned1 Stunned1 requested a review from a team as a code owner July 9, 2026 21:18
@github-actions github-actions Bot added the domain: sources Anything related to the Vector's sources label Jul 9, 2026
@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@datadog-vectordotdev

This comment has been minimized.

@Stunned1

Stunned1 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

I have read the CLA Document and I hereby sign the CLA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: sources Anything related to the Vector's sources

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vector source: gzip decompression fails on arm64 0.56.0 with "reached impossible error during decompressor finalization"

1 participant