feat(opentelemetry sink): support legacy nested config fallback and fix gRPC gzip/zstd#25757
Open
sakateka wants to merge 365 commits into
Open
feat(opentelemetry sink): support legacy nested config fallback and fix gRPC gzip/zstd#25757sakateka wants to merge 365 commits into
sakateka wants to merge 365 commits into
Conversation
…dev#25280) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.32.4 to 4.35.2. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@89a39a4...95e58e9) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.35.2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(ci): bump actions/github-script from 7 to 9 Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Use pinned github-script sha --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Thomas <thomas.schneider@datadoghq.com>
…tifact group across 1 directory (vectordotdev#25276) chore(ci): bump actions/upload-artifact Bumps the artifact group with 1 update in the / directory: [actions/upload-artifact](https://github.com/actions/upload-artifact). Updates `actions/upload-artifact` from 7.0.0 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@bbbca2d...043fb46) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: artifact ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…otdev#25230) * fix(splunk_hec source): Emit warn on unauthenticated request * add changelog * Address comments * fix(splunk_hec source): use authentication_failed error type for unauthenticated requests Add a new `authentication_failed` error_type constant so consumers can filter auth failures from other request errors in `component_errors_total`. * Update changelog.md --------- Co-authored-by: Thomas <thomas.schneider@datadoghq.com>
…otdev#25255) Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
…onfigured w/ disk buffers (vectordotdev#24949) * fix(buffers): prevent reload stall when disk buffer config changes - Changing a disk buffer's configuration (e.g. `max_size`) during a live config reload would stall indefinitely or fail with a `buffer.lock` error. This was caused by two issues: 1. The sink's detach trigger was only cancelled for buffer-reuse cases, so sinks with changed disk buffer configs never had their input stream terminated — disk buffer readers do not return `None` when the writer disconnects, causing the old sink task to hang forever. 2. The source output pump only processed fanout control messages (Remove/Pause) during active sends, so idle sources would never drop their `BufferSender` clone, keeping the `Arc<Ledger>` file lock alive even after the sink task completed. - Cancel the detach trigger for changed disk buffer sinks so the old sink task can complete. - Make the source output pump `select!` on both events and fanout control messages, so Remove/Pause is processed promptly even when the source is idle. - Add a retry loop (30s timeout) when acquiring the disk buffer lock to handle the small race window between the sink task completing and the fanout releasing the writer. - Add `BufferConfig::has_disk_stage()` helper for identifying disk buffer configurations. * Add unit tests for reload when disk buffer config modified * Add changelog fragment * Integration test exercising full disk buffer path w/ s3 sink - This test sets up a pipeline that sends using the s3 sinks with disk buffers enabled. - Data is sent through the disk buffer before and after configuration reload. * Clean-up changelog fragment * Fix spelling issues * Fix merge error and cleanup use of config() - Make consistent use of the config() method in all tests within this file * Simplify set of control_channel_open status * Remove unnecessary mutable modifiers * Prefer tempDir to just writing to /tmp --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
…otdev#25259) * chore(ci): remove dead nightly artifact-redirect loop The loop referenced an undefined $i (copy-paste from the release branch where $i iterates over version tags). Under set -u the subshell errored, the for loop received empty input, and the body never ran — so this has been a no-op since it was added. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(ci): retry verify_artifact on CDN cache staleness packages.timber.io is fronted by a CDN; after `aws s3 rm --recursive` + `cp --recursive` on nightly/latest, the edge can keep serving stale bytes for longer than the existing 30s VERIFY_TIMEOUT, failing `cmp` and the whole job. wget's --retry-on-http-error=404 only retries 404s, not a 200 with stale content. Wrap the compare in an exponential-backoff retry loop (1, 2, 4, 8, 16, 32s; 7 attempts, ~63s of total sleep) so a transient stale-cache hit no longer fails the release. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Fetch and fast-forward the release branch before inspecting HEAD so `git show --stat HEAD` reflects origin, not a stale local tip. - Use `git push --force-with-lease` when resetting the `website` branch to the release branch's HEAD; a plain `git push` is rejected as non-fast-forward, which is the expected outcome of `reset --hard` to a different branch. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…g errors (vectordotdev#25199) * fix(codecs): centralize events_dropped emission for batch encoding errors Move events_dropped emission from individual internal events inside serializers to a single wrapper in (Transformer, BatchEncoder)::encode_input. This ensures all batch encoding error paths (Arrow IPC and Parquet) consistently emit events_dropped without requiring each new error path to remember to add it. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: fix formatting Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(codecs): add unit test for type mismatch in Parquet encoding Covers the build_record_batch ArrowJsonDecode error path where a schema expects int64 but the event contains a string value. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(codecs): emit ComponentEventsDropped directly to avoid double-counting Replace EncoderWriteError with a direct ComponentEventsDropped emission in the batch encode wrapper. EncoderWriteError was incrementing component_errors_total and logging "Failed writing bytes." which double-counted errors (codec-specific events already increment component_errors_total) and was misleading (the failure was encoding, not writing). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: clarify comment about error counting and overcount edge case Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(codecs): guard batch-drop imports behind codecs-arrow feature Move ComponentEventsDropped and UNINTENTIONAL imports inside the cfg(feature = "codecs-arrow") impl block to avoid unused import errors when the feature is disabled. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: clarify overcount edge case is a misconfiguration scenario Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore(codecs): emit codec event for build_record_batch failures The new EncoderRecordBatchError fires from build_record_batch's RecordBatchCreation and ArrowJsonDecode paths, so type-mismatch and decoder-build failures emit a granular component_errors_total counter at stage="sending" with a specific error_code, instead of relying solely on the downstream SinkRequestBuildError at stage="processing". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(codecs): add changelog fragment for batch encoding event coverage Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(codecs): assert EncoderRecordBatchError fires on Arrow type mismatch Drives (Transformer, BatchEncoder)::encode_input through ArrowStreamSerializer with an Int64 schema field and a string-valued event to trigger the ArrowJsonDecode path in build_record_batch. Asserts both EncoderRecordBatchError and ComponentEventsDropped are recorded so a regression in either emission fails the test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ctordotdev#25321) * fix(releasing): enable codecs-parquet in all release feature sets The aws_s3 sink's `batch_encoding` field is gated behind the `codecs-parquet` Cargo feature, but the v0.55.0 release artifacts did not include that feature. Users running the precompiled binaries hit `unknown field batch_encoding` even though the feature was advertised in the v0.55.0 release notes. Enable `codecs-parquet` in `target-base` (covers every Linux release triple), `default` (macOS), and `default-msvc` (Windows), plus the related aggregator features for consistency. Closes vectordotdev#25313 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(releasing): correct changelog author handle Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(releasing): trim changelog wording Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(external docs): regenerate component docs with parquet codec Adds the missing `docs::enum_tag_description` metadata on `ParquetCompression` (required by the doc generator for internally tagged enums) and regenerates the aws_s3 and clickhouse component Cue files now that `codecs-parquet` and `codecs-arrow` are part of the default release feature set. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: address review wording on batch encoding fields Apply @maycmlee's suggestions in the source rustdocs (and regenerate the affected Cue): - aws_s3 `batch_encoding`: `e.g., Parquet` -> `for example, Parquet` - arrow `allow_nullable_fields`: tighter wording for both enabled and disabled cases - parquet `Zstd`/`Gzip` `level`: drop "currently" from "Vector currently supports" Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(releasing): consolidate codecs-parquet into base feature Every release feature aggregator (default, default-cmake, default-msvc, default-musl, default-no-api-client, target-base) transitively pulls `base` either directly or via `enable-api-client`/`enable-unix`. Drop the redundant per-aggregator listings and add `codecs-parquet` to `base` once so the feature can't be silently lost when a future aggregator gets added. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(aws_s3, clickhouse): note supported batch_encoding codec per sink The `batch_encoding` field on both sinks types as `Option<BatchSerializerConfig>`, which advertises every codec variant in the generated reference. Each sink only accepts a subset at runtime: aws_s3 rejects everything except `parquet`, clickhouse rejects `parquet`. Call out the supported codec in the field's rustdoc so the published reference matches what the sink will actually accept. A follow-up should replace the shared enum with sink-specific config types so the type system enforces the constraint. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
vectordotdev#25283) * chore(deps): bump the patches group across 1 directory with 13 updates Bumps the patches group with 13 updates in the / directory: | Package | From | To | | --- | --- | --- | | [async-rs](https://github.com/amqp-rs/async-rs) | `0.8.1` | `0.8.4` | | [async-compression](https://github.com/Nullus157/async-compression) | `0.4.41` | `0.4.42` | | [inventory](https://github.com/dtolnay/inventory) | `0.3.22` | `0.3.24` | | [postgres-openssl](https://github.com/rust-postgres/rust-postgres) | `0.5.2` | `0.5.3` | | [pulsar](https://github.com/streamnative/pulsar-rs) | `6.7.1` | `6.7.2` | | [roaring](https://github.com/RoaringBitmap/roaring-rs) | `0.11.3` | `0.11.4` | | [libc](https://github.com/rust-lang/libc) | `0.2.182` | `0.2.186` | | [pastey](https://github.com/as1100k/pastey) | `0.2.1` | `0.2.2` | | [semver](https://github.com/dtolnay/semver) | `1.0.27` | `1.0.28` | | [env_logger](https://github.com/rust-cli/env_logger) | `0.11.6` | `0.11.9` | | [quote](https://github.com/dtolnay/quote) | `1.0.44` | `1.0.45` | | [schannel](https://github.com/steffengy/schannel-rs) | `0.1.28` | `0.1.29` | | [web-sys](https://github.com/wasm-bindgen/wasm-bindgen) | `0.3.91` | `0.3.97` | Updates `async-rs` from 0.8.1 to 0.8.4 - [Commits](amqp-rs/async-rs@v0.8.1...v0.8.4) Updates `async-compression` from 0.4.41 to 0.4.42 - [Release notes](https://github.com/Nullus157/async-compression/releases) - [Commits](Nullus157/async-compression@async-compression-v0.4.41...async-compression-v0.4.42) Updates `inventory` from 0.3.22 to 0.3.24 - [Release notes](https://github.com/dtolnay/inventory/releases) - [Commits](dtolnay/inventory@0.3.22...0.3.24) Updates `postgres-openssl` from 0.5.2 to 0.5.3 - [Release notes](https://github.com/rust-postgres/rust-postgres/releases) - [Commits](rust-postgres/rust-postgres@postgres-openssl-v0.5.2...postgres-openssl-v0.5.3) Updates `pulsar` from 6.7.1 to 6.7.2 - [Release notes](https://github.com/streamnative/pulsar-rs/releases) - [Commits](streamnative/pulsar-rs@v6.7.1...v6.7.2) Updates `roaring` from 0.11.3 to 0.11.4 - [Release notes](https://github.com/RoaringBitmap/roaring-rs/releases) - [Commits](RoaringBitmap/roaring-rs@v0.11.3...v0.11.4) Updates `libc` from 0.2.182 to 0.2.186 - [Release notes](https://github.com/rust-lang/libc/releases) - [Changelog](https://github.com/rust-lang/libc/blob/0.2.186/CHANGELOG.md) - [Commits](rust-lang/libc@0.2.182...0.2.186) Updates `pastey` from 0.2.1 to 0.2.2 - [Release notes](https://github.com/as1100k/pastey/releases) - [Changelog](https://github.com/AS1100K/pastey/blob/master/CHANGELOG.md) - [Commits](AS1100K/pastey@v0.2.1...v0.2.2) Updates `semver` from 1.0.27 to 1.0.28 - [Release notes](https://github.com/dtolnay/semver/releases) - [Commits](dtolnay/semver@1.0.27...1.0.28) Updates `env_logger` from 0.11.6 to 0.11.9 - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.11.6...v0.11.9) Updates `quote` from 1.0.44 to 1.0.45 - [Release notes](https://github.com/dtolnay/quote/releases) - [Commits](dtolnay/quote@1.0.44...1.0.45) Updates `schannel` from 0.1.28 to 0.1.29 - [Release notes](https://github.com/steffengy/schannel-rs/releases) - [Commits](steffengy/schannel-rs@v0.1.28...v0.1.29) Updates `web-sys` from 0.3.91 to 0.3.97 - [Release notes](https://github.com/wasm-bindgen/wasm-bindgen/releases) - [Changelog](https://github.com/wasm-bindgen/wasm-bindgen/blob/main/CHANGELOG.md) - [Commits](https://github.com/wasm-bindgen/wasm-bindgen/commits) --- updated-dependencies: - dependency-name: async-compression dependency-version: 0.4.42 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: patches - dependency-name: async-rs dependency-version: 0.8.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: patches - dependency-name: env_logger dependency-version: 0.11.9 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: patches - dependency-name: inventory dependency-version: 0.3.24 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: patches - dependency-name: libc dependency-version: 0.2.186 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: patches - dependency-name: pastey dependency-version: 0.2.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: patches - dependency-name: postgres-openssl dependency-version: 0.5.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: patches - dependency-name: pulsar dependency-version: 6.7.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: patches - dependency-name: quote dependency-version: 1.0.45 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: patches - dependency-name: roaring dependency-version: 0.11.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: patches - dependency-name: schannel dependency-version: 0.1.29 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: patches - dependency-name: semver dependency-version: 1.0.28 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: patches - dependency-name: web-sys dependency-version: 0.3.95 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: patches ... Signed-off-by: dependabot[bot] <support@github.com> * update licenses --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
…ectordotdev#25327) * feat(website): improve search ranking for component reference pages * Improve regex match
…ectordotdev#25227) * chore(deps): bump the serde group across 1 directory with 2 updates Bumps the serde group with 2 updates in the / directory: [serde_with](https://github.com/jonasbb/serde_with) and [serde_json](https://github.com/serde-rs/json). Updates `serde_with` from 3.14.0 to 3.17.0 - [Release notes](https://github.com/jonasbb/serde_with/releases) - [Commits](jonasbb/serde_with@v3.14.0...v3.17.0) Updates `serde_json` from 1.0.145 to 1.0.149 - [Release notes](https://github.com/serde-rs/json/releases) - [Commits](serde-rs/json@v1.0.145...v1.0.149) --- updated-dependencies: - dependency-name: serde_with dependency-version: 3.17.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: serde - dependency-name: serde_json dependency-version: 1.0.149 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: serde ... Signed-off-by: dependabot[bot] <support@github.com> * fix(core): update float size estimator to match serde_json's new formatter serde_json 1.0.147 switched from Ryū to Żmij for float-to-string formatting. The EstimatedJsonEncodedSizeOf impls for f32/f64 were using ryu directly to predict serde_json's output size, which silently drifts after the bump (for example ryu emits `1e16` while serde_json now emits `1e+16`). Swap to zmij so the estimate tracks what serde_json actually produces. * chore(codecs): update native_json fixtures for new serde_json float formatter Round-tripped the affected fixtures through serde_json 1.0.149 so positive exponents render as `e+N` (e.g. `1.797...e+308`) to match the new Żmij formatter introduced in 1.0.147. Event content is unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(deps): regenerate 3rd-party license file for zmij Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(ci): allow `mij` in spell check The changelog mentions the Żmij formatter; check-spelling tokenizes around `Ż` and flags the residual `mij`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(deps): centralize serde_with as a workspace dependency Move the serde_with version + features into [workspace.dependencies], mirroring how serde and serde_json are already declared. The workspace entry carries the union of features previously enabled across the three dependents (std, macros, chrono_0_4); since Cargo unions features across the workspace anyway, the resulting workspace build is identical. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(deps): use workspace serde_json in file-source-common Addresses review feedback from @thomasqueirozb on vectordotdev#25227. Cargo unions features across the workspace, so the union for any binary that pulls file-source-common alongside other serde_json dependents is unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update Cargo.lock --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Thomas <thomas.schneider@datadoghq.com>
…ectordotdev#25057) * feat(sinks): introduce a configurable retry strategy for http * feat(sinks): wire the RetryStrategy with the HttpRetryLogic * feat(sinks): wire RetryStrategy with all HttpStatusRetryLogic and all affected sinks * docs(sinks): http example config with custom retry strategy * feat(http sink): add integration test for custom retry strategy * docs(sinks): add changelog fragment * fix(http sink): update is_retriable_error logic to respect retry strategy * fix(http sink): add handling for NOT_IMPLEMENTED status in retry strategy * chore(sinks): format code * feat(sinks): implement timeout retry logic and add tests for non-retriable timeouts * docs(sinks): clarify that success status codes are not retried on RetryAll strategy * feat(sinks): improve typesafety of RetryStrategy StatusCode * docs(sinks http): enhance transport error classification for HTTP sinks with shared retry logic * feat(sinks http): add debug logging for unsuccessful HTTP responses in retry logic * refactor(http sink): avoid heap allocation on reasoning happy path * fix(http sink): format example config * chore: add custom attributes for numeric type in StatusCode metadata * docs(sinks http): update docs for http sinks supporting retry_strategy --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
…v#25339) The github.rest.issues.removeLabel call (DELETE /repos/{owner}/{repo}/issues/{issue_number}/labels/{name}) requires the issues:write permission per the GitHub REST API. The workflow only declared pull-requests:write, so the call returned 403 the first time all preconditions actually held simultaneously (MEMBER reviewer + label present + approval). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…event lost wakeup (vectordotdev#25195) * fix(windows_event_log): fix pre-drain ResetEvent race and add lost-wakeup regression tests The Windows Event Log service signals the pull-mode wait handle via SetEvent each time a new matching event is recorded. Because the handle is manual-reset, any SetEvent that fires between the last EvtNext call and the post-drain ResetEvent is silently lost — the subscription then hangs until the next OS event arrives (vectordotdev#25194). Fix: reset the handle *before* entering the drain loop. Signals raised during the drain are preserved because SetEvent on an already-signaled handle is a no-op. Re-arm (SetEvent) on early exits so the next pull_events revisits the channel without waiting for a fresh OS notification: - budget exhaustion - bookmark failure mid-batch - transient EvtNext error Regression tests: - test_pull_events_preserves_setevent_during_drain: installs DRAIN_STEP_HOOK to fire SetEvent mid-drain and asserts wait_for_events_blocking returns EventsAvailable, not Timeout. - test_speculative_pull_recovers_without_signal: manually clears the channel signal via ResetEvent, confirms wait times out, then asserts pull_events still returns events — proving the speculative timeout pull in mod.rs self-heals independently of signal state. Also: comment re-subscription break paths (ERROR_EVT_QUERY_RESULT_STALE and INVALID_POSITION) noting the speculative pull as a safety net if the re-subbed channel does not immediately re-signal; add serialization note to DRAIN_STEP_HOOK. * fix(windows_event_log): add speculative timeout pull, deduplicate processing, fix error handling Four related improvements to mod.rs: 1. Speculative pull on WaitResult::Timeout: call pull_events on every timeout cycle as a belt-and-suspenders self-heal. EvtNext returns ERROR_NO_MORE_ITEMS immediately on an empty channel (near-zero cost). If events are recovered a warning is emitted. Guarantees recovery within one timeout period regardless of the root cause of the lost wakeup. 2. Extract with_subscription_blocking helper: wraps the spawn_blocking ownership-transfer pattern (move subscription in, run blocking fn, return subscription + result). All three blocking calls (wait, normal pull, speculative pull) now use this helper instead of inlining spawn_blocking. 3. Extract process_event_batch helper: the parse/emit/send_batch/finalize sequence was duplicated verbatim between the EventsAvailable arm and the speculative-timeout arm. Extracted into a shared free async function. Rate limiting is applied consistently in both paths via the helper. 4. Fix error-handling asymmetry: the speculative pull Err branch previously only logged warn! and continued, so a non-recoverable error (access denied, channel not found) would spam warnings indefinitely. Now mirrors the EventsAvailable path: emit WindowsEventLogQueryError, break on non-recoverable errors, apply exponential backoff on recoverable ones. * test(windows_event_log source): harden lost-wakeup regression tests against flakiness Address two independent flakiness sources in the vectordotdev#25194 regression tests so the suite is stable on real Windows CI runners. test_pull_events_preserves_setevent_during_drain: - Replaced a 1000ms blocking wait with an immediate 0ms poll after pull_events returns, so the check measures only the reset/preserve behavior of pull_events and is not contaminated by unrelated Windows system events signaling the handle during a nonzero wait window. - Keyed the DRAIN_STEP_HOOK fire-once to the subscription's own signal handle so concurrent pull_events calls from other tests can't flip the hook first and SetEvent the wrong handle. test_speculative_pull_recovers_without_signal: - Same 500ms→0ms poll change, opposite direction: real events arriving during the wait would have re-signaled the manually-cleared handle and flipped the expected Timeout into a real signal result. - Seed a deterministic record via 'eventcreate' before subscription creation so the non-empty-events assertion is independent of whatever backlog the runner happens to have. Freshly provisioned images can have an empty Application log, which would otherwise cause pull_events(100) to legitimately return empty and false-fail the test. * fix(windows_event_log): implement Sync for subscription types to satisfy Send bound on source future EventLogSubscription and ChannelSubscription had unsafe impl Send but no Sync impl. Since &T: Send requires T: Sync, process_event_batch holding &EventLogSubscription across an .await made the entire source future !Send, causing a compile error (ICE + future-not-Send) in release builds. All mutation on these types requires &mut self; &self methods are read-only or delegate to already-Sync types (RateLimiter). The underlying Windows kernel handles are safe for concurrent access. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore(changelog): simplify windows_event_log fix fragment to user-focused one-liner Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(windows_event_log): prioritize shutdown signal * fix(windows_event_log): lighten speculative timeout pulls --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
…tdev#25345) fix(vector-config): remove type-level default on StatusCode The Configurable impl for http::StatusCode set a type-level default of 200, which the CUE generator propagated into the `uint` schema for each item of `Vec<StatusCode>` — including RetryStrategy::Custom.status_codes introduced in vectordotdev#25057. The CUE schema for #TypeArray hardcodes items.type._args.required = true, which forbids `default` on item primitives, so cue-build failed on every http-based sink with: components.sinks.<name>.configuration.retry_strategy.type.object: field not allowed Drop the type-level default. The only field that needs a default for StatusCode is sources::http_server::response_code, which already sets its own via #[serde(default = "default_http_response_code")]. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(windows_event_log): add standard source metadata * chore(changelog): add windows_event_log metadata fix note
…dev#25346) * fix(vector-config): remove type-level default on StatusCode The Configurable impl for http::StatusCode set a type-level default of 200, which the CUE generator propagated into the `uint` schema for each item of `Vec<StatusCode>` — including RetryStrategy::Custom.status_codes introduced in vectordotdev#25057. The CUE schema for #TypeArray hardcodes items.type._args.required = true, which forbids `default` on item primitives, so cue-build failed on every http-based sink with: components.sinks.<name>.configuration.retry_strategy.type.object: field not allowed Drop the type-level default. The only field that needs a default for StatusCode is sources::http_server::response_code, which already sets its own via #[serde(default = "default_http_response_code")]. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(ci): bump cue from v0.10.0 to v0.16.1 CI's cue v0.10.0 was lax enough that schema violations like the "field not allowed" error fixed in the previous commit slipped through the existing `Check Cue docs` step. Newer cue versions enforce closed struct constraints in disjunctions strictly, matching what local developers see when running the website build. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci(test): also run make cue-build in Check Cue docs The Check Cue docs job previously only ran make check-docs, which runs cue vet. Adding make cue-build (which runs cue export and materializes website/data/docs.json) catches schema-unification failures that the website build would hit and that vet may miss. The new step reuses the cue + VRL docs already produced by the prior make check-docs invocation, so it adds only the cost of one cue export. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(ci): bump cue to v0.16.1 in remaining installers Keep all three cue installers in lockstep with .github/actions/setup to avoid version skew that would let developer / website-Dockerfile paths accept cue sources the CI Check Cue docs job rejects. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…vectordotdev#25340) * Fix the clickhouse sink so that it only accepts arrow * Do not allow arrow stream or parquet on AWS S3 * reformat * Add changelog fragment * Remove stale comments * chore(aws_s3 sink): exhaustive match for batch_encoding default extension So adding a future S3BatchEncoding variant is a compile error rather than silently defaulting to "parquet". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(aws_s3 sink): rename shadowing parquet_config bind in tests Avoid rebinding parquet_config to a borrow of itself from config.batch_encoding; use a short p binding instead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(changelog): rewrite batch_encoding fragment for users Drop internal type names and dev jargon; lead with the user-visible behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(aws_s3 sink, clickhouse sink): reject unsupported batch_encoding codec at parse time Add per-sink deserialization-failure tests that pin the schema-tightening behavior introduced by the dedicated wrapper enums: - aws_s3 rejects codec: arrow_stream - clickhouse rejects codec: parquet Previously both codecs were accepted by serde and rejected later at sink-build time; the new wrapper enums move rejection up to parse time, and these tests prevent silent regression of that contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(aws_s3 sink): re-export S3BatchEncoding from aws_s3 module Programmatic users constructing S3SinkConfig directly need to be able to set the batch_encoding field. With config kept private in src/sinks/aws_s3/mod.rs, S3BatchEncoding was unnameable outside the crate, regressing callers that previously used BatchSerializerConfig::Parquet(...) at the same field. Gated by codecs-parquet to mirror the type and field. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add MetricName into vector-common * Use EventName in lib/ * Use MetricName in src/internal_events * Format * Fix EventName imports * Use MetricName in k8s * Use MetricName in aws_sqs * Use EventName everywhere and make clippy pass * Separate MetricName -> CounterName/GaugeName/HistogramName * Merge imports * Finish MetricName separation * chore: update check-events script to support typed metric name enums * chore: fix stale MetricName doc references in macro wrappers * chore: remove stale doc paragraph from counter! macro * Fix clippy.toml * Use first metric names instead of made up names to remove clippy::disallowed_macros * chore: replace MetricName with CounterName in Windows internal events
…otdev#25353) `HistogramName` was only used inside the `mod s3` block (gated on `sources-aws_s3`) but was imported at the parent module level under a broader cfg, breaking `cargo check --features sources-aws_sqs` with `#[deny(warnings)]`. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… tag cardinality transforms public (vectordotdev#25358) Make transform-related functions in aggregate & tag cardinality processors public
…dev#25355) Rename `add_wip_label.yml` and `remove_wip_label.yml` to `add_docs_review_label.yml` and `remove_docs_review_label.yml`, and switch the managed label from "work in progress" to "docs review on hold". The label string is now hoisted to a workflow-level `LABEL_NAME` env var so future renames are a one-line edit. Header comments note we can promote these to a reusable `workflow_call` workflow if a second use case appears. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
vectordotdev#25354) * chore(deps): upgrade hickory-proto to 0.26.1, ignore RUSTSEC-2026-0119 * Format * fix(dnstap source): include EDNS OPT record in additional_count
…ectordotdev#25731) Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…otdev#25728) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… across 1 directory (vectordotdev#25714) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
….2 (vectordotdev#25736) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…lticast group joining (vectordotdev#25711)
Restore the retry_strategy option lost when the sink config was flattened, and add a deprecated fallback that still parses the old protocol.* nested format while logging a migration warning. Signed-off-by: Sergey Kacheev <s.kacheev@gmail.com>
…C transport Signed-off-by: Sergey Kacheev <s.kacheev@gmail.com>
github-actions
Bot
added
docs review on hold
sakateka
changed the title
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi, Thomas (@thomasqueirozb)!
I’m hoping that the code for gRPC support in the OpenTelemetry sink will be merged into the main branch and won’t get lost. That’s why I’ve decided to help update the pull request and fix the issues that I encountered in my environment to get this PR working.
I’ve deployed a vector with gRPC support on a cluster consisting of several dozen Kubernetes nodes. In total, there are about 11 pipelines running there - all of them send data to the OpenTelemetry collector via gRPC with compression using zstd.
Everything is working just fine!
Summary
Vector configuration
How did you test this PR?
Change Type
Is this a breaking change?
Does this PR include user facing changes?
References
Notes
@vectordotdev/vectorto reach out to us regarding this PR.pre-pushhook, please see this template.make fmtmake check-clippy(if there are failures it's possible some of them can be fixed withmake clippy-fix)make testgit merge origin masterandgit push.Cargo.lock), pleaserun
make build-licensesto regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.