feat(data-pipeline): add meta struct blob setter - #2303
Conversation
Expose a C setter that copies an opaque binary value into a span's `meta_struct` map. Native tracers can now supply pre-encoded MessagePack without transferring ownership or requiring Rust to interpret it. Reject null span handles and non-UTF-8 keys. Reusing a key replaces its value, and the copied bytes remain owned by the span after the call.
BenchmarksComparisonCandidateCandidate benchmark detailsBaselineBaseline benchmark details |
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 220d6f8 | Docs | Datadog PR Page | Give us feedback! |
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
|
Draft dd-trace-rb example consumer: DataDog/dd-trace-rb#6130. |
69926bd to
27ad1af
Compare
| }; | ||
| span.0 | ||
| .meta_struct | ||
| .insert(key, Bytes::copy_from_slice(value.as_bytes())); |
There was a problem hiding this comment.
value.as_bytes() can panic when there is invalid input (example: null pointer, non-zero length). You can use try_as_bytes instead to get a more meaningful error.
| /// The `key` and opaque binary `value` are copied into the span. The value is | ||
| /// not interpreted or validated as MessagePack. | ||
| /// | ||
| /// Repeating a `key` appends a new entry rather than replacing the previous one. The last value |
There was a problem hiding this comment.
This exposes an internal implementation detail, so maybe shouldn't be here? I figure it's worth it since it could have performance implications for meta_struct
There was a problem hiding this comment.
Your comment was confusing to me until I read the following line:
/// written for a key is the one read back and the one serialized
And with that, agreed: it just appears to behave as a regular setter to the API consumer.
ekump
left a comment
There was a problem hiding this comment.
LGTM but someone else should review since I also pushed commits
hoolioh
left a comment
There was a problem hiding this comment.
LGTM, a little too much verbosity regarding comments though.
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated. # What does this PR do? Ensures the combined libdatadog profiling artifact enables `catch_panic` for data-pipeline FFI entry points. It also adds a packaged C regression that triggers capacity overflow and verifies that the FFI returns an error instead of aborting the process. # Motivation The standalone data-pipeline FFI crate enables panic containment by default, but the combined profiling artifact disables dependency defaults and did not re-enable it. This closes [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830) before the native trace exporter payload APIs ship together. # Additional Notes **Stack position: L0, current PR: [DataDog/libdatadog PR 2302](#2302 These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). Complete expected libdatadog review and merge order for that single v39 release: | Level | libdatadog PR | Matching dd-trace-rb consumer | | --- | --- | --- | | **L0 (current)** | **[DataDog/libdatadog PR 2302](#2302 | Release prerequisite; no payload API consumer | | L1 | [DataDog/libdatadog PR 2303](#2303) | [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130) | | L2 | [DataDog/libdatadog PR 2305](#2305) | [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129) | | L3 | [DataDog/libdatadog PR 2301](#2301) | [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134) | | L4 | [DataDog/libdatadog PR 2304](#2304) | [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) | | L5 | [DataDog/libdatadog PR 2300](#2300) | [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) | L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4. The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7. [DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake. L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # How to test the change? Prior local validation completed successfully: - `nix develop -c cargo test -p libdd-profiling-ffi --features data-pipeline-ffi` passed the combined profiling/data-pipeline FFI suite: 57 tests. - `nix develop -c cargo test -p libdd-data-pipeline-ffi --features catch_panic trace_chunks_new_with_overflow_capacity_returns_panic_error` passed. - `nix develop -c cargo test -p libdd-data-pipeline-ffi --features catch_panic begin_chunk_with_overflow_capacity_returns_panic_error` passed. - `nix develop -c cargo ffi-test --filter trace_exporter_panic_containment` built and passed the packaged C panic-containment regression. - `nix develop .#nightly --command cargo fmt --all -- --check` passed. - `nix develop -c cargo clippy -p libdd-profiling-ffi --all-targets --features data-pipeline-ffi -- -D warnings` passed. [APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Co-authored-by: ekump <edmund.kump@datadoghq.com> Co-authored-by: hoolioh <julio.gonzalez@datadoghq.com>
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated. # What does this PR do? Adds owned span-event construction and atomic attachment through the data-pipeline FFI. It supports typed scalar attributes, homogeneous typed arrays, empty arrays, event ordering, and UTF-8 validation. # Motivation The Rust trace model supports span events, but native exporter consumers cannot populate them. This fills the span-event portion of the payload parity work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # Additional Notes **Stack position: L3, current PR: [DataDog/libdatadog PR 2301](#2301 These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). Complete expected libdatadog review and merge order for that single v39 release: | Level | libdatadog PR | Matching dd-trace-rb consumer | | --- | --- | --- | | L0 | [DataDog/libdatadog PR 2302](#2302) | Release prerequisite; no payload API consumer | | L1 | [DataDog/libdatadog PR 2303](#2303) | [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130) | | L2 | [DataDog/libdatadog PR 2305](#2305) | [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129) | | **L3 (current)** | **[DataDog/libdatadog PR 2301](#2301 | **[R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134 | | L4 | [DataDog/libdatadog PR 2304](#2304) | [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) | | L5 | [DataDog/libdatadog PR 2300](#2300) | [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) | L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4. The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7. [DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake. L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # How to test the change? Prior local validation completed successfully: - `nix develop --command cargo test -p libdd-data-pipeline-ffi` passed 58 tests, including scalar values, homogeneous typed arrays, empty arrays, ordering, ownership, and atomic failure coverage. - `nix develop --command cargo clippy -p libdd-data-pipeline-ffi --all-targets -- -D warnings` passed. - `nix develop .#nightly --command cargo fmt --all -- --check` passed. - `nix develop --command env LIBDATADOG_VENDOR_OVERRIDE=/tmp/opencode/native-span-events-ruby bundle exec rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb spec/datadog/tracing/transport/native/send_traces_spec.rb spec/datadog/tracing/transport/native/transport_spec.rb spec/datadog/tracing/transport/traces_spec.rb spec/datadog/tracing/span_event_spec.rb spec/datadog/tracing/span_operation_spec.rb` passed 123 native and producer examples, including native wire output and legacy JSON fallback. - Targeted Standard and Steep checks for the native transport and span-event paths passed. [APMSP-3219]: https://datadoghq.atlassian.net/browse/APMSP-3219?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Julio <julio.gonzalez@datadoghq.com> Co-authored-by: Edmund Kump <edmund.kump@datadoghq.com> Co-authored-by: Julio Gonzalez <107922352+hoolioh@users.noreply.github.com>
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated. # What does this PR do? Adds an atomic batched FFI for complete span links, including 128-bit trace IDs, span IDs, flags, tracestate, attributes, and ordering. All borrowed input is validated and copied before insertion. # Motivation The Rust trace model supports span links, but native exporter consumers cannot populate it. This fills the span-link portion of the payload parity work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # Additional Notes `dropped_attributes_count` is intentionally deferred by additive commit [961044e](961044e). Neither the Agent v0.4 nor indexed V1 span-link schema defines the field, and current Datadog tracers do not produce non-zero counts. The original implementation remains in the preceding commit and can be restored by reverting the deferral commit. **Stack position: L2, current PR: [DataDog/libdatadog PR 2305](#2305 These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). Complete expected libdatadog review and merge order for that single v39 release: | Level | libdatadog PR | Matching dd-trace-rb consumer | | --- | --- | --- | | L0 | [DataDog/libdatadog PR 2302](#2302) | Release prerequisite; no payload API consumer | | L1 | [DataDog/libdatadog PR 2303](#2303) | [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130) | | **L2 (current)** | **[DataDog/libdatadog PR 2305](#2305 | **[R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129 | | L3 | [DataDog/libdatadog PR 2301](#2301) | [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134) | | L4 | [DataDog/libdatadog PR 2304](#2304) | [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) | | L5 | [DataDog/libdatadog PR 2300](#2300) | [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) | L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4. The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7. [DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake. L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # How to test the change? Prior local validation completed successfully: - `nix develop -c cargo nextest run -p libdd-data-pipeline-ffi` passed 59 tests, including atomic invalid-UTF-8 failure, ordering, and ownership coverage. - `nix develop -c cargo test -p libdd-trace-utils` passed the unit, integration, and doc-test coverage for all affected encoders and decoders. - `nix develop -c cargo clippy -p libdd-trace-utils -p libdd-data-pipeline-ffi --all-targets -- -D warnings` passed. - `nix develop .#nightly --command cargo fmt --all -- --check` passed. - `nix develop --command bundle exec rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb spec/datadog/tracing/transport/native/transport_spec.rb spec/datadog/tracing/transport/native/conformance_spec.rb` passed 119 native transport examples against the locally built artifact. - `nix develop --command bundle exec rspec spec/datadog/tracing/span_link_spec.rb spec/datadog/tracing/distributed/propagation_spec.rb` passed 69 distributed-propagation examples with 1 existing pending example. - `nix develop --command env BUNDLE_GEMFILE=gemfiles/ruby_4.0_opentelemetry.gemfile bundle exec rspec spec/datadog/opentelemetry_spec.rb` passed 164 OpenTelemetry examples. - Full Standard and Steep checks passed on the matching consumer branch. [APMSP-3219]: https://datadoghq.atlassian.net/browse/APMSP-3219?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Edmund Kump <edmund.kump@datadoghq.com> Co-authored-by: Julio <julio.gonzalez@datadoghq.com> Co-authored-by: Julio Gonzalez <107922352+hoolioh@users.noreply.github.com>
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated.
What does this PR do?
Adds an opaque per-key
meta_structblob setter toTracerSpan. Keys are validated as UTF-8, while values are copied as uninterpreted binary data. This transitional API lets consumers pass their existing per-key MessagePack blobs without changing the Rust trace model.Motivation
Native exporter consumers cannot currently populate
meta_struct, which blocks payload parity tracked by APMSP-3219. The opaque setter provides the safe first stage before the language-neutral structured-value APIs.Additional Notes
Stack position: L1, current PR: DataDog/libdatadog PR 2303.
These open L0-L5 changes are not included in libdatadog v38.0.0. They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by R3 DataDog/dd-trace-rb PR 6143.
Complete expected libdatadog review and merge order for that single v39 release:
L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4.
The matching payload consumers are R4 DataDog/dd-trace-rb PR 6130, R5 DataDog/dd-trace-rb PR 6129, R6 DataDog/dd-trace-rb PR 6134, R7 DataDog/dd-trace-rb PR 6132, and R8 DataDog/dd-trace-rb PR 6133. All depend through the common v39 uptake in R3 DataDog/dd-trace-rb PR 6143. The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7.
DataDog/libdatadog-rb v37 PR 65 and DataDog/libdatadog-rb v38 PR 66 are separate ordered prerequisite releases only for R2 DataDog/dd-trace-rb PR 6131 and its
ForkSafeRuntimeuptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake.L0 tracks APMSP-3830; L1-L5 implement the payload work tracked by APMSP-3219.
How to test the change?
Prior local validation completed successfully:
nix develop -c cargo test -p libdd-data-pipeline-ffipassed, including arbitrary binary values, overwrite semantics, null handles, and invalid UTF-8 keys.nix develop -c cargo clippy -p libdd-data-pipeline-ffi --all-targets -- -D warningspassed.nix develop .#nightly -c cargo fmt --all -- --checkpassed.nix develop -c cargo run --bin release -- --out /tmp/opencode/native-meta-struct-libdatadog/x86_64-linuxand consumed successfully by the matching dd-trace-rb branch.nix develop .#ruby34 --command bundle exec rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb spec/datadog/tracing/transport/native/conformance_spec.rb spec/datadog/tracing/transport/native/transport_spec.rb spec/datadog/appsec/actions_handler/serializable_backtrace_spec.rbpassed 116 examples, including AppSec and AI Guard-shaped wire conformance.