feat(data-pipeline): add span events FFI - #2301
Merged
Merged
Conversation
Expose an owned span event builder through the C API. Native tracers can set string, boolean, integer, and double attributes, including homogeneous arrays, before consuming the event into a span. Copy string inputs and preserve event insertion order. Invalid or null handles return errors without attaching a partial event.
Verify that every typed array setter accepts an empty array and records it as an array value rather than dropping the attribute. This preserves the caller's type information even when no elements are present. Also verify that invalid UTF-8 in scalar and array string values returns `InvalidInput` without partially mutating the detached event.
BenchmarksComparisonCandidateCandidate benchmark detailsBaselineBaseline benchmark details |
Contributor
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 1512b95 | Docs | Datadog PR Page | Give us feedback! |
Contributor
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
Member
Author
|
Draft dd-trace-rb example consumer: DataDog/dd-trace-rb#6134. |
lloeki
force-pushed
the
lloeki/native-span-events
branch
from
July 30, 2026 12:03
344d10b to
127a5d1
Compare
This was referenced Jul 30, 2026
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Aug 4, 2026
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated. # What does this PR do? Ensures the combined libdatadog profiling artifact enables `catch_panic` for data-pipeline FFI entry points. It also adds a packaged C regression that triggers capacity overflow and verifies that the FFI returns an error instead of aborting the process. # Motivation The standalone data-pipeline FFI crate enables panic containment by default, but the combined profiling artifact disables dependency defaults and did not re-enable it. This closes [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830) before the native trace exporter payload APIs ship together. # Additional Notes **Stack position: L0, current PR: [DataDog/libdatadog PR 2302](#2302 These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). Complete expected libdatadog review and merge order for that single v39 release: | Level | libdatadog PR | Matching dd-trace-rb consumer | | --- | --- | --- | | **L0 (current)** | **[DataDog/libdatadog PR 2302](#2302 | Release prerequisite; no payload API consumer | | L1 | [DataDog/libdatadog PR 2303](#2303) | [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130) | | L2 | [DataDog/libdatadog PR 2305](#2305) | [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129) | | L3 | [DataDog/libdatadog PR 2301](#2301) | [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134) | | L4 | [DataDog/libdatadog PR 2304](#2304) | [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) | | L5 | [DataDog/libdatadog PR 2300](#2300) | [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) | L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4. The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7. [DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake. L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # How to test the change? Prior local validation completed successfully: - `nix develop -c cargo test -p libdd-profiling-ffi --features data-pipeline-ffi` passed the combined profiling/data-pipeline FFI suite: 57 tests. - `nix develop -c cargo test -p libdd-data-pipeline-ffi --features catch_panic trace_chunks_new_with_overflow_capacity_returns_panic_error` passed. - `nix develop -c cargo test -p libdd-data-pipeline-ffi --features catch_panic begin_chunk_with_overflow_capacity_returns_panic_error` passed. - `nix develop -c cargo ffi-test --filter trace_exporter_panic_containment` built and passed the packaged C panic-containment regression. - `nix develop .#nightly --command cargo fmt --all -- --check` passed. - `nix develop -c cargo clippy -p libdd-profiling-ffi --all-targets --features data-pipeline-ffi -- -D warnings` passed. [APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Co-authored-by: ekump <edmund.kump@datadoghq.com> Co-authored-by: hoolioh <julio.gonzalez@datadoghq.com>
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Aug 4, 2026
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated. # What does this PR do? Adds an opaque per-key `meta_struct` blob setter to `TracerSpan`. Keys are validated as UTF-8, while values are copied as uninterpreted binary data. This transitional API lets consumers pass their existing per-key MessagePack blobs without changing the Rust trace model. # Motivation Native exporter consumers cannot currently populate `meta_struct`, which blocks payload parity tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). The opaque setter provides the safe first stage before the language-neutral structured-value APIs. # Additional Notes **Stack position: L1, current PR: [DataDog/libdatadog PR 2303](#2303 These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). Complete expected libdatadog review and merge order for that single v39 release: | Level | libdatadog PR | Matching dd-trace-rb consumer | | --- | --- | --- | | L0 | [DataDog/libdatadog PR 2302](#2302) | Release prerequisite; no payload API consumer | | **L1 (current)** | **[DataDog/libdatadog PR 2303](#2303 | **[R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130 | | L2 | [DataDog/libdatadog PR 2305](#2305) | [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129) | | L3 | [DataDog/libdatadog PR 2301](#2301) | [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134) | | L4 | [DataDog/libdatadog PR 2304](#2304) | [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) | | L5 | [DataDog/libdatadog PR 2300](#2300) | [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) | L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4. The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7. [DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake. L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # How to test the change? Prior local validation completed successfully: - `nix develop -c cargo test -p libdd-data-pipeline-ffi` passed, including arbitrary binary values, overwrite semantics, null handles, and invalid UTF-8 keys. - `nix develop -c cargo clippy -p libdd-data-pipeline-ffi --all-targets -- -D warnings` passed. - `nix develop .#nightly -c cargo fmt --all -- --check` passed. - The generated C header was rebuilt through `nix develop -c cargo run --bin release -- --out /tmp/opencode/native-meta-struct-libdatadog/x86_64-linux` and consumed successfully by the matching dd-trace-rb branch. - `nix develop .#ruby34 --command bundle exec rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb spec/datadog/tracing/transport/native/conformance_spec.rb spec/datadog/tracing/transport/native/transport_spec.rb spec/datadog/appsec/actions_handler/serializable_backtrace_spec.rb` passed 116 examples, including AppSec and AI Guard-shaped wire conformance. - Targeted Standard and Steep checks for the native transport passed. [APMSP-3219]: https://datadoghq.atlassian.net/browse/APMSP-3219?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Co-authored-by: ekump <edmund.kump@datadoghq.com> Co-authored-by: loic.nageleisen <loic.nageleisen@datadoghq.com>
- Rprevent as_slice from generating a panic by using try_as_slice, malformed slices returns ErrorCode::InvalidArgument. Invalid UTF-8 still returns ErrorCode::InvalidInput. - Accept NULL in ddog_tracer_span_free, matching ddog_tracer_span_event_free. - Deduplicate the bool/int/double array setters behind set_event_array_attribute. - Collect ddog_tracer_span_event_set_string_array once rather than building an intermediate Vec<BytesString>, saving one allocation per call. - Replace the match arms with Result combinators, removing the early returns that behaved differently inside and outside catch_panic!. - Cover the untested boundary paths: NULL event on every setter, invalid UTF-8 keys, ddog_tracer_span_event_new failures leaving out_handle unwritten, attribute overwrite semantics, NULL event in ddog_tracer_span_add_event, and attributes surviving attachment.
hoolioh
approved these changes
Aug 4, 2026
hoolioh
left a comment
hoolioh
left a comment
Contributor
There was a problem hiding this comment.
Made some changes, mostly improving handling inputs, solving UB and fix some extra allocations.
Contributor
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
Build pipeline has failing jobs for dab2abc: What to do next?
DetailsSince those jobs are not marked as being allowed to fail, the pipeline will most likely fail. |
Contributor
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
Build pipeline has failing jobs for 0e7979a: What to do next?
DetailsSince those jobs are not marked as being allowed to fail, the pipeline will most likely fail. |
Contributor
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
Build pipeline has failing jobs for 3c480f7: What to do next?
DetailsSince those jobs are not marked as being allowed to fail, the pipeline will most likely fail. |
hoolioh
added a commit
that referenced
this pull request
Aug 5, 2026
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated. # What does this PR do? Adds an atomic batched FFI for complete span links, including 128-bit trace IDs, span IDs, flags, tracestate, attributes, and ordering. All borrowed input is validated and copied before insertion. # Motivation The Rust trace model supports span links, but native exporter consumers cannot populate it. This fills the span-link portion of the payload parity work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # Additional Notes `dropped_attributes_count` is intentionally deferred by additive commit [961044e](961044e). Neither the Agent v0.4 nor indexed V1 span-link schema defines the field, and current Datadog tracers do not produce non-zero counts. The original implementation remains in the preceding commit and can be restored by reverting the deferral commit. **Stack position: L2, current PR: [DataDog/libdatadog PR 2305](#2305 These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). Complete expected libdatadog review and merge order for that single v39 release: | Level | libdatadog PR | Matching dd-trace-rb consumer | | --- | --- | --- | | L0 | [DataDog/libdatadog PR 2302](#2302) | Release prerequisite; no payload API consumer | | L1 | [DataDog/libdatadog PR 2303](#2303) | [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130) | | **L2 (current)** | **[DataDog/libdatadog PR 2305](#2305 | **[R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129 | | L3 | [DataDog/libdatadog PR 2301](#2301) | [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134) | | L4 | [DataDog/libdatadog PR 2304](#2304) | [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) | | L5 | [DataDog/libdatadog PR 2300](#2300) | [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) | L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4. The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7. [DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake. L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # How to test the change? Prior local validation completed successfully: - `nix develop -c cargo nextest run -p libdd-data-pipeline-ffi` passed 59 tests, including atomic invalid-UTF-8 failure, ordering, and ownership coverage. - `nix develop -c cargo test -p libdd-trace-utils` passed the unit, integration, and doc-test coverage for all affected encoders and decoders. - `nix develop -c cargo clippy -p libdd-trace-utils -p libdd-data-pipeline-ffi --all-targets -- -D warnings` passed. - `nix develop .#nightly --command cargo fmt --all -- --check` passed. - `nix develop --command bundle exec rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb spec/datadog/tracing/transport/native/transport_spec.rb spec/datadog/tracing/transport/native/conformance_spec.rb` passed 119 native transport examples against the locally built artifact. - `nix develop --command bundle exec rspec spec/datadog/tracing/span_link_spec.rb spec/datadog/tracing/distributed/propagation_spec.rb` passed 69 distributed-propagation examples with 1 existing pending example. - `nix develop --command env BUNDLE_GEMFILE=gemfiles/ruby_4.0_opentelemetry.gemfile bundle exec rspec spec/datadog/opentelemetry_spec.rb` passed 164 OpenTelemetry examples. - Full Standard and Steep checks passed on the matching consumer branch. [APMSP-3219]: https://datadoghq.atlassian.net/browse/APMSP-3219?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Edmund Kump <edmund.kump@datadoghq.com> Co-authored-by: Julio <julio.gonzalez@datadoghq.com> Co-authored-by: Julio Gonzalez <107922352+hoolioh@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated.
What does this PR do?
Adds owned span-event construction and atomic attachment through the data-pipeline FFI. It supports typed scalar attributes, homogeneous typed arrays, empty arrays, event ordering, and UTF-8 validation.
Motivation
The Rust trace model supports span events, but native exporter consumers cannot populate them. This fills the span-event portion of the payload parity work tracked by APMSP-3219.
Additional Notes
Stack position: L3, current PR: DataDog/libdatadog PR 2301.
These open L0-L5 changes are not included in libdatadog v38.0.0. They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by R3 DataDog/dd-trace-rb PR 6143.
Complete expected libdatadog review and merge order for that single v39 release:
L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4.
The matching payload consumers are R4 DataDog/dd-trace-rb PR 6130, R5 DataDog/dd-trace-rb PR 6129, R6 DataDog/dd-trace-rb PR 6134, R7 DataDog/dd-trace-rb PR 6132, and R8 DataDog/dd-trace-rb PR 6133. All depend through the common v39 uptake in R3 DataDog/dd-trace-rb PR 6143. The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7.
DataDog/libdatadog-rb v37 PR 65 and DataDog/libdatadog-rb v38 PR 66 are separate ordered prerequisite releases only for R2 DataDog/dd-trace-rb PR 6131 and its
ForkSafeRuntimeuptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake.L0 tracks APMSP-3830; L1-L5 implement the payload work tracked by APMSP-3219.
How to test the change?
Prior local validation completed successfully:
nix develop --command cargo test -p libdd-data-pipeline-ffipassed 58 tests, including scalar values, homogeneous typed arrays, empty arrays, ordering, ownership, and atomic failure coverage.nix develop --command cargo clippy -p libdd-data-pipeline-ffi --all-targets -- -D warningspassed.nix develop .#nightly --command cargo fmt --all -- --checkpassed.nix develop --command env LIBDATADOG_VENDOR_OVERRIDE=/tmp/opencode/native-span-events-ruby bundle exec rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb spec/datadog/tracing/transport/native/send_traces_spec.rb spec/datadog/tracing/transport/native/transport_spec.rb spec/datadog/tracing/transport/traces_spec.rb spec/datadog/tracing/span_event_spec.rb spec/datadog/tracing/span_operation_spec.rbpassed 123 native and producer examples, including native wire output and legacy JSON fallback.