Skip to content

Encode meta struct values natively - #6132

Draft
lloeki wants to merge 4 commits into
lloeki/native-meta-struct-m1-consumerfrom
lloeki/native-meta-struct-m2-consumer
Draft

Encode meta struct values natively#6132
lloeki wants to merge 4 commits into
lloeki/native-meta-struct-m1-consumerfrom
lloeki/native-meta-struct-m2-consumer

Conversation

@lloeki

@lloeki lloeki commented Jul 30, 2026

Copy link
Copy Markdown
Member

AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated.

What does this PR do?

Replaces Ruby MessagePack encoding for supported meta_struct values with the native structured-value encoder while retaining M1 blob insertion. It uses stable C-owned token memory, cycle and depth validation, and deterministic cleanup. Known AppSec and AI Guard shapes are supported; unknown custom objects fail visibly.

Motivation:

M1 preserves payload parity through opaque blobs, but native structured encoding removes Ruby-side serialization for supported values and makes conversion ownership explicit. This is the next structured-metadata step tracked by APMSP-3219.

Change log entry

Yes. Native trace export now encodes supported structured span metadata values without Ruby MessagePack encoding.

Additional Notes:

How to test the change?

  • Ran the native transport suite: 122 examples, 0 failures, covering supported scalar and collection values, stable token memory, cycle and depth rejection, deterministic cleanup, and visible custom-object failures.
  • Ran the AppSec suite: 807 examples, 0 failures.
  • Ran the AI Guard suite: 125 examples, 0 failures.
  • Ran Standard and Steep successfully.

@lloeki lloeki added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Jul 30, 2026
@dd-octo-sts dd-octo-sts Bot added the appsec Application Security monitoring product label Jul 30, 2026
@dd-octo-sts

dd-octo-sts Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Thank you for updating Change log entry section 👏

Visited at: 2026-07-30 12:12:00 UTC

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Pipelines

⚠️ Warnings

🚦 3 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-rb | build-gem   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-rb | microbenchmarks: [other]   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-rb | microbenchmarks: [profiling]   View in Datadog   GitLab

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 9729457 | Docs | Datadog PR Page | Give us feedback!

@lloeki
lloeki force-pushed the lloeki/native-meta-struct-m1-consumer branch from 5061fef to 3c80594 Compare July 30, 2026 12:01
@lloeki
lloeki force-pushed the lloeki/native-meta-struct-m2-consumer branch from 95f8e81 to 18715d4 Compare July 30, 2026 12:03
@lloeki
lloeki force-pushed the lloeki/native-meta-struct-m1-consumer branch from 3c80594 to c193d99 Compare August 3, 2026 14:54
@lloeki
lloeki force-pushed the lloeki/native-meta-struct-m2-consumer branch from 18715d4 to c31165e Compare August 3, 2026 14:54
gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/libdatadog that referenced this pull request Aug 4, 2026
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated.

# What does this PR do?

Ensures the combined libdatadog profiling artifact enables `catch_panic` for data-pipeline FFI entry points. It also adds a packaged C regression that triggers capacity overflow and verifies that the FFI returns an error instead of aborting the process.

# Motivation

The standalone data-pipeline FFI crate enables panic containment by default, but the combined profiling artifact disables dependency defaults and did not re-enable it. This closes [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830) before the native trace exporter payload APIs ship together.

# Additional Notes

**Stack position: L0, current PR: [DataDog/libdatadog PR 2302](#2302

These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143).

Complete expected libdatadog review and merge order for that single v39 release:

| Level | libdatadog PR | Matching dd-trace-rb consumer |
| --- | --- | --- |
| **L0 (current)** | **[DataDog/libdatadog PR 2302](#2302 | Release prerequisite; no payload API consumer |
| L1 | [DataDog/libdatadog PR 2303](#2303) | [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130) |
| L2 | [DataDog/libdatadog PR 2305](#2305) | [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129) |
| L3 | [DataDog/libdatadog PR 2301](#2301) | [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134) |
| L4 | [DataDog/libdatadog PR 2304](#2304) | [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) |
| L5 | [DataDog/libdatadog PR 2300](#2300) | [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) |

L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4.

The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7.

[DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake.

L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219).

# How to test the change?

Prior local validation completed successfully:

- `nix develop -c cargo test -p libdd-profiling-ffi --features data-pipeline-ffi` passed the combined profiling/data-pipeline FFI suite: 57 tests.
- `nix develop -c cargo test -p libdd-data-pipeline-ffi --features catch_panic trace_chunks_new_with_overflow_capacity_returns_panic_error` passed.
- `nix develop -c cargo test -p libdd-data-pipeline-ffi --features catch_panic begin_chunk_with_overflow_capacity_returns_panic_error` passed.
- `nix develop -c cargo ffi-test --filter trace_exporter_panic_containment` built and passed the packaged C panic-containment regression.
- `nix develop .#nightly --command cargo fmt --all -- --check` passed.
- `nix develop -c cargo clippy -p libdd-profiling-ffi --all-targets --features data-pipeline-ffi -- -D warnings` passed.




[APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
[APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


Co-authored-by: ekump <edmund.kump@datadoghq.com>
Co-authored-by: hoolioh <julio.gonzalez@datadoghq.com>
@lloeki
lloeki force-pushed the lloeki/native-meta-struct-m1-consumer branch from c193d99 to 27476f6 Compare August 4, 2026 11:20
lloeki added 2 commits August 4, 2026 13:20
Convert supported `meta_struct` primitives and containers directly to
libdatadog value tokens instead of invoking arbitrary MessagePack
encoders. Native validation now rejects unsupported types, cycles,
excessive depth, invalid UTF-8, and out-of-range integers predictably.

Expose `SerializableBacktrace#to_h` so AppSec stack data uses the same
primitive representation for native and MessagePack serialization.
Ruby normalization can mutate strings or trigger GC after an earlier
`RSTRING_PTR` was borrowed. Copy every key and value byte into owned
scratch storage before creating the Rust span so all token pointers stay
valid through the libdatadog calls.

Prepare the complete `meta_struct` under deterministic cleanup before
transferring any values to the native span.
@lloeki
lloeki force-pushed the lloeki/native-meta-struct-m2-consumer branch from c31165e to 2ec40fd Compare August 4, 2026 11:20
gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/libdatadog that referenced this pull request Aug 4, 2026
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated.

# What does this PR do?

Adds an opaque per-key `meta_struct` blob setter to `TracerSpan`. Keys are validated as UTF-8, while values are copied as uninterpreted binary data. This transitional API lets consumers pass their existing per-key MessagePack blobs without changing the Rust trace model.

# Motivation

Native exporter consumers cannot currently populate `meta_struct`, which blocks payload parity tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). The opaque setter provides the safe first stage before the language-neutral structured-value APIs.

# Additional Notes

**Stack position: L1, current PR: [DataDog/libdatadog PR 2303](#2303

These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143).

Complete expected libdatadog review and merge order for that single v39 release:

| Level | libdatadog PR | Matching dd-trace-rb consumer |
| --- | --- | --- |
| L0 | [DataDog/libdatadog PR 2302](#2302) | Release prerequisite; no payload API consumer |
| **L1 (current)** | **[DataDog/libdatadog PR 2303](#2303 | **[R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130 |
| L2 | [DataDog/libdatadog PR 2305](#2305) | [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129) |
| L3 | [DataDog/libdatadog PR 2301](#2301) | [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134) |
| L4 | [DataDog/libdatadog PR 2304](#2304) | [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) |
| L5 | [DataDog/libdatadog PR 2300](#2300) | [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) |

L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4.

The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7.

[DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake.

L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219).

# How to test the change?

Prior local validation completed successfully:

- `nix develop -c cargo test -p libdd-data-pipeline-ffi` passed, including arbitrary binary values, overwrite semantics, null handles, and invalid UTF-8 keys.
- `nix develop -c cargo clippy -p libdd-data-pipeline-ffi --all-targets -- -D warnings` passed.
- `nix develop .#nightly -c cargo fmt --all -- --check` passed.
- The generated C header was rebuilt through `nix develop -c cargo run --bin release -- --out /tmp/opencode/native-meta-struct-libdatadog/x86_64-linux` and consumed successfully by the matching dd-trace-rb branch.
- `nix develop .#ruby34 --command bundle exec rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb spec/datadog/tracing/transport/native/conformance_spec.rb spec/datadog/tracing/transport/native/transport_spec.rb spec/datadog/appsec/actions_handler/serializable_backtrace_spec.rb` passed 116 examples, including AppSec and AI Guard-shaped wire conformance.
- Targeted Standard and Steep checks for the native transport passed.




[APMSP-3219]: https://datadoghq.atlassian.net/browse/APMSP-3219?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
[APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


Co-authored-by: ekump <edmund.kump@datadoghq.com>
Co-authored-by: loic.nageleisen <loic.nageleisen@datadoghq.com>
@lloeki

lloeki commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2ec40fd78a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +412 to +413
if (rb_enc_str_coderange(string) == ENC_CODERANGE_BROKEN) {
rb_raise(rb_eEncodingError, "meta_struct string is not valid UTF-8");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve truncated multibyte AI Guard content

When AI Guard's max_content_size_bytes boundary falls inside a multibyte character, the existing byteslice calls in lib/datadog/ai_guard/evaluation.rb:91,101 produce a UTF-8-encoded string with invalid bytes. This new rejection then raises while converting the span, causing build_and_send_traces to abort the entire native trace batch, whereas the previous MessagePack path serialized those bytes. Make AI Guard truncation character-safe or preserve such values without rejecting the span.

Useful? React with 👍 / 👎.

lloeki added 2 commits August 5, 2026 09:59
Carry the reviewed M1 contract documentation and coverage into the next
structured-metadata stage without rewriting its existing commits.

# Conflicts:
#	ext/libdatadog_api/trace_exporter.c
#	spec/datadog/tracing/transport/native/tracer_span_spec.rb
Byte limits can split the trailing bytes of a multibyte character and make
structured metadata invalid. Remove incomplete sequences after truncation for
both text and multimodal content while preserving the configured byte limit.
@dd-octo-sts dd-octo-sts Bot added the ai-guard label Aug 5, 2026
hoolioh added a commit to DataDog/libdatadog that referenced this pull request Aug 5, 2026
AI-generated code disclosure: this PR was implemented with substantial
AI assistance and manually reviewed and validated.

# What does this PR do?

Adds owned span-event construction and atomic attachment through the
data-pipeline FFI. It supports typed scalar attributes, homogeneous
typed arrays, empty arrays, event ordering, and UTF-8 validation.

# Motivation

The Rust trace model supports span events, but native exporter consumers
cannot populate them. This fills the span-event portion of the payload
parity work tracked by
[APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219).

# Additional Notes

**Stack position: L3, current PR: [DataDog/libdatadog PR
2301](#2301

These open L0-L5 changes are not included in [libdatadog
v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0).
They are planned to ship together in a single libdatadog v39 release,
with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR
6143](DataDog/dd-trace-rb#6143).

Complete expected libdatadog review and merge order for that single v39
release:

| Level | libdatadog PR | Matching dd-trace-rb consumer |
| --- | --- | --- |
| L0 | [DataDog/libdatadog PR
2302](#2302) | Release
prerequisite; no payload API consumer |
| L1 | [DataDog/libdatadog PR
2303](#2303) | [R4
DataDog/dd-trace-rb PR
6130](DataDog/dd-trace-rb#6130) |
| L2 | [DataDog/libdatadog PR
2305](#2305) | [R5
DataDog/dd-trace-rb PR
6129](DataDog/dd-trace-rb#6129) |
| **L3 (current)** | **[DataDog/libdatadog PR
2301](#2301 | **[R6
DataDog/dd-trace-rb PR
6134](DataDog/dd-trace-rb#6134 |
| L4 | [DataDog/libdatadog PR
2304](#2304) | [R7
DataDog/dd-trace-rb PR
6132](DataDog/dd-trace-rb#6132) |
| L5 | [DataDog/libdatadog PR
2300](#2300) | [R8
DataDog/dd-trace-rb PR
6133](DataDog/dd-trace-rb#6133) |

L0-L3 are independent and can be reviewed in parallel. Merging them in
the listed order is still recommended because it reduces integration and
release-branch conflicts, and lands panic containment before the new
payload surfaces. L4 is stacked on L1, and L5 is stacked on L4.

The matching payload consumers are [R4 DataDog/dd-trace-rb PR
6130](DataDog/dd-trace-rb#6130), [R5
DataDog/dd-trace-rb PR
6129](DataDog/dd-trace-rb#6129), [R6
DataDog/dd-trace-rb PR
6134](DataDog/dd-trace-rb#6134), [R7
DataDog/dd-trace-rb PR
6132](DataDog/dd-trace-rb#6132), and [R8
DataDog/dd-trace-rb PR
6133](DataDog/dd-trace-rb#6133). All depend
through the common v39 uptake in [R3 DataDog/dd-trace-rb PR
6143](DataDog/dd-trace-rb#6143). The downstream
review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also
depends on R4, and R8 depends on R7.

[DataDog/libdatadog-rb v37 PR
65](DataDog/libdatadog-rb#65) and
[DataDog/libdatadog-rb v38 PR
66](DataDog/libdatadog-rb#66) are separate
ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR
6131](DataDog/dd-trace-rb#6131) and its
`ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or
payload APIs. R3 is stacked on R2, but R3, rather than either
prerequisite package, is the common v39 payload-API uptake.

L0 tracks
[APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5
implement the payload work tracked by
[APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219).

# How to test the change?

Prior local validation completed successfully:

- `nix develop --command cargo test -p libdd-data-pipeline-ffi` passed
58 tests, including scalar values, homogeneous typed arrays, empty
arrays, ordering, ownership, and atomic failure coverage.
- `nix develop --command cargo clippy -p libdd-data-pipeline-ffi
--all-targets -- -D warnings` passed.
- `nix develop .#nightly --command cargo fmt --all -- --check` passed.
- `nix develop --command env
LIBDATADOG_VENDOR_OVERRIDE=/tmp/opencode/native-span-events-ruby bundle
exec rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb
spec/datadog/tracing/transport/native/send_traces_spec.rb
spec/datadog/tracing/transport/native/transport_spec.rb
spec/datadog/tracing/transport/traces_spec.rb
spec/datadog/tracing/span_event_spec.rb
spec/datadog/tracing/span_operation_spec.rb` passed 123 native and
producer examples, including native wire output and legacy JSON
fallback.
- Targeted Standard and Steep checks for the native transport and
span-event paths passed.




[APMSP-3219]:
https://datadoghq.atlassian.net/browse/APMSP-3219?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
[APMSP-3830]:
https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

---------

Co-authored-by: Julio <julio.gonzalez@datadoghq.com>
Co-authored-by: Edmund Kump <edmund.kump@datadoghq.com>
Co-authored-by: Julio Gonzalez <107922352+hoolioh@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos ai-guard appsec Application Security monitoring product

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant