feat(trace sampler): implement rare sampler

[thieman]

Summary

Implements the rare sampler from datadog-agent/pkg/trace/sampler/rare_sampler.go matching the Go V0 behavior (runSamplers). Ensures low-traffic trace shapes that would otherwise be dropped by probabilistic or priority sampling remain visible.

Changes

• rare_sampler.rs — new module with RareSampler, a per-(env,service)-shard SeenSpans with cardinality-bounded TTL tracking and modular-hash shrinking on overflow
• signature.rs — expose span_hash_for_rare helper (compute_span_hash with env="", with_resource=true)
• apm.rs — add RareSamplerConfig with tps, cooldown, cardinality fields; enable_rare_sampler lives at the ApmConfiguration wrapper level to support env var remapping (APM_ENABLE_RARE_SAMPLER); disabled by default
• mod.rs — wire rare sampler into run_samplers before all other samplers in both probabilistic and legacy (priority) paths
• saluki-common/src/rate.rs — new TokenBucket (tokens-per-second refill, burst capacity) extracted as a reusable primitive

Behavioral notes

• Rare sampler runs before all other samplers on every trace
• In the probabilistic path: rare short-circuits probabilistic/error sampling (else-if chain matching Go V0)
• In the legacy path: rare is checked after manual-drop but before priority/no-priority samplers
• When rare wins: _dd.rare = 1 set on the first rare span; no _dd.p.dm set (matches Go)
• Default: disabled (apm_config.enable_rare_sampler: false), 5 TPS, 300s cooldown, 200 cardinality per shard

Test plan

• rare_sampler.rs: 11 unit tests covering disabled mode, new signature kept, TTL drop, expiry re-sample, measured spans, non-top-level ignored, multi-top-level flag placement, rate limiting, cardinality shrink, span eligibility
• saluki-common/src/rate.rs: 4 unit tests for TokenBucket
• apm.rs: 5 config tests (disabled by default, enable via YAML, enable via env var, env var overrides YAML, tuning via YAML)
• mod.rs: 7 interaction tests (rare catches 0% probabilistic drop, _dd.rare=1 set, no resample within TTL, rare disabled drops, legacy path PriorityAutoDrop caught, 100%/0% probabilistic baseline)

Partial fix for #1134 (rare sampler only; error tracking standalone and priority/no-priority sampler gaps remain).

🤖 Generated with Claude Code

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Target: ffeb787 (baseline) vs c12651b (comparison) diff
Analysis Type: Stripped binaries (debug symbols excluded)
Baseline Size: 26.46 MiB
Comparison Size: 26.48 MiB
Size Change: +21.16 KiB (+0.08%)
Pass/Fail Threshold: +5%
Result: PASSED ✅

Changes by Module

Module	File Size	Symbols
core	+6.96 KiB	20
saluki_components::transforms::trace_sampler	+6.65 KiB	7
hashbrown	+4.46 KiB	3
[Unmapped]	+3.26 KiB	1
saluki_components::common::datadog	-1.34 KiB	5
serde_core	+796 B	10
saluki_core::data_model::event	+269 B	1
[sections]	-133 B	6
agent_data_plane::cli::run	+116 B	3
unicode_segmentation	+104 B	1
saluki_components::transforms::dogstatsd_prefix_filter	+48 B	1
saluki_core::topology::blueprint	+9 B	1
aho_corasick	+8 B	1
figment	-6 B	1
std	+4 B	9
serde_yaml	-4 B	1
agent_data_plane::components::tag_filterlist	+1 B	1
tonic	+0 B	2
crossbeam_channel	+0 B	6
tracing_subscriber	+0 B	4

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +22.0Ki  [NEW] +21.8Ki    saluki_components::transforms::trace_sampler::TraceSampler::process_trace::haae362d51dae69a0
  [NEW] +13.6Ki  [NEW] +13.4Ki    _<core::marker::PhantomData<T> as serde_core::de::DeserializeSeed>::deserialize::h1d9cfb93998519c7
  [NEW] +9.63Ki  [NEW] +9.51Ki    saluki_components::common::datadog::apm::ApmConfig::from_configuration::h8e84793e7f99a6e6
  [NEW] +5.20Ki  [NEW] +5.05Ki    _<core::marker::PhantomData<T> as serde_core::de::DeserializeSeed>::deserialize::had646bb5b345f1a8
  +0.3% +3.74Ki  +0.3% +3.03Ki    [108 Others]
   +46% +3.26Ki  [ = ]       0    [Unmapped]
  [NEW] +1.88Ki  [NEW] +1.78Ki    hashbrown::raw::RawTable<T,A>::reserve_rehash::hd0ce83933bc8fbac
  [NEW] +1.79Ki  [NEW] +1.65Ki    <tonic::transport::channel::endpoint::Endpoint as core::clone::Clone>::clone.12982
  [NEW] +1.79Ki  [NEW] +1.69Ki    hashbrown::raw::RawTable<T,A>::reserve_rehash::h30b016c18f118cb7
  [NEW] +1.10Ki  [NEW] +1.00Ki    crossbeam_channel::waker::SyncWaker::notify.13122
  [NEW] +1.04Ki  [NEW]    +913    <tracing_subscriber::filter::env::directive::Directive as core::clone::Clone>::clone.13350
  [NEW]    +878  [NEW]    +673    _<saluki_components::transforms::trace_sampler::TraceSampler as saluki_core::components::transforms::SynchronousTransform>::transform_buffer::h2e85609667304799
  [NEW]    +812  [NEW]    +713    hashbrown::map::HashMap<K,V,S,A>::remove::hbf94225925c826fd
  [DEL]    -772  [DEL]    -671    crossbeam_channel::waker::SyncWaker::unregister.13124
 -15.1%    -896 -15.1%    -896    [section .rodata]
  [DEL] -1.04Ki  [DEL]    -913    <tracing_subscriber::filter::env::directive::Directive as core::clone::Clone>::clone.13348
  [DEL] -1.10Ki  [DEL] -1.00Ki    crossbeam_channel::waker::SyncWaker::notify.13120
  [DEL] -1.79Ki  [DEL] -1.65Ki    <tonic::transport::channel::endpoint::Endpoint as core::clone::Clone>::clone.12980
  [DEL] -11.0Ki  [DEL] -10.9Ki    saluki_components::common::datadog::apm::ApmConfig::from_configuration::hf8c47abaacda9d53
  [DEL] -12.5Ki  [DEL] -12.4Ki    _<core::marker::PhantomData<T> as serde_core::de::DeserializeSeed>::deserialize::ha43058272b282e7a
  [DEL] -16.3Ki  [DEL] -16.1Ki    _<saluki_components::transforms::trace_sampler::TraceSampler as saluki_core::components::transforms::SynchronousTransform>::transform_buffer::h999b6ca59cc02d7a
  +0.1% +21.2Ki  +0.1% +16.7Ki    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

This comment was omitted because it was over 65,536 characters.Please check the Gitlab Job logs to see its output.

[thieman]

Confirmed these against https://github.com/DataDog/datadog-agent/blob/main/pkg/trace/config/config.go#L615-L617

[thieman]

Keys sourced from https://github.com/DataDog/datadog-agent/blob/3e0a1a6b1996a915b88a9b5cbc569726a172bd60/comp/trace/config/impl/setup.go#L313-L324

[thieman]

Env var sourced from https://github.com/DataDog/datadog-agent/blob/3e0a1a6b1996a915b88a9b5cbc569726a172bd60/pkg/config/setup/apm.go#L117

[thieman]

What is our expected behavior when RC eventually changes our config at runtime? Do we end up restarting the entire process to reset things, or do we need to respond to those config changes here somehow?

[thieman]

Spoke with @tobz on this, going to leave a TODO and an issue here for now. #1326

[thieman]

I figured it made sense to add trace sampler as an accepted PR title since there will be a few more of these coming, seems like it will let us merge with that failing since it checks against the action definition on main.

[andrewqian2001datadog]

For this code

I think we want to add a rare check?

In the agent, there is a rare check before the no priority "legacy" path.

Added a test to show that the we decide to drop even when rare matches

/// Demonstrates the current ADP/agent divergence for OTLP traces without a sampling priority.
    ///
    /// The Datadog Agent still honors the rare decision for this shape, but ADP currently routes
    /// OTLP no-priority traces through the upstream OTLP sampling branch and drops them when that
    /// rate check fails.
    #[test]
    fn rare_sampler_otlp_no_priority_differs_from_agent() {
        let mut sampler = create_sampler_with_rare_enabled();
        sampler.probabilistic_sampler_enabled = false;
        sampler.error_sampling_enabled = false;
        sampler.otlp_sampling_rate = 0.0;

        let mut meta = saluki_common::collections::FastHashMap::default();
        meta.insert(
            MetaString::from_static(OTEL_TRACE_ID_META_KEY),
            MetaString::from("00000000000000000000000000000001"),
        );
        let span = create_top_level_span(888, 1).with_meta(meta);
        let mut trace = create_test_trace(vec![span]);

        let (keep, priority, decision_maker, root_idx) = sampler.run_samplers(&mut trace);
        assert!(!keep, "ADP currently drops this OTLP trace even though rare matched");
        assert_eq!(priority, PRIORITY_AUTO_DROP);
        assert_eq!(decision_maker, "");
        assert_eq!(
            trace.spans()[root_idx.unwrap()]
                .metrics()
                .get(rare_sampler::RARE_KEY)
                .copied(),
            Some(1.0),
            "the rare sampler ran, but its keep decision was not honored"
        );
    }

[andrewqian2001datadog]

Non blocking but I think if we are planning to port over the trace agent to ADP then we would have to add the tracer logic to sampling.

[thieman]

[Claude Sonnet 4.6] Good point. Fixed in efd14f6 — is_top_level_or_measured now checks _dd.top_level in addition to _top_level, mirroring HasTopLevel in the Go agent's traceutil package. We don't have an equivalent of UpdateTracerTopLevel / ComputeTopLevel in ADP's ingest pipeline, so checking both keys directly in the sampler is the right approach for now. Also added tracer-top-level to the span_eligibility test case table.

[thieman]

[Claude Sonnet 4.6] Good catch — fixed in 7457f0b. Added the rare check at the top of the OTLP branch before the rate-based sampling, matching the agent's behavior at the linked lines. Also adapted the test from documenting the divergence to asserting the correct behavior (keep + _dd.rare = 1 on first occurrence).

[webern]

Assisted code review. Looks good.

It may have identified some testing gaps if they are of interest:

Go agent_test.go TestSampling — the interesting gaps are in the integration-level interactions. Three scenarios from Go aren't covered in the PR:

1. Manual drop short-circuits rare — Go's priority < 0 check fires before rare. A UserDrop(-1) trace should be dropped even if rare=true. The PR has no test for this. Relevant because the Rust code gates rare behind priority < PRIORITY_AUTO_DROP.
2. Rare wins over probabilistic when both would keep — Go's probabilistic-rare-100 tests that when probabilistic is 100% and rare is enabled, the trace is kept via the rare path (samplerName=rare, no _dd.p.dm tag). The PR's rare_sampler_catches_unsampled_trace only tests the case where probabilistic is 0%. A test at 100% would verify decision_maker stays empty when rare takes precedence over probabilistic.
3. Rare + error trace interaction — Go tests (error-sampled-prio-unsampled, error-sampled) show that rare catches error traces before the error sampler runs. The PR has no test where contains_error=true and rare is enabled — verifying the else-if chain means error sampler is never consulted when rare fires.

All three are run_samplers-level interaction tests (they'd go in mod.rs tests).

[webern]

(Claude Opus 4.6) Minor divergence from Go V0: In the Go legacy path (agent.go:1129-1131), if rare { return true, true } does not modify the trace chunk's priority — it preserves whatever the tracer originally set (could be AutoDrop=0, AutoKeep=1, or UserKeep=2). Here, we explicitly return PRIORITY_AUTO_KEEP, which means a UserKeep(2) trace caught by rare gets downgraded to AutoKeep(1).

The no-priority path (line 377) and OTLP path (line 359) already follow this same pattern, so this is consistent within the Rust codebase. Just flagging the Go delta — could matter if downstream consumers distinguish UserKeep from AutoKeep.

[webern]

(Claude Opus 4.6) Pre-existing issue (not introduced by this PR), but worth noting now that the rare sampler amplifies its impact: In Go V0 (agent.go:1070-1077), ErrorTrackingStandalone is checked before the rare sampler call, gating all samplers from running on non-error traces. In Rust, run_samplers has no ETS guard — the rare sampler runs unconditionally, and the ETS check at process_trace:454 only fires on the keep=false fallback path. This means with ETS enabled, the rare sampler (and other samplers) can keep non-error traces that Go would drop, violating the ETS contract.

[thieman]

[Claude Sonnet 4.6] Re: webern's first comment (priority downgrade) — fixed in 750a672. The rare branch in the priority path now returns priority instead of PRIORITY_AUTO_KEEP, matching Go's behavior of leaving the chunk priority untouched. Added a test covering the UserKeep(2) case specifically.

Re: the ETS concern — noted on #1314 for follow-up there.

[thieman]

[Claude Sonnet 4.6] Re: webern's test gap review — added all three in b9fb81e:

1. manual_drop_short_circuits_before_rare — UserDrop(-1) is dropped even when rare would match
2. rare_wins_over_probabilistic_no_decision_maker_tag — rare takes precedence at 100% probabilistic rate, _dd.p.dm not set
3. rare_catches_error_trace_before_error_sampler — rare keeps a no-priority error trace before the error sampler is consulted

[thieman]

/merge

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-04-08 18:44:07 UTC ℹ️ Start processing command /merge

---

2026-04-08 18:44:13 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

---

2026-04-08 18:52:09 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in main is approximately 15m (p90).

---

2026-04-08 19:04:22 UTC ℹ️ MergeQueue: This merge request was merged