Skip to content

[APMSVLS-469] fix(traces): filter sampled-out trace chunks when tracer or extension computes stats - #1358

Draft
lucaspimentel wants to merge 1 commit into
mainfrom
lpimentel/filter-sampled-traces-by-stats-owner
Draft

[APMSVLS-469] fix(traces): filter sampled-out trace chunks when tracer or extension computes stats#1358
lucaspimentel wants to merge 1 commit into
mainfrom
lpimentel/filter-sampled-traces-by-stats-owner

Conversation

@lucaspimentel

Copy link
Copy Markdown
Member

Overview

Follow-up to #1320. Filters sampled-out trace chunks from trace intake whenever trace stats are computed before the backend, by either the tracer or the extension. Previously the filtering gate only checked lambda_extension_compute_stats, so tracers that compute stats themselves and report Datadog-Client-Computed-Stats: true (for example, dd-trace-java v1.63.0 with DD_TRACE_STATS_COMPUTATION_ENABLED on by default) still had their rejected chunks forwarded to intake.

Stats owner Sampled-out chunks sent to trace intake
Backend (lambda_extension_compute_stats=false, client_computed_stats=false) Yes, the backend needs them to compute stats
Extension (lambda_extension_compute_stats=true) No, filtered
Tracer (client_computed_stats=true) No, filtered

Behavior notes:

  • Stats-ownership precedence is unchanged: client_computed_stats=true means the tracer owns stats even when extension-side stats are also configured.
  • Sampling priorities are never rewritten.
  • The pre-filter payload is still cloned before filtering, so extension-side stats include sampled-out traces.
  • The error sampler from [APMSVLS-469] feat(traces): add error sampler to keep/rescue traces with errors #1320 now also applies on the tracer-owned path, rescuing eligible errored AutoDrop chunks; explicit/user drops (negative priority) remain dropped, and _dd.errors_sr is stamped on rescued root spans.

Testing

  • Unit tests in trace_processor.rs: table-driven over all four ownership combinations, asserting filtering vs. no-filtering, the pre-filter stats payload contents, and error rescue on both non-backend paths. The tracer-owned case was confirmed failing before the fix.
  • Fake-intake E2E tests in apm_integration_test.rs: e2e_sampled_out_chunks_filtered_by_stats_owner (backend keeps all chunks; extension and tracer paths filter to the kept trace only) and e2e_error_sampler_rescues_only_errored_p0_traces extended to both non-backend owners.
  • cargo nextest run --workspace: 678 passed, 0 failed.
  • cargo fmt --all -- --check, cargo check --workspace, and both clippy passes (default and fips features) pass.

Found while investigating excess trace volume in the serverless-e2e-tests java17-sampling scenario: 20 traces observed at a 0.2 sample rate over 50 invocations where roughly 10 were expected.

…xtension computes stats

Trace intake now drops sampled-out chunks whenever stats are computed
before the backend, by either the tracer (Datadog-Client-Computed-Stats)
or the extension, instead of only when extension-side stats are enabled.
The backend still receives them when it owns stats computation, since it
needs them to compute stats. The error sampler now rescues eligible
errored drops on both non-backend paths.
@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Sep 4, 2026

Copy link
Copy Markdown

Pipelines

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b57d663 | Docs | View more details | Give us feedback!

@lucaspimentel lucaspimentel left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one high-severity trace-loss issue.


Review by Claude Code

config.ext.lambda_extension_compute_stats,
header_tags.generic.client_computed_stats,
);
if stats_computed_by != StatsComputedBy::Backend

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bottlecap/src/traces/trace_processor.rs:583: This newly runs drop_sampled_out_chunks for tracer-owned stats, but that helper drops a chunk solely from its sampling priority. Tracers can send a priority-0 partial trace containing spans retained by single-span sampling and marked with _dd.span_sampling.mechanism (Java does this); the standard Trace Agent preserves those spans and promotes the chunk to user-keep. Here the whole chunk is discarded, so configured single-span sampling silently loses the selected spans whenever Datadog-Client-Computed-Stats: true. Before removing a dropped chunk, retain its span-sampled spans and promote the resulting chunk as the Trace Agent does, with a tracer-owned regression test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant