[APMSVLS-485] feat(traces): span-derived primary tags

[lucaspimentel]

Stacked PRs:

• feat(trace-sampler): add shared datadog-agent-trace-sampler crate serverless-components#141
• chore(deps): update libdatadog to 72fa8685 and serverless-components to 9daae40 #1332
• [APMSVLS-485] fix(traces): bound trace stats cardinality and report collapsed keys #1338
• [APMSVLS-485] feat(traces): span-derived primary tags #1336 👈🏽 This PR

Overview

Adds bottlecap-side config for span-derived primary tags (APMSVLS-485), matching the Serverless Compatibility Layer (datadog-trace-agent):

Variable	Default	Notes
DD_TRACE_EXPERIMENTAL_FEATURES_ENABLED	false	Gates the two settings below.
DD_TRACE_STATS_ADDITIONAL_TAGS	empty	Comma-separated span meta keys to use as additional stats aggregation dimensions.
DD_TRACE_STATS_ADDITIONAL_TAGS_CARDINALITY_LIMIT	100	Per-bucket cap on distinct tag-value combinations.

These feed StatsConcentratorService → SpanConcentrator::new's additional_metric_tag_keys, which libdd-trace-stats (via #1332's bump to 72fa8685) already implements end-to-end: extraction from span meta, inclusion in the aggregation key hash, cardinality limiting, and export as ClientGroupedStats.additional_metric_tags. So this PR is pure wiring, exactly as #1332 predicted.

Both gated settings reset to their defaults when the experimental gate is off, so a stale env var can't leak through.

No libdatadog change is needed. The ticket points at span_derived_primary_tags (field 22) being stubbed as vec![], but that field is deprecated in favor of additional_metric_tags (field 23) — see the comment in stats.proto. Field 22 intentionally stays empty; this PR uses the modern replacement.

Misconfiguration handling

libdatadog warns about bad input but still applies it, so both settings are validated here and the warnings restated in bottlecap's terms.

DD_TRACE_STATS_ADDITIONAL_TAGS_CARDINALITY_LIMIT:

• 0 would collapse every additional metric tag into the tracer_blocked_value sentinel — silent, total data loss for the dimension → falls back to the default of 100. Worth noting the Go trace agent reads 0 as no cap, so a user carrying that setting over would otherwise lose exactly what they meant to keep unbounded. "Unbounded" is deliberately not offered: chore(deps): update libdatadog to 72fa8685 and serverless-components to 9daae40 #1332 bounded this precisely to cap concentrator memory in a memory-capped Lambda.
• Values at or above the whole-key limit (7000) → clamped to 6999, mainly to silence libdatadog's misconfiguration warning. Per-field limits are applied before the whole-key limit, so such a value is effectively unbounded rather than strictly inert; either way, reaching it needs ~7k distinct tag combinations inside a single 10s bucket, which will not happen in a Lambda invocation.

DD_TRACE_STATS_ADDITIONAL_TAGS: libdatadog aggregates on at most 4 keys, sorting alphabetically and dropping the rest. So zone,tenant_id,region,shard,customer silently drops zone — by alphabetical accident, not by anything the user expressed. This is arguably the nastier failure of the two, since the config appears to work and only the dimensions are wrong. libdatadog's warning names the dropped keys but not the kept ones, the selection rule, or the env var, so bottlecap now logs all three. Truncation itself stays in libdatadog.

SCL parity divergence

serverless-components@9daae40 applies this env var raw (.parse::<usize>().ok(), no range check), so after this PR DD_TRACE_STATS_ADDITIONAL_TAGS_CARDINALITY_LIMIT=0 means "default 100" in Lambda and "collapse everything" in Azure Functions. Given the SCL-parity framing, resolve_cardinality_limits should be ported upstream to datadog-trace-agent as a follow-up.

On per-bucket cardinality resets

The cardinality limit is tracked per 10s StatsBucket and resets per bucket. Unlike the rate limiter in #1320, this is not broken by Lambda freeze/thaw, so it is deliberately left alone:

• A rate limiter's budget is a rate over wall-clock read at decision time, so a freeze inflates elapsed time without any work happening and the budget refills for free.
• This limit is a count of distinct keys inside a bucket that gets flushed and freed, and buckets are keyed by span timestamps (span.start + span.duration), not processing time. A freeze just routes the next invocation's spans into a different bucket, which is correct.

Bucketing stays load-bearing for the concentrator memory and /v0.6/stats payload bounds.

Testing

Config (src/config/mod.rs) — the experimental gate on/off for both settings, comma-separated parsing with surrounding whitespace, and an unparseable cardinality limit falling back to None.

Concentrator (src/traces/stats_concentrator_service.rs):

• test_additional_metric_tags_populated_when_configured — a configured key present in span meta surfaces as datacenter:us-east-1 on the exported ClientGroupedStats.
• test_additional_metric_tags_empty_by_default — unset config exports no additional tags even when the span carries a matching meta key.
• test_resolve_cardinality_limits — unset/0/in-range/at-or-above-whole-key validation, and that overriding additional_tags_limit leaves the other limits at their defaults.
• test_kept_and_dropped_additional_metric_tag_keys — the alphabetical keep/drop split reported for an over-cap key list, derived from the concentrator's own additional_metric_tag_keys() getter (so the kept set can't drift from libdatadog's real cap), and that duplicates collapse before the cap applies.
• test_observe_collapsed_fields — per-field collapse detection scans each field independently, the whole-key overflow entry (every field set to the sentinel) is skipped rather than misreported as all four collapsing, and the two sentinel encodings (bare key for peer_tags, trailing colon for additional_metric_tags) are both recognized.
• test_resource_collapse_observed_without_whole_key_overflow — exceeding resource_limit (1,024) while staying under whole_key_limit (7,000) is observable via the payload scan with no whole-key overflow entry, covering the observability trap where collapsed_spans alone stays at 0.
• test_collapse_warns_once_per_signal and test_collapse_warns_once_per_signal_with_additional_tags — whole-key overflow and per-field collapse are reported independently, each at most once per sandbox, and repeated flushes leave state untouched; the additional-tags variant covers the second possible-fields shape where ADDITIONAL_TAGS joins the saturation mask, including the fabricated payload carrying the sentinel only when additional tags are configured.

cargo test --lib, cargo clippy --lib --all-targets -- -D warnings, and cargo fmt --check all pass.

Fake intake (tests/apm_integration_test.rs) — payload-level coverage through concentrator → StatsFlusher → msgpack/gzip → in-process fake intake, with the config built through the real parsing path (get_config + figment::Jail, so the env vars and gate are exercised, not just struct fields):

• stats_additional_metric_tags_through_fake_intake — spans carrying meta["region"] arrive with additional_metric_tags populated: distinct region values split into separate groups with encoded region:<value> tags, repeated values aggregate into the same group (hit count 2), and an unconfigured meta key (tenant_id) is not exported.
• stats_additional_metric_tags_gated_off_through_fake_intake — DD_TRACE_STATS_ADDITIONAL_TAGS set without the gate: spans differing only in region merge into a single group with empty additional_metric_tags, proving the gate affects the emitted payload, not only the parsed config.
• stats_additional_metric_tags_cardinality_limit_through_fake_intake — DD_TRACE_STATS_ADDITIONAL_TAGS_CARDINALITY_LIMIT=1 with two distinct values in one bucket: the first value is admitted, the second collapses into the tracer_blocked_value: overflow group, and the total hit count is preserved.
• stats_additional_metric_tags_multiple_keys_through_fake_intake — two configured keys (region,tenant_id) group on both values together: one group per distinct combination, repeats of the same combination aggregating, and a change in either key producing a new group.

cargo nextest run --workspace (667 passed) and RUSTFLAGS="-D warnings" cargo clippy --workspace --all-targets --features default pass.

[datadog-datadog-prod-us1]

⚠️ Warnings

❌ Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 1 Pipeline job failed

DataDog/datadog-lambda-extension | publish layer e2e sandbox (amd64, fips) — 🔄 Retry may pass, looks flaky

View more details · View in GitLab

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 4fcc27b | Docs | View more details | Give us feedback!}

[lucaspimentel]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7ff07f23b8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Adds experimental span-derived dimensions to Lambda trace-stat aggregation.

Changes:

• Parses and gates additional tag configuration.
• Wires tag keys and cardinality limits into the stats concentrator.
• Adds validation, warnings, and tests.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
bottlecap/src/config/mod.rs	Adds gated configuration and tests.
bottlecap/src/traces/stats_concentrator_service.rs	Applies tags and cardinality limits.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a446908753

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[litianningdatadog]

Since this PR is the tip of PR stacks, besides unit tests it would be great to provide test manifest to prove the feature works as expected.

[lucaspimentel]

Since this PR is the tip of PR stacks, besides unit tests it would be great to provide test manifest to prove the feature works as expected.

Thanks, @litianningdatadog! I added additional tests to this PR using the fake intake to assert that the trace stats in the generated payload look as expected. I'm also adding e2e tests in serverless-e2e-test#330.