Skip to content

feat(trace-utils): add v1-native agentless JSON encoder brick - #2370

Open
anais-raison wants to merge 9 commits into
mainfrom
anais/v1-agentless-encoder-brick
Open

feat(trace-utils): add v1-native agentless JSON encoder brick#2370
anais-raison wants to merge 9 commits into
mainfrom
anais/v1-agentless-encoder-brick

Conversation

@anais-raison

@anais-raison anais-raison commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds a v1-native equivalent of the existing v0.4 agentless JSON encoder, as groundwork for making v1::Span/v1::TraceChunk the exporter's canonical internal type:

  • agentless_encoder::encode_payload_from_v1 and its helpers (collect_attrs_v1, flatten_attr_into_v1, encode_span_link_v1, encode_span_event_v1) (libdd-trace-utils)

Nothing is wired into the live pipeline yet.

Motivation

Part of APMSP-2812: migration of the exporter from v0.4 to v1 with isolated bricks first before one final breaking PR that will handle the actual swap.

Additional Notes

Pure addition, no behavior change — these functions aren't called anywhere yet, so there's no regression risk.

Isolated brick for APMSP-2812: adds encode_payload_from_v1 and its
v1::Span-native helpers (collect_attrs_v1, flatten_attr_into_v1,
encode_span_link_v1, encode_span_event_v1) alongside the existing v0.4
agentless encoder. Not wired into any live send path yet.
@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

📚 Documentation Check Results

⚠️ 765 documentation warning(s) found

📦 libdd-trace-utils - 765 warning(s)


Updated: 2026-09-03 12:38:15 UTC | Commit: f738610 | missing-docs job results

@anais-raison anais-raison self-assigned this Aug 19, 2026
@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

🔒 Cargo Deny Results

⚠️ 3 issue(s) found, showing only errors (advisories, bans, sources)

📦 libdd-trace-utils - 3 error(s)

Show output
error[vulnerability]: Invalid pointer dereference in `fmt::Pointer` impl for `Atomic` and `Shared` when the underlying pointer is invalid
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:63:1
   │
63 │ crossbeam-epoch 0.9.18 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
   │
   ├ ID: RUSTSEC-2026-0204
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0204
   ├ Affected versions of `fmt::Display` dereference the underlying pointer. This causes a invalid pointer dereference e.g., when a pointer created with `Atomic::null` or `Shared::null`. `fmt::Debug` impls and pre-0.9 `fmt::Display` impls, which do not dereference pointers, are not affected by this issue.
   ├ Announcement: https://github.com/crossbeam-rs/crossbeam/pull/1276
   ├ Solution: Upgrade to >=0.9.20 (try `cargo update -p crossbeam-epoch`)
   ├ crossbeam-epoch v0.9.18
     └── crossbeam-deque v0.8.5
         └── rayon-core v1.12.1
             └── rayon v1.10.0
                 └── criterion v0.5.1
                     ├── libdd-common v5.2.0
                     │   ├── libdd-capabilities-impl v4.0.0
                     │   │   └── libdd-trace-utils v11.0.0
                     │   │       └── (dev) libdd-trace-utils v11.0.0 (*)
                     │   └── libdd-trace-utils v11.0.0 (*)
                     ├── (dev) libdd-trace-normalization v4.0.0
                     │   └── libdd-trace-utils v11.0.0 (*)
                     └── (dev) libdd-trace-utils v11.0.0 (*)

error[vulnerability]: h2 unbounded empty DATA frames
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:97:1
   │
97 │ h2 0.4.6 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
   │
   ├ ID: RUSTSEC-2026-0258
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0258
   ├ The h2 crate, used internally by hyper, had a flaw that would accept and queue empty DATA frames without limit.
     If streams were not actively drained, this could lead to unbounded memory usage, or a panic if the length overflows.
     
     Low severity.
     
     Patched in v0.4.16.
   ├ Announcement: https://github.com/hyperium/hyper/security/advisories/GHSA-q83h-524g-xf6h
   ├ Solution: Upgrade to >=0.4.16 (try `cargo update -p h2`)
   ├ h2 v0.4.6
     └── hyper v1.6.0
         ├── httpmock v0.8.0-alpha.1
         │   └── libdd-trace-utils v11.0.0
         │       └── (dev) libdd-trace-utils v11.0.0 (*)
         ├── hyper-rustls v0.27.7
         │   └── libdd-common v5.2.0
         │       ├── libdd-capabilities-impl v4.0.0
         │       │   └── libdd-trace-utils v11.0.0 (*)
         │       └── libdd-trace-utils v11.0.0 (*)
         ├── hyper-util v0.1.17
         │   ├── httpmock v0.8.0-alpha.1 (*)
         │   ├── hyper-rustls v0.27.7 (*)
         │   └── libdd-common v5.2.0 (*)
         ├── libdd-common v5.2.0 (*)
         └── libdd-trace-utils v11.0.0 (*)

error[unsound]: Rand is unsound with a custom logger using `rand::rng()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:184:1
    │
184 │ rand 0.8.5 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0097
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0097
    ├ It has been reported (by [@lopopolo](https://github.com/lopopolo)) that the `rand` library is [unsound](https://rust-lang.github.io/unsafe-code-guidelines/glossary.html#soundness-of-code--of-a-library) (i.e. that safe code using the public API can cause Undefined Behaviour) when all the following conditions are met:
      
      - The `log` and `thread_rng` features are enabled
      - A [custom logger](https://docs.rs/log/latest/log/#implementing-a-logger) is defined
      - The custom logger accesses `rand::rng()` (previously `rand::thread_rng()`) and calls any `TryRng` (previously `RngCore`) methods on `ThreadRng`
      - The `ThreadRng` (attempts to) reseed while called from the custom logger (this happens every 64 kB of generated data)
      - Trace-level logging is enabled or warn-level logging is enabled and the random source (the `getrandom` crate) is unable to provide a new seed
      
      `TryRng` (previously `RngCore`) methods for `ThreadRng` use `unsafe` code to cast `*mut BlockRng<ReseedingCore>` to `&mut BlockRng<ReseedingCore>`. When all the above conditions are met this results in an aliased mutable reference, violating the Stacked Borrows rules. Miri is able to detect this violation in sample code. Since construction of [aliased mutable references is Undefined Behaviour](https://doc.rust-lang.org/stable/nomicon/references.html), the behaviour of optimized builds is hard to predict.
    ├ Announcement: https://github.com/rust-random/rand/pull/1763
    ├ Solution: Upgrade to >=0.10.1 OR <0.10.0, >=0.9.3 OR <0.9.0, >=0.8.6 (try `cargo update -p rand`)
    ├ rand v0.8.5
      ├── (dev) libdd-common v5.2.0
      │   ├── libdd-capabilities-impl v4.0.0
      │   │   └── libdd-trace-utils v11.0.0
      │   │       └── (dev) libdd-trace-utils v11.0.0 (*)
      │   └── libdd-trace-utils v11.0.0 (*)
      ├── (dev) libdd-trace-normalization v4.0.0
      │   └── libdd-trace-utils v11.0.0 (*)
      ├── libdd-trace-utils v11.0.0 (*)
      └── proptest v1.5.0
          ├── (dev) libdd-common v5.2.0 (*)
          └── (dev) libdd-tinybytes v1.1.2
              ├── (dev) libdd-tinybytes v1.1.2 (*)
              └── libdd-trace-utils v11.0.0 (*)

advisories FAILED, bans ok, sources ok

Updated: 2026-09-03 12:39:47 UTC | Commit: f738610 | dependency-check job results

@datadog-datadog-us1-prod

datadog-datadog-us1-prod Bot commented Aug 19, 2026

Copy link
Copy Markdown

Tests

All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 96.71%
Overall Coverage: 77.26% (+0.59%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 7d3deb7 | Docs | View more details | Give us feedback!

@pr-commenter

pr-commenter Bot commented Aug 19, 2026

Copy link
Copy Markdown

Benchmarks

Comparison

Benchmark execution time: 2026-09-03 13:10:10

Comparing candidate commit 7d3deb7 in PR branch anais/v1-agentless-encoder-brick with baseline commit efbbf4c in branch main.

📊 Benchmarking dashboard

Found 1 performance improvements and 2 performance regressions! Performance is the same for 137 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:redis/obfuscate_redis_string

  • 🟥 execution_time [+1.372µs; +1.914µs] or [+4.462%; +6.227%]

scenario:tags/replace_trace_tags

  • 🟩 execution_time [-238.215ns; -229.018ns] or [-8.974%; -8.627%]

scenario:vec_map/as_deduped_map/needs_dedup_1_in_4/8

  • 🟥 execution_time [+16.438ns; +16.658ns] or [+4.181%; +4.237%]

Benchmark execution time: 2026-09-03 13:08:40

Comparing candidate commit 7d3deb7 in PR branch anais/v1-agentless-encoder-brick with baseline commit efbbf4c in branch main.

📊 Benchmarking dashboard

Found 1 performance improvements and 0 performance regressions! Performance is the same for 74 metrics, 10 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:datadog_sample_span/resource_pattern_rule_not_matching/wall_time

  • 🟩 execution_time [-8.067ns; -7.930ns] or [-4.779%; -4.698%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:datadog_sample_span/parent_not_sampled_short_circuit/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+514.861%; -514.861%]

scenario:datadog_sample_span/parent_sampled_short_circuit/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+514.861%; -514.861%]

scenario:glob_matcher/ascii_case_insensitive_match/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+514.861%; -514.861%]

scenario:glob_matcher/ascii_exact_match/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+515.472%; -515.130%]

scenario:glob_matcher/ascii_exact_miss/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+513.751%; -514.373%]

scenario:glob_matcher/ascii_wildcard_backtrack_match/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+514.331%; -514.628%]

scenario:glob_matcher/ascii_wildcard_heavy_backtrack/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+514.802%; -514.835%]

scenario:glob_matcher/ascii_wildcard_question_match/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+515.020%; -514.931%]

scenario:glob_matcher/ascii_wildcard_star_match/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+514.861%; -514.861%]

scenario:glob_matcher/star_short_circuit/allocated_bytes

  • unstable execution_time [-0.000ns; +0.000ns] or [+515.052%; -514.945%]

Candidate

Omitted due to size.

Baseline

Omitted due to size.

@dd-octo-sts

dd-octo-sts Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Artifact Size Benchmark Report

aarch64-alpine-linux-musl
Artifact Baseline Commit Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a 89.74 MB 89.74 MB +0% (+8 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so 8.26 MB 8.26 MB 0% (0 B) 👌
aarch64-unknown-linux-gnu
Artifact Baseline Commit Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a 100.97 MB 100.97 MB 0% (0 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so 11.13 MB 11.13 MB 0% (0 B) 👌
libdatadog-x64-windows
Artifact Baseline Commit Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll 26.68 MB 26.68 MB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib 94.96 KB 94.96 KB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb 181.16 MB 181.15 MB -0% (-16.00 KB) 👌
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib 770.98 MB 770.87 MB --.01% (-109.90 KB) 💪
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll 8.75 MB 8.75 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib 94.96 KB 94.96 KB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb 25.69 MB 25.70 MB +.03% (+8.00 KB) 🔍
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib 51.17 MB 51.17 MB 0% (0 B) 👌
libdatadog-x86-windows
Artifact Baseline Commit Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll 23.25 MB 23.25 MB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib 96.45 KB 96.45 KB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb 186.12 MB 186.10 MB -0% (-16.00 KB) 👌
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib 756.68 MB 756.60 MB --.01% (-84.50 KB) 💪
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll 6.76 MB 6.76 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib 96.45 KB 96.45 KB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb 27.64 MB 27.64 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib 48.71 MB 48.71 MB 0% (0 B) 👌
x86_64-alpine-linux-musl
Artifact Baseline Commit Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a 80.03 MB 80.03 MB 0% (0 B) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so 9.18 MB 9.18 MB 0% (0 B) 👌
x86_64-unknown-linux-gnu
Artifact Baseline Commit Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a 95.76 MB 95.76 MB 0% (0 B) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so 11.20 MB 11.20 MB 0% (0 B) 👌

Comment thread libdd-trace-utils/src/agentless_encoder/mod.rs Outdated
Comment thread libdd-trace-utils/src/agentless_encoder/mod.rs
Comment thread libdd-trace-utils/src/agentless_encoder/mod.rs Outdated
Comment thread libdd-trace-utils/src/agentless_encoder/mod.rs Outdated
Comment thread libdd-trace-utils/src/agentless_encoder/mod.rs Outdated
Comment thread libdd-trace-utils/src/agentless_encoder/mod.rs
@anais-raison
anais-raison marked this pull request as ready for review August 28, 2026 13:19
@anais-raison
anais-raison requested review from a team as code owners August 28, 2026 13:19

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1fbc4b65b4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread libdd-trace-utils/src/agentless_encoder/mod.rs Outdated
Comment thread libdd-trace-utils/src/agentless_encoder/mod.rs
Comment thread libdd-trace-utils/src/agentless_encoder/mod.rs Outdated
Comment thread libdd-trace-utils/src/agentless_encoder/mod.rs Outdated
Comment thread libdd-trace-utils/src/agentless_encoder/mod.rs Outdated
Comment thread libdd-trace-utils/src/agentless_encoder/mod.rs
chunk: &TraceChunk<T>,
metadata: &TracerMetadata,
) -> Result<S::Ok, S::Error> {
let mut map = ser.serialize_map(None)?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I think we can pre-compute the size here, avoiding useless allocations: (!metadata.env.is_empty() as usize) + (....) + 1

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yannham I looked into this but actually for serde_json, the len passed to serialize_map doesn't actually help. The len only matter in the special-case of len == Some(0) (where it closes the object immediately). For any other value it takes the exact same path as None and starts writing fields one by one directly to the output writer. There's no length-prefixed header, no upfront Vec::with_capacity sized from the lenght and Compound::Map doesn't store the length either. So Some(3) vs None produce byte-identical output, no allocation avoided (verified that with a small test comparing the two to be 100% sure).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I think I had msgpack in head, but you're right that it doesn't make a difference for JSON which is streaming-style anyway. It could though if in the future agentless move from JSON to a different encoding, so I would say if it's trivial to pre-compute (just re-order some declaration), it's a sanity step that doesn't cost much to do anyway. If it's too complicated or if it does mangle the code, then it's probably not worth it 👍

Comment thread libdd-trace-utils/src/agentless_encoder/mod.rs
Comment thread libdd-trace-utils/src/agentless_encoder/mod.rs
Comment thread libdd-trace-utils/src/agentless_encoder/mod.rs
)?;
map.end()
}
other => encode_event_scalar_v1(ser, other),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a matter of taste, but I wonder if we shouldn't just inline encode_event_scalar_v1 here. It's still relatively small and avoid an unreachable. Though one could argue that keep smaller function is better, so no strong opinion either way

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants