Skip to content

[ICPRO-41] Add SDK_CONFIGURATION RC parametric coverage - #7640

Open
mtoffl01 wants to merge 19 commits into
mainfrom
mtoffler/sdk-configuration-rc-tests
Open

[ICPRO-41] Add SDK_CONFIGURATION RC parametric coverage#7640
mtoffl01 wants to merge 19 commits into
mainfrom
mtoffler/sdk-configuration-rc-tests

Conversation

@mtoffl01

@mtoffl01 mtoffl01 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Motivation

Tracers are migrating away from reporting one APM_TRACING_* Remote Config capability bit per setting, toward a single SDK_CONFIGURATION bit that covers a generic, env-var-keyed sdk_config payload (a sibling of the legacy lib_config payload). As each tracer makes that switch, we want confidence that settings already covered by the legacy per-field bits keep behaving the same way once delivered via sdk_config instead — i.e. that the migration doesn't quietly change behavior.

RFC: https://docs.google.com/document/d/13aKzR8U97-yzaJZDs8rnNVGn4C1NCmfPhAoDkBtH9NM/edit?tab=t.0#heading=h.f9tdtrzge1bf

This PR adds parametric coverage for that: a tracer that declares SDK_CONFIGURATION should apply sdk_config the same way it used to apply the equivalent lib_config setting.

What changed?

TestDynamicConfigSdkConfiguration in tests/parametric/test_dynamic_configuration.py is a new test class covering the sdk_config delivery path. test_capability_sdk_configuration and test_sdk_config_tracing_enabled_matches_lib_config confirm DD_TRACE_ENABLED delivered via sdk_config disables tracing the same way the legacy lib_config/env-var path does. test_sdk_config_profiling_enabled covers DD_PROFILING_ENABLED, which has no lib_config equivalent to compare against (it only exists in sdk_config), so it just checks the tracer reports the setting as applied.

test_sdk_config_field_is_applied is a single parametrized test that loops over every lib_config setting with a corresponding APM_TRACING_* capability bit, and, for each one this tracer version actually declares (checked live against capabilities.yml), delivers it via sdk_config and asserts it was applied — using non-default test values so a tracer that silently no-ops on sdk_config can't pass by accident. Not every one of those settings has a test_telemetry.py name-mapping entry yet; where one's missing, the assertion fails with a message pointing at the gap, so whichever tracer next needs that setting covered will discover it there rather than the coverage being silently skipped.

Since no tracer implements SDK_CONFIGURATION today, the whole class is gated as missing_feature in every language's manifest.

Test plan

  • ./format.sh --check (mypy/ruff passed locally; yamlfmt step failed locally due to an unrelated macOS tooling issue, confirmed via direct yaml.safe_load validation of edited manifests)
  • CI parametric test run

🤖 Generated with Claude Code

mtoffl01 and others added 2 commits September 2, 2026 11:23
Validates that a tracer advertising the SDK_CONFIGURATION RC capability
consumes a migrated setting delivered via the generic sdk_config block
and produces behavior identical to the equivalent lib_config delivery,
starting with DD_TRACE_ENABLED. No tracer implements the capability yet,
so it's gated missing_feature across all languages until one ships.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Uses the tracer's reported/telemetry config value as a proxy for "the
setting was applied" since system-tests has no signal for real profiling
activity (no mock profiling intake, no RC-driven E2E profiling coverage).
Reuses the existing SDK_CONFIGURATION test class, which is already gated
missing_feature in every language manifest.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

manifests/cpp.yml                                                       @DataDog/system-tests-reviewers
manifests/dotnet.yml                                                    @DataDog/system-tests-reviewers
manifests/golang.yml                                                    @DataDog/system-tests-reviewers
manifests/java.yml                                                      @DataDog/system-tests-reviewers
manifests/nodejs.yml                                                    @DataDog/system-tests-reviewers
manifests/php.yml                                                       @DataDog/system-tests-reviewers
manifests/python.yml                                                    @DataDog/system-tests-reviewers
manifests/ruby.yml                                                      @DataDog/system-tests-reviewers
manifests/rust.yml                                                      @DataDog/system-tests-reviewers
tests/parametric/test_dynamic_configuration.py                          @DataDog/system-tests-core @DataDog/apm-sdk-capabilities
tests/parametric/test_telemetry.py                                      @DataDog/system-tests-core @DataDog/apm-sdk-capabilities
tests/parametric/utils.py                                               @DataDog/system-tests-core @DataDog/apm-sdk-capabilities

Drop the dd-go implementation-detail reference, stop framing this as a
PoC, describe sdk_config in terms of behavior compatibility rather than
being a lib_config mirror, and rename the profiling test since it has
no lib_config equivalent to compare against.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Sep 2, 2026

Copy link
Copy Markdown

Pipelines  Tests

⚠️ Warnings

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 2 Pipeline jobs failed

Testing the test | System Tests (nodejs, prod) / End-to-end #1 / express4-typescript 1

View more details · View in GitHub Actions

Error: read ECONNRESET during request; Container weblog is not running (exited) as per logs.

Testing the test | all-jobs-are-green

View more details · View in GitHub Actions

Job did not run: blocked by a failed job in an earlier pipeline stage.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 70b88c4 | Docs | View more details | Give us feedback!

test_library.config() is missing several fields and is permanently
empty for nodejs by design, so it can't serve as a cross-language echo
mechanism. Switch to the tracer's real telemetry configuration
(reused from test_telemetry.py's name mapping) for both the profiling
test and a new parametrized test covering the remaining migrated
lib_config fields (sample rate, logs injection, header tags, tags,
data streams, dynamic instrumentation) with non-default values.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@mtoffl01 mtoffl01 changed the title Add SDK_CONFIGURATION RC parametric tests (DD_TRACE_ENABLED, DD_PROFILING_ENABLED) Add SDK_CONFIGURATION RC parametric coverage Sep 2, 2026
mtoffl01 and others added 11 commits September 2, 2026 17:01
Taking the highest-seq_id entry passed only because RC happens to be
the highest-precedence source, not because we verified the value
actually came from RC. Filter on origin="remote_config" explicitly so
a tracer that silently ignores sdk_config can't pass by coincidence.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…e per-version support from capabilities.yml

Adds telemetry_name_mapping entries for tracing_sampling_rules, code_origin_enabled,
exception_replay_enabled, and live_debugging_enabled (previously missing, which had
silently excluded them from _SDK_CONFIG_FIELDS despite each having a real capability
bit). Collapses the per-field tests into a single test that iterates the full
backend-level field list and only exercises fields the current tracer version's
legacy APM_TRACING_* bits cover, read live via get_expected_capabilities_for_version
-- capabilities.yml is version-range-based and purely additive, so it keeps answering
"did this exact version support the legacy bit" even after a later version migrates
to SDK_CONFIGURATION.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…play/live_debugging

TestDynamicConfigSdkConfiguration is missing_feature-gated off for every language today,
so nothing exercises these fields yet. Adding their telemetry_name_mapping entries
speculatively risks a wrong key silently passing/failing rather than a missing one
loudly erroring -- better to add each field, with its mapping entry, when the language
that unblocks this class actually needs it covered.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
When _assert_telemetry_config_applied can't find the expected telemetry config
under any known key, hint that the tracer may report it under a name not yet
present in test_telemetry.py's telemetry_name_mapping -- this is the signal that
should surface it when a language starts reporting a currently-deferred
capability bit (see the comment above _SDK_CONFIG_FIELDS), rather than a
separate capability-tracking check.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…l of system-tests

The E2E tests/test_profiling.py suite does validate real profiler payloads via
interfaces.agent/interfaces.library; only parametric lacks a mock profiling intake
to capture such payloads from.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…plied docstring

_SDK_CONFIG_FIELDS is defined at module level above this class, not below this
docstring -- name it directly instead of a vague, wrong spatial reference.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Drop the paragraph explaining capabilities.yml's additive/version-range design --
that belongs with get_expected_capabilities_for_version, not repeated at every
call site.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…skip message

- _SDK_CONFIG_FIELDS is restored to the complete backend-level truth (all 10
  LibConfig settings with an APM_TRACING_* capability bit). telemetry_name_mapping
  stays incomplete for tracing_sampling_rules/code_origin_enabled/
  exception_replay_enabled/live_debugging_enabled -- a language that actually
  supports one of these will surface the gap via
  _assert_telemetry_config_applied's failure message rather than being silently
  excluded here.
- Drop comments that restated the above or pointed at stale/confusing references.
- Reword the "nothing applicable" skip message to be explicit that the test ran
  nothing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Explain in the class docstring how the single deep DD_TRACE_ENABLED
behavioral check combines with the per-field telemetry-only checks to
give regression confidence without a custom assertion per setting.
get_telemetry_config_by_origin's return type is a union of dict/str/int/bool/None,
so indexing the result with ["value"] doesn't type-check. Use return_value_only=True
to get the scalar directly instead.
@mtoffl01
mtoffl01 marked this pull request as ready for review September 3, 2026 20:03
@mtoffl01
mtoffl01 requested review from a team as code owners September 3, 2026 20:03
@mtoffl01
mtoffl01 requested review from genesor and iunanua and removed request for a team September 3, 2026 20:03

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f91c9e4f6f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread tests/parametric/test_dynamic_configuration.py Outdated
Comment thread tests/parametric/test_dynamic_configuration.py Outdated
Comment thread tests/parametric/test_dynamic_configuration.py Outdated
mtoffl01 and others added 3 commits September 3, 2026 16:41
…config test class

- Stop clearing telemetry/RC state before reading it in the sdk_config
  tests; clear only after each field's assertion so per-iteration
  isolation doesn't destroy data the assertion still needs.
- Replace the live capabilities.yml lookup in
  test_sdk_config_field_is_applied with a frozen snapshot, since
  capabilities.yml is expected to drift as tracers migrate settings to
  sdk_config and drop their old APM_TRACING_* bits.
- Rename TestDynamicConfigSdkConfiguration to
  Test_DynamicConfigSdkConfiguration per naming convention, and
  re-sort the affected manifest entries accordingly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
code_origin_enabled, live_debugging_enabled, and tracing_sampling_rules
had no mapping entry, so Test_DynamicConfigSdkConfiguration always
failed to find the tracer's telemetry config regardless of whether it
was actually applied correctly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@mtoffl01 mtoffl01 changed the title Add SDK_CONFIGURATION RC parametric coverage [ICPRO-41] Add SDK_CONFIGURATION RC parametric coverage Sep 4, 2026
Comment thread tests/parametric/test_dynamic_configuration.py Outdated
Avoids a cross-test-file import between test_dynamic_configuration.py
and test_telemetry.py, per PR review (CI optimization requirement).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants