test(llmobs): subset-match metadata in assert_llmobs_span_event - #7642
Merged
Conversation
Contributor
|
|
Yun-Kim
force-pushed
the
yun-kim/llmobs-metadata-subset-match
branch
from
September 2, 2026 21:06
9ad62c6 to
809d575
Compare
Yun-Kim
commented
Sep 2, 2026
Yun-Kim
force-pushed
the
yun-kim/llmobs-metadata-subset-match
branch
from
September 2, 2026 21:12
809d575 to
ae467e2
Compare
assert_llmobs_span_event compares the whole span event by equality, so metadata is matched exactly. Metadata is the one part of the payload that grows on its own: when a tracer starts surfacing a new provider field it lands on every span of that kind, and every test covering that integration fails at once — for a change that is additive by design. Assert metadata as a shallow subset and leave mock.ANY in its place on the expected side, so declared keys still have to match exactly while unlisted extras are tolerated. Tags already work this way in _assert_tags_span_event_tags, and dd-trace-py's own assert_llmobs_span_data treats metadata/tags/metrics as a shallow subset. Nothing else is relaxed: the rest of the event stays exact equality, and the metadata=None path is unchanged (all 33 call sites in the llm suites pass metadata=, so that branch is unused today). Unblocks DataDog/dd-trace-py#19752, which adds finish_reason/stop_reason to openai, litellm and anthropic LLM span metadata. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Yun-Kim
force-pushed
the
yun-kim/llmobs-metadata-subset-match
branch
from
September 2, 2026 21:14
ae467e2 to
b4124e2
Compare
Yun-Kim
marked this pull request as ready for review
September 2, 2026 21:16
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b4124e2d8a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
✅ All CI checks and tests passed. 🎉 All green!🧪 All tests passed 🔗 Commit SHA: dd51b1b | Docs | View more details | Give us feedback! |
sabrenner
reviewed
Sep 2, 2026
test_openai_llmobs.py:322 and :723 pass mock.ANY as the whole metadata argument, not a dict. The subset comparison called .items() on it and raised AttributeError before the span was validated, so both OpenAI error-case tests hard-errored in the INTEGRATION_FRAMEWORKS scenario. Only subset-compare when metadata is a dict. The wildcard keeps its previous meaning: metadata must be present, any value. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sabrenner
approved these changes
Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
assert_llmobs_span_eventcompares the whole span event by equality, sometadatamust match exactly. When a tracer starts emitting a new provider field it lands on every span of that kind, failing every test for that integration at once — for a purely additive change.That's blocking DataDog/dd-trace-py#19752 (adds
finish_reason/stop_reason): 4 jobs + thesystem-tests finishedgate are red only because the actual metadata gained one key. The alternative is editing ~23 metadata dicts, and repeating that for every tracer and future addition.Changes
Assert
metadataas a shallow subset — declared keys must still match exactly, unlisted extras are tolerated — and leavemock.ANYin its place so the whole-event equality still passes.Same semantics as
tagsin_assert_tags_span_event_tags, and as dd-trace-py's ownassert_llmobs_span_data.Nothing else is relaxed: the rest of the event stays exact equality, and the
metadata=Nonebranch is untouched (all 33 call sites passmetadata=, so it's unused today).Tradeoff: a test can no longer assert a declared key is the only metadata present. Notably
metadata={}(4 google_genai tests) now means "metadata present, any value" rather than "exactly{}". Happy to add an opt-inmetadata_exact=Trueif you'd rather keep that available.Testing
Exercised the real helper against: exact match, the
finish_reason/stop_reasonextras, wrong declared value, missing declared key, differing nested dict, empty metadata,mock.ANYas a declared value,ignore_values=["meta.metadata"], non-mutation of the caller's event, andmetadata=Nonestill rejecting stray metadata. Wrong values and missing keys still fail.mypy+ruffclean.Real signal is dd-trace-py#19752 going green against this branch; not repinned yet since this should land first.
🤖 Generated with Claude Code
Claude session:
acefbd3c-6218-49bb-88c2-c77f319c0b78Resume:
claude --resume acefbd3c-6218-49bb-88c2-c77f319c0b78