[DOCS-15584] Add Multimodal Support page for Agent Observability - #39643
Open
ZStriker19 wants to merge 9 commits into
Open
[DOCS-15584] Add Multimodal Support page for Agent Observability#39643ZStriker19 wants to merge 9 commits into
ZStriker19 wants to merge 9 commits into
Conversation
Document how audio and images are attached to LLM spans, covering both the integrations that populate image_parts and audio_parts automatically and the manual annotation path for Python, Node.js, the HTTP API, and OpenTelemetry. Also cover the formats the trace view renders, span event and per-part size limits, and cross-link the new page from the SDK reference. Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
|
Thanks for the PR! I've created DOCS-15584 for an in-depth review from the docs team. |
…-multimodal-docs # Conflicts: # hugo/content/en/llm_observability/instrument/otel_instrumentation.md
Contributor
Preview links (active after the
|
Co-Authored-By: Claude <noreply@anthropic.com>
Incorporate review feedback on the Multimodal Support page: link the overview to the instrumentation sections, trim the requirements table, link the HTTP API reference and the OpenTelemetry GenAI message-part conventions, and tighten the limits and best-practices sections. Remove attachment_key from the AudioPart and ImagePart schemas in the API reference and from the SDK reference argument descriptions. The value is populated by Datadog during ingestion and is not something to set, so content is now the documented way to attach media. Co-Authored-By: Claude <noreply@anthropic.com>
ZStriker19
marked this pull request as ready for review
September 2, 2026 18:40
CFLJacquet
approved these changes
Sep 3, 2026
CFLJacquet
left a comment
CFLJacquet
left a comment
Contributor
There was a problem hiding this comment.
Good to go after the small changes
Add a Multimodal support section to the OpenTelemetry instrumentation page summarizing which conventions carry audio and images, plus a media subsection under each convention's message mapping: - OpenTelemetry 1.37+: blob parts for audio and images, uri parts for base64 image data URIs, text references for remote URLs and file IDs. - OpenInference: ordered image content, images only, base64 data URIs only. - Langfuse: image_url blocks, images only, base64 data URIs only. - OpenLLMetry: text content only, no media extracted. Replace the OpenTelemetry tab on the Multimodal Support page with a coverage table linking to those sections, and reword the overview to lead with what the feature lets you do. Co-Authored-By: Claude <noreply@anthropic.com>
ZStriker19
changed the title
Remove the per-convention media coverage for OpenInference, Langfuse, and OpenLLMetry. Their extraction behavior is being improved first, and the docs will follow in a separate PR rather than describing limitations that are about to change. Keeps the OpenTelemetry GenAI 1.37+ media mapping, which is unaffected. Co-Authored-By: Claude <noreply@anthropic.com>
brett0000FF
approved these changes
Sep 4, 2026
brett0000FF
left a comment
brett0000FF
left a comment
Collaborator
There was a problem hiding this comment.
Thanks, approved! I added a small review commit with some clarifications and editorial suggestions. Feel free to adjust those if you prefer your original presentation. 🚀
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do? What is the motivation?
Adds a new Multimodal Support page under
llm_observability/instrument/, documenting how audio and images are attached to LLM spans and rendered in the trace view.The
audio_parts/image_partsfields are already documented as schema entries in the SDK and HTTP API references, but there was no page explaining which integrations populate them automatically, what formats render, or what the size limits are. This page covers that and links back to the existing references rather than duplicating them.Content:
ddtrace/dd-traceversions for audio and image fields.dropped_io, the per-part cap in automatic instrumentation, and the fact that media is not scanned by Sensitive Data Scanner.In
sdk.md, the two rendered-trace screenshots move to the new page so they live in one place, and the surrounding sentences now link to it. A sentence in the Node.js tab that used "currently" and "planned for a future release" is replaced with a statement of current behavior.attachment_keyis removed from the reference docs. It was documented as a settable field onAudioPartandImagePartinapi.md, and in theinput_data/output_dataargument descriptions insdk.md, but there is no customer-facing way to produce one: the only attachments route is a read-onlyGET, and the ingestion pipeline mints the keys itself from inlinecontent.contentis now marked required and is the documented way to attach media.Reviewer notes
mainindd-trace-py(DOCS-5949 Saved Views Documentation #19148, Bump datadog-lambda-python layer to version 80 #19690) and ships in4.15.0, which the Python automatic-instrumentation table reflects. It is not in a tagged release yet, so those rows describe behavior that lands with that version.attachment_keyfrom the reference tables is intentional and was confirmed as safe: no one was setting it. The SDKs still accept the field, so nothing breaks for anyone who was.Merge readiness
For Datadog employees:
<name>/<description>convention and include the forward slash (/). If you've already created your PR with an incorrect branch name, please rename your branch and open a fresh PR./reviewto run an automated check that catches common issues before a Documentation team member reviews your PR.AI assistance
Drafted with Claude Code from an existing internal outline, with the SDK behavior, version numbers, and rendering/limit details verified against
dd-trace-py,dd-trace-js, and the ingestion and UI source before writing.Additional notes