From b50f75ea79cd77fd5df9974936df7a154ad2558a Mon Sep 17 00:00:00 2001 From: Zachary Groves Date: Thu, 27 Aug 2026 13:57:02 -0400 Subject: [PATCH 1/8] Add searchable collection_errors values to OTel mapping warning reference --- .../instrument/otel_instrumentation.md | 51 ++++++++++--------- 1 file changed, 27 insertions(+), 24 deletions(-) diff --git a/hugo/content/en/llm_observability/instrument/otel_instrumentation.md b/hugo/content/en/llm_observability/instrument/otel_instrumentation.md index fedfbdc4976..e4e69d80fb6 100644 --- a/hugo/content/en/llm_observability/instrument/otel_instrumentation.md +++ b/hugo/content/en/llm_observability/instrument/otel_instrumentation.md @@ -865,30 +865,33 @@ Mapping warnings are detected at ingestion. They do not affect billing or span r A span can display warnings not listed in the following table if Datadog adds checks. These render with a generated title based on the check that triggered the warning. -| Warning | Attribute | Fix | -|---------|-----------|-----| -| Malformed model identifier | On `gen_ai.request.model` | Emit `gen_ai.response.model` directly, so the model name doesn't need to be parsed out of `gen_ai.request.model`. | -| Missing input and output | Expected `gen_ai.input.messages` | Set `gen_ai.input.messages` and `gen_ai.output.messages`. | -| Malformed input | On `gen_ai.input.messages` | Emit `gen_ai.input.messages` as a valid JSON array of messages. | -| Malformed output | On `gen_ai.output.messages` | Emit `gen_ai.output.messages` as a valid JSON array of messages. | -| Malformed message | On `gen_ai.input.messages` | Emit each message with a `role` and `content` field. | -| Empty message content | On `gen_ai.output.messages` | Emit a `content` string, or `parts` entries with a recognized `type`, for example `text`, `tool_call`, `tool_call_response`. | -| Missing embedding input | Expected `gen_ai.input.messages` | Set `gen_ai.input.messages` to the embedded text. | -| Malformed embedding input | On `gen_ai.input.messages` | Emit `gen_ai.input.messages` as a valid JSON array of messages. | -| Unreadable token counts | On `gen_ai.usage.input_tokens` | Emit token counts as integers, not strings or objects. | -| Invalid token counts | On `gen_ai.usage.input_tokens` | Emit non-negative integer token counts. | -| Unreadable cost metrics | On `gen_ai.cost.estimated_total` | Emit cost metrics as integers or floats, not strings or objects. | -| Invalid cost metrics | On `gen_ai.cost.estimated_total` | Emit non-negative cost values. | -| Malformed invocation parameters | On invocation parameters | Emit invocation parameters as a valid JSON object, or set them individually as `gen_ai.request.*` attributes (such as `temperature`, `top_p`, and `max_tokens`). | -| Malformed tool definitions | On `gen_ai.tool.definitions` | Emit `gen_ai.tool.definitions` as a valid JSON array of tool definitions. | -| Malformed tool definition | On `gen_ai.tool.definitions` | Give each tool definition a `name`, and make its `parameters` a JSON object. | -| Missing tool name | Expected `gen_ai.tool.name` | Set `gen_ai.tool.name` on tool spans. | -| Missing tool call name | On `gen_ai.output.messages` | Give each tool call a `name`. | -| Missing operation name | Expected `gen_ai.operation.name` | Set `gen_ai.operation.name` to one of `chat`, `text_completion`, `embeddings`, `execute_tool`, `invoke_agent`, or `retriever`. | -| Malformed span events | On `events` | Emit valid JSON in span events, or set `gen_ai.input.messages` and `gen_ai.output.messages` directly. | -| Unreadable document score | On `output.documents` | Emit each document score as a number. | -| Malformed document metadata | On `output.documents` | Emit each document's metadata as a JSON object. | -| Unrecognized instrumentation | Expected `gen_ai.operation.name` | Set `gen_ai.operation.name` and `gen_ai.system` so Datadog can identify the instrumentation. | +The **Search value** column lists the value stored on the span. Use it with the `@collection_errors` attribute to find flagged spans, for example `@collection_errors:otel_warning_missing_model_name`. The UI strips the `otel_warning_` prefix when it displays a warning, so the label you see does not match the value you search for. + +| Warning | Search value (`@collection_errors:`) | Attribute | Fix | +|---------|--------------------------------------|-----------|-----| +| Missing model name | `otel_warning_missing_model_name` | Expected `gen_ai.response.model` | Emit `gen_ai.response.model` directly, so the model name doesn't need to be parsed out of `gen_ai.request.model`. | +| Missing model provider | `otel_warning_missing_model_provider` | Expected `gen_ai.provider.name` | Emit `gen_ai.provider.name` directly, so the provider doesn't need to be inferred from `gen_ai.system`. | +| Malformed model identifier | `otel_warning_strands_model_malformed` | On `gen_ai.request.model` | Emit `gen_ai.response.model` directly, so the model name doesn't need to be parsed out of `gen_ai.request.model`. | +| Malformed input | `otel_warning_input_malformed` | On `gen_ai.input.messages` | Emit `gen_ai.input.messages` as a valid JSON array of messages. | +| Malformed output | `otel_warning_output_malformed` | On `gen_ai.output.messages` | Emit `gen_ai.output.messages` as a valid JSON array of messages. | +| Malformed message | `otel_warning_message_malformed` | On `gen_ai.input.messages` | Emit each message with a `role` and `content` field. | +| Empty message content | `otel_warning_invalid_parts` | On `gen_ai.output.messages` | Emit a `content` string, or `parts` entries with a recognized `type`, for example `text`, `tool_call`, `tool_call_response`. | +| Missing embedding input | `otel_warning_embedding_input_missing` | Expected `gen_ai.input.messages` | Set `gen_ai.input.messages` to the embedded text. | +| Malformed embedding input | `otel_warning_embedding_input_malformed` | On `gen_ai.input.messages` | Emit `gen_ai.input.messages` as a valid JSON array of messages. | +| Unreadable token counts | `otel_warning_token_usage_unparseable` | On `gen_ai.usage.input_tokens` | Emit token counts as integers, not strings or objects. | +| Invalid token counts | `otel_warning_token_usage_invalid` | On `gen_ai.usage.input_tokens` | Emit non-negative integer token counts. | +| Unreadable cost metrics | `otel_warning_cost_metrics_unparseable` | On `gen_ai.cost.estimated_total` | Emit cost metrics as integers or floats, not strings or objects. | +| Invalid cost metrics | `otel_warning_cost_metrics_invalid` | On `gen_ai.cost.estimated_total` | Emit non-negative cost values. | +| Malformed invocation parameters | `otel_warning_params_malformed` | On invocation parameters | Emit invocation parameters as a valid JSON object, or set them individually as `gen_ai.request.*` attributes (such as `temperature`, `top_p`, and `max_tokens`). | +| Malformed tool definitions | `otel_warning_tool_definitions_malformed` | On `gen_ai.tool.definitions` | Emit `gen_ai.tool.definitions` as a valid JSON array of tool definitions. | +| Malformed tool definition | `otel_warning_tool_definition_entry_malformed` | On `gen_ai.tool.definitions` | Give each tool definition a `name`, and make its `parameters` a JSON object. | +| Missing tool name | `otel_warning_tool_span_name_missing` | Expected `gen_ai.tool.name` | Set `gen_ai.tool.name` on tool spans. | +| Missing tool call name | `otel_warning_tool_call_name_missing` | On `gen_ai.output.messages` | Give each tool call a `name`. | +| Missing operation name | `otel_warning_operation_missing` | Expected `gen_ai.operation.name` | Set `gen_ai.operation.name` to one of `chat`, `text_completion`, `embeddings`, `execute_tool`, `invoke_agent`, or `retriever`. | +| Malformed span events | `otel_warning_failed_to_parse_span_events` | On `events` | Emit valid JSON in span events, or set `gen_ai.input.messages` and `gen_ai.output.messages` directly. | +| Unreadable document score | `otel_warning_failed_to_parse_document_score` | On `output.documents` | Emit each document score as a number. | +| Malformed document metadata | `otel_warning_document_metadata_malformed` | On `output.documents` | Emit each document's metadata as a JSON object. | +| Unrecognized instrumentation | `otel_warning_spec_version_unknown` | Expected `gen_ai.operation.name` | Set `gen_ai.operation.name` and `gen_ai.system` so Datadog can identify the instrumentation. | ### Find raw span attributes for a flagged span From a83bb459b25957891a2f76d12212e6547020b2ad Mon Sep 17 00:00:00 2001 From: Zachary Groves Date: Thu, 27 Aug 2026 14:02:53 -0400 Subject: [PATCH 2/8] Remove prefix-stripping sentence from mapping warning reference --- .../en/llm_observability/instrument/otel_instrumentation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hugo/content/en/llm_observability/instrument/otel_instrumentation.md b/hugo/content/en/llm_observability/instrument/otel_instrumentation.md index e4e69d80fb6..a42fe983231 100644 --- a/hugo/content/en/llm_observability/instrument/otel_instrumentation.md +++ b/hugo/content/en/llm_observability/instrument/otel_instrumentation.md @@ -865,7 +865,7 @@ Mapping warnings are detected at ingestion. They do not affect billing or span r A span can display warnings not listed in the following table if Datadog adds checks. These render with a generated title based on the check that triggered the warning. -The **Search value** column lists the value stored on the span. Use it with the `@collection_errors` attribute to find flagged spans, for example `@collection_errors:otel_warning_missing_model_name`. The UI strips the `otel_warning_` prefix when it displays a warning, so the label you see does not match the value you search for. +The **Search value** column lists the value stored on the span. Use it with the `@collection_errors` attribute to find flagged spans, for example `@collection_errors:otel_warning_missing_model_name`. | Warning | Search value (`@collection_errors:`) | Attribute | Fix | |---------|--------------------------------------|-----------|-----| From 764a60025be7b5926421a8c6677676843e434f55 Mon Sep 17 00:00:00 2001 From: Zachary Groves Date: Tue, 1 Sep 2026 17:26:56 -0400 Subject: [PATCH 3/8] Add Multimodal Support page for Agent Observability Document how audio and images are attached to LLM spans, covering both the integrations that populate image_parts and audio_parts automatically and the manual annotation path for Python, Node.js, the HTTP API, and OpenTelemetry. Also cover the formats the trace view renders, span event and per-part size limits, and cross-link the new page from the SDK reference. Co-Authored-By: Claude --- hugo/config/_default/menus/main.en.yaml | 5 + .../instrument/multimodal.md | 271 ++++++++++++++++++ .../en/llm_observability/instrument/sdk.md | 6 +- 3 files changed, 279 insertions(+), 3 deletions(-) create mode 100644 hugo/content/en/llm_observability/instrument/multimodal.md diff --git a/hugo/config/_default/menus/main.en.yaml b/hugo/config/_default/menus/main.en.yaml index 66af956243c..60f7135d933 100644 --- a/hugo/config/_default/menus/main.en.yaml +++ b/hugo/config/_default/menus/main.en.yaml @@ -5614,6 +5614,11 @@ menu: parent: llm_obs_instrumentation identifier: llm_obs_prompt_tracking weight: 311 + - name: Multimodal + url: llm_observability/instrument/multimodal + parent: llm_obs_instrumentation + identifier: llm_obs_multimodal + weight: 312 - name: Investigate url: llm_observability/investigate/ parent: llm_obs diff --git a/hugo/content/en/llm_observability/instrument/multimodal.md b/hugo/content/en/llm_observability/instrument/multimodal.md new file mode 100644 index 00000000000..7e0ed797645 --- /dev/null +++ b/hugo/content/en/llm_observability/instrument/multimodal.md @@ -0,0 +1,271 @@ +--- +title: Multimodal Support +description: Attach audio and images to LLM spans in Agent Observability and render them inline in the trace view, automatically or through manual annotation. +further_reading: + - link: '/llm_observability/instrument/sdk/' + tag: 'Documentation' + text: 'Agent Observability SDK Reference' + - link: '/llm_observability/instrument/auto_instrumentation/' + tag: 'Documentation' + text: 'Automatic Instrumentation for Agent Observability' + - link: '/llm_observability/instrument/api/' + tag: 'Documentation' + text: 'Agent Observability HTTP API Reference' +--- + +## Overview + +Multimodal support renders media attached to LLM spans directly in the trace view, alongside the rest of the span's data. Voice agents and agents that reason over images handle this content as part of normal operation, and multimodal support makes it visible in traces. + +Multimodal support enables you to: + +- Attach images to LLM span messages and see them rendered inline in the trace view +- Attach audio to LLM span messages and play it back from an inline audio player +- See per-turn transcripts alongside the audio for voice conversations +- Inspect the exact image a model was shown, and hear the exact audio it received and produced, without reconstructing it from logs or a separate storage system + +{{< img src="llm_observability/instrumentation/image_example.png" alt="An LLM span in the Agent Observability trace view. The input USER message shows the prompt 'What is in this image?' with an inline photo of a black puppy, and the output ASSISTANT message describes it as a black Labrador Retriever puppy on a wooden surface." style="width:100%;" >}} + +Media is attached per message using the typed fields `image_parts` and `audio_parts`. Automatic instrumentation populates these fields for supported integrations. For everything else, annotate them yourself. + +## Requirements + +| Capability | Python (`ddtrace`) | Node.js (`dd-trace`) | +| ---------- | ------------------ | -------------------- | +| Audio on messages | >= 4.12.0 | >= 6.7.0, or >= 5.118.0 on the v5 line | +| Images on messages | >= 4.13.0 | >= 6.10.0, or >= 5.121.0 on the v5 line | + +The [Agent Observability HTTP API][1] and [OpenTelemetry instrumentation][2] have no version requirement. + +## Automatic instrumentation + +Datadog's [LLM integrations][3] populate media parts for the following libraries, with no code change on your part. + +{{< tabs >}} +{{% tab "Python" %}} + +| Integration | Media captured | Tracer version | +| ----------- | -------------- | -------------- | +| OpenAI chat completions | Input `input_audio` content and output audio, including the model's audio transcript | >= 4.12.0 | +| OpenAI Realtime API | Per-turn input and output audio, transcripts, token usage, and tool calls | >= 4.13.0 | +| LiteLLM | Input and output audio, through the OpenAI chat completions format | >= 4.12.0 | +| OpenAI chat completions and Responses API | Inline base64-encoded images | >= 4.15.0 | +| OpenAI Agents | Inline base64-encoded images | >= 4.15.0 | +| Anthropic | Inline base64-encoded images | >= 4.15.0 | + +Each turn of an OpenAI Realtime API conversation produces its own LLM span. Raw `pcm16` and G.711 audio is converted to WAV so that it plays in the browser. To turn off Realtime API instrumentation, set `DD_OPENAI_REALTIME_ENABLED=false`. + +Audio in streamed chat completions (`stream=True`) is not captured. + +{{% /tab %}} +{{% tab "Node.js" %}} + +| Integration | Media captured | Tracer version | +| ----------- | -------------- | -------------- | +| OpenAI chat completions | Input `input_audio` content and output audio, including the model's audio transcript | >= 6.7.0, or >= 5.118.0 | +| OpenAI Agents | Input and output audio, through the OpenAI chat completions format | >= 6.7.0, or >= 5.118.0 | + +Audio in streamed chat completions is not captured. The Node.js SDK does not capture images automatically, and does not instrument the OpenAI Realtime API. To attach images, or to trace a Realtime API conversation, use [manual instrumentation](#manual-instrumentation). + +{{% /tab %}} +{{< /tabs >}} + +Images referenced by a remote URL or a file ID are not fetched. Only images the application sends inline as base64 data are captured. Where an integration finds media it cannot capture, it records a text marker such as `[image]` or `[audio]` in the message content so that the media is still accounted for. + +## Manual instrumentation + +Attach media to an LLM span by adding `image_parts` or `audio_parts` to a message in `input_data` or `output_data`. + +Each media part takes the following fields: + +| Field | Type | Required | Description | +| ----- | ---- | -------- | ----------- | +| `mime_type` | string | Yes | The media type of the attachment, such as `image/png` or `audio/wav`. | +| `content` | string | Yes | The base64-encoded media. | + +Datadog moves inline media out of the span payload into attachment storage during ingestion, and the trace view fetches it on demand. The `attachment_key` field described in the [HTTP API reference][4] holds the resulting reference and is populated by Datadog. Set `content`. + +{{< tabs >}} +{{% tab "Python" %}} + +```python +import base64 +from ddtrace.llmobs import LLMObs +from ddtrace.llmobs.decorators import llm + +@llm(model_name="gpt-4o", model_provider="openai") +def describe_image(image_bytes): + resp = ... # multimodal (vision) LLM call + LLMObs.annotate( + input_data=[ + { + "role": "user", + "content": "What is in this image?", + "image_parts": [ + {"mime_type": "image/png", "content": base64.b64encode(image_bytes).decode("utf-8")} + ], + } + ], + output_data=[{"role": "assistant", "content": "The image shows a golden retriever puppy."}], + ) + return resp +``` + +```python +import base64 +from ddtrace.llmobs import LLMObs +from ddtrace.llmobs.decorators import llm + +@llm(model_name="gpt-realtime", model_provider="openai") +def voice_turn(user_audio_bytes): + resp = ... # multimodal (audio) LLM call + LLMObs.annotate( + input_data=[ + { + "role": "user", + "content": "Hey, how are you?", # transcript of the input audio + "audio_parts": [ + {"mime_type": "audio/wav", "content": base64.b64encode(user_audio_bytes).decode("utf-8")} + ], + } + ], + output_data=[ + { + "role": "assistant", + "content": "Hey! I'm doing great, thanks for asking. How about you?", + "audio_parts": [ + {"mime_type": "audio/wav", "content": base64.b64encode(resp.audio_bytes).decode("utf-8")} + ], + } + ], + ) + return resp +``` + +A media part that is missing `mime_type`, or that sets neither `content` nor `attachment_key`, raises a `TypeError`. + +{{% /tab %}} +{{% tab "Node.js" %}} + +Node.js uses the camelCase field names `imageParts` and `audioParts`, with `mimeType` on each part. + +```javascript +function describeImage (imageBytes) { + const resp = ... // multimodal (vision) LLM call + llmobs.annotate({ + inputData: [ + { + role: 'user', + content: 'What is in this image?', + imageParts: [{ mimeType: 'image/png', content: imageBytes.toString('base64') }] + } + ], + outputData: [{ role: 'assistant', content: 'The image shows a golden retriever puppy.' }] + }) + return resp +} +describeImage = llmobs.wrap({ kind: 'llm', modelName: 'gpt-4o', modelProvider: 'openai' }, describeImage) +``` + +```javascript +function voiceTurn (userAudioBytes) { + const resp = ... // multimodal (audio) LLM call + llmobs.annotate({ + inputData: [ + { + role: 'user', + content: 'Hey, how are you?', // transcript of the input audio + audioParts: [{ mimeType: 'audio/wav', content: userAudioBytes.toString('base64') }] + } + ], + outputData: [ + { + role: 'assistant', + content: "Hey! I'm doing great, thanks for asking. How about you?", + audioParts: [{ mimeType: 'audio/wav', content: resp.audioBuffer.toString('base64') }] + } + ] + }) + return resp +} +voiceTurn = llmobs.wrap({ kind: 'llm', modelName: 'gpt-audio', modelProvider: 'openai' }, voiceTurn) +``` + +A media part that is missing `mimeType`, or that sets neither `content` nor `attachmentKey`, is dropped. The rest of the message is still recorded. + +{{% /tab %}} +{{% tab "API" %}} + +Submit `audio_parts` and `image_parts` on the message objects you send to the [Spans API endpoint][1]. The field names match the Python SDK. + +```json +{ + "role": "user", + "content": "What is in this image?", + "image_parts": [ + { + "mime_type": "image/png", + "content": "iVBORw0KGgoAAAANSUhEUgAA..." + } + ] +} +``` + +See [Message][2], [AudioPart][3], and [ImagePart][4] in the HTTP API reference. + +[1]: /llm_observability/instrument/api/#spans-api +[2]: /llm_observability/instrument/api/#message +[3]: /llm_observability/instrument/api/#audiopart +[4]: /llm_observability/instrument/api/#imagepart +{{% /tab %}} +{{% tab "OpenTelemetry" %}} + +If you use [OpenTelemetry instrumentation][1], no code change is required. Datadog extracts media from message parts that follow the OpenTelemetry GenAI semantic conventions: + +- A `blob` part with a `mime_type` and inline bytes becomes an image or audio part. When the part omits `modality`, Datadog infers it from the MIME type. +- A `uri` part carrying a base64 image data URI, such as `data:image/png;base64,...`, becomes an image part. + +Audio reaches the trace view through `blob` parts only. An audio data URI on a `uri` part is recorded as text. + +[1]: /llm_observability/instrument/otel_instrumentation/ +{{% /tab %}} +{{< /tabs >}} + +## View media in the trace view + +Open a trace in [Trace Explorer][5] and select an LLM span. Media attached to the span's messages renders inside the message it belongs to. + +Images appear as thumbnails. Select a thumbnail to open the image at full size. Up to 12 images render per message. + +Audio appears as a player, with the message text shown alongside it. For a voice conversation, that text is the turn transcript. Select the player to load and play the audio. + +{{< img src="llm_observability/instrumentation/audio_example.png" alt="An LLM span in the Agent Observability trace view. The input message from the USER shows an inline audio player with the transcript 'Hey, how are you?', and the output ASSISTANT message shows a 'Click to play audio' control with the transcript 'Hey! I'm doing great, thanks for asking. How about you?'." style="width:100%;" >}} + +### Supported formats + +Images render when the MIME type is one of `image/png`, `image/jpeg`, `image/webp`, or `image/gif`, and the declared MIME type matches the encoded bytes. SVG images do not render. The trace view makes no outbound requests, so an image referenced by a remote URL is shown as text rather than fetched. + +Audio plays through the browser's native audio player, so use a container the browser can decode, such as `audio/wav`, `audio/mpeg`, `audio/ogg`, or `audio/webm`. Raw formats such as `pcm16`, `g711_ulaw`, and `g711_alaw` do not play. Convert raw audio to WAV before attaching it. The Python OpenAI Realtime API integration does this conversion for you. + +## Limits and behavior + +- **Span event size.** A span event is capped at 5 MB. When a span exceeds the cap, its input and output are replaced with a placeholder and `dropped_io` is added to the span's `collection_errors` attribute. The span itself is kept. In Python, configure the cap with `DD_LLMOBS_EVENT_SIZE_BYTES`. +- **Per-part size.** Automatic instrumentation in Python caps a single inline media part at 4 MiB. Above the cap, the integration records a text marker such as `[audio]` or `[image omitted: too large]` and preserves the surrounding message text and model response. +- **Sensitive Data Scanner.** Media content is not scanned by [Sensitive Data Scanner][6]. Message text, tool arguments, and tool results are scanned as usual. + +## Best practices + +- **Use the typed media fields.** Attach media with `image_parts` and `audio_parts` rather than embedding base64 strings in a message's `content`. Only typed fields render in the trace view, and content that reaches `content` as raw base64 is scanned by Sensitive Data Scanner. +- **Keep individual media parts small.** Several media parts that each fit under the per-part cap can still add up to more than the 5 MB span event limit, which drops the span's entire input and output. Downsample or trim media before attaching it. +- **Attach the transcript with the audio.** Set the message `content` to the turn transcript so that the span is readable, searchable, and usable by evaluations even before the audio is played. + +## Further Reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: /llm_observability/instrument/api/ +[2]: /llm_observability/instrument/otel_instrumentation/ +[3]: /llm_observability/instrument/auto_instrumentation/ +[4]: /llm_observability/instrument/api/#audiopart +[5]: https://app.datadoghq.com/llm/traces +[6]: /security/sensitive_data_scanner/ diff --git a/hugo/content/en/llm_observability/instrument/sdk.md b/hugo/content/en/llm_observability/instrument/sdk.md index 12a8fdfcd66..3c06f1a655e 100644 --- a/hugo/content/en/llm_observability/instrument/sdk.md +++ b/hugo/content/en/llm_observability/instrument/sdk.md @@ -1435,7 +1435,7 @@ def describe_image(image_bytes): {{< /code-block >}} -Messages annotated with `audio_parts` or `image_parts` render as inline audio players and images in the trace view: +Messages annotated with `audio_parts` or `image_parts` render as inline audio players and images in the trace view. For supported formats, size limits, and the integrations that populate these fields automatically, see [Multimodal Support](/llm_observability/instrument/multimodal/). {{< img src="llm_observability/instrumentation/audio_example.png" alt="An LLM span in the Agent Observability trace view. The input message from the USER shows an inline audio player with the transcript 'Hey, how are you?', and the output ASSISTANT message shows a 'Click to play audio' control with the transcript 'Hey! I'm doing great, thanks for asking. How about you?'." style="width:100%;" >}} @@ -1576,13 +1576,13 @@ describeImage = llmobs.wrap({ kind: 'llm', modelName: 'gpt-4o', modelProvider: ' {{< /code-block >}} -Messages annotated with `audioParts` or `imageParts` render as inline audio players and images in the trace view: +Messages annotated with `audioParts` or `imageParts` render as inline audio players and images in the trace view. For supported formats, size limits, and the integrations that populate these fields automatically, see [Multimodal Support](/llm_observability/instrument/multimodal/). {{< img src="llm_observability/instrumentation/audio_example.png" alt="An LLM span in the Agent Observability trace view. The input message from the USER shows an inline audio player with the transcript 'Hey, how are you?', and the output ASSISTANT message shows a 'Click to play audio' control with the transcript 'Hey! I'm doing great, thanks for asking. How about you?'." style="width:100%;" >}} {{< img src="llm_observability/instrumentation/image_example.png" alt="An LLM span in the Agent Observability trace view. The input USER message shows the prompt 'What is in this image?' with an inline photo of a black puppy, and the output ASSISTANT message describes it as a black Labrador Retriever puppy on a wooden surface." style="width:100%;" >}} -For OpenAI audio chat completions, `audioParts` are also captured automatically by [Datadog's LLM integrations](/llm_observability/instrument/auto_instrumentation/)—no manual annotation required. Unlike `audioParts`, `imageParts` are not currently captured automatically and must be annotated manually; automatic capture is planned for a future release. +For OpenAI audio chat completions, `audioParts` are also captured automatically by [Datadog's LLM integrations](/llm_observability/instrument/auto_instrumentation/), with no manual annotation required. The Node.js SDK does not capture `imageParts` automatically. Annotate them as shown above. {{% /tab %}} {{% tab "Java" %}} From cb44c9ccfe31e9152e9e38305bc4b4228ccd912c Mon Sep 17 00:00:00 2001 From: Zachary Groves Date: Wed, 2 Sep 2026 13:04:13 -0400 Subject: [PATCH 4/8] Soften overview wording and link to the instrumentation sections Co-Authored-By: Claude --- hugo/content/en/llm_observability/instrument/multimodal.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hugo/content/en/llm_observability/instrument/multimodal.md b/hugo/content/en/llm_observability/instrument/multimodal.md index 7e0ed797645..4bd052d95f1 100644 --- a/hugo/content/en/llm_observability/instrument/multimodal.md +++ b/hugo/content/en/llm_observability/instrument/multimodal.md @@ -26,7 +26,7 @@ Multimodal support enables you to: {{< img src="llm_observability/instrumentation/image_example.png" alt="An LLM span in the Agent Observability trace view. The input USER message shows the prompt 'What is in this image?' with an inline photo of a black puppy, and the output ASSISTANT message describes it as a black Labrador Retriever puppy on a wooden surface." style="width:100%;" >}} -Media is attached per message using the typed fields `image_parts` and `audio_parts`. Automatic instrumentation populates these fields for supported integrations. For everything else, annotate them yourself. +Media is attached per message using the typed fields `image_parts` and `audio_parts`. [Automatic instrumentation](#automatic-instrumentation) populates these fields for supported integrations, so media from those libraries appears in traces without any code change. For anything else, [manual instrumentation](#manual-instrumentation) covers how to attach media to a span. ## Requirements From 39694cbcea90c0a7b74a2d5f3fe43ef824e57362 Mon Sep 17 00:00:00 2001 From: Zachary Groves Date: Wed, 2 Sep 2026 14:37:53 -0400 Subject: [PATCH 5/8] Revamp multimodal docs and drop attachment_key from the reference Incorporate review feedback on the Multimodal Support page: link the overview to the instrumentation sections, trim the requirements table, link the HTTP API reference and the OpenTelemetry GenAI message-part conventions, and tighten the limits and best-practices sections. Remove attachment_key from the AudioPart and ImagePart schemas in the API reference and from the SDK reference argument descriptions. The value is populated by Datadog during ingestion and is not something to set, so content is now the documented way to attach media. Co-Authored-By: Claude --- .../en/llm_observability/instrument/api.md | 13 +++--- .../instrument/multimodal.md | 44 +++++++++---------- .../en/llm_observability/instrument/sdk.md | 8 ++-- 3 files changed, 31 insertions(+), 34 deletions(-) diff --git a/hugo/content/en/llm_observability/instrument/api.md b/hugo/content/en/llm_observability/instrument/api.md index d8275086e97..363417a5cf1 100644 --- a/hugo/content/en/llm_observability/instrument/api.md +++ b/hugo/content/en/llm_observability/instrument/api.md @@ -206,23 +206,21 @@ If the request is successful, the API responds with a 202 network code and an em #### AudioPart -An audio segment on a message. Provide either `content` or `attachment_key`. +An audio segment on a message. For supported formats and size limits, see [Multimodal Support][6]. | Field | Type | Description | |-------|------|-------------| -| mime_type [*required*] | string | The media type of the audio, such as `audio/wav` or `audio/pcm`. | -| content | string | The base64-encoded audio, carried inline with the message. | -| attachment_key | string | A reference to audio stored outside the span payload, instead of inline `content`. | +| mime_type [*required*] | string | The media type of the audio, such as `audio/wav` or `audio/mpeg`. | +| content [*required*] | string | The base64-encoded audio, carried inline with the message. | #### ImagePart -An image on a message. Provide either `content` or `attachment_key`. +An image on a message. For supported formats and size limits, see [Multimodal Support][6]. | Field | Type | Description | |-------|------|-------------| | mime_type [*required*] | string | The media type of the image, such as `image/png` or `image/jpeg`. | -| content | string | The base64-encoded image, carried inline with the message. | -| attachment_key | string | A reference to an image stored outside the span payload, instead of inline `content`. | +| content [*required*] | string | The base64-encoded image, carried inline with the message. | #### ToolDefinition @@ -704,3 +702,4 @@ For feedback events, provide exactly one of `span_id`, `trace_id`, `session_id`, [3]: /getting_started/tagging/ [4]: /llm_observability/investigate/evaluations/end_user_feedback [5]: /llm_observability/instrument/sdk/?tab=python#enriching-spans +[6]: /llm_observability/instrument/multimodal/ diff --git a/hugo/content/en/llm_observability/instrument/multimodal.md b/hugo/content/en/llm_observability/instrument/multimodal.md index 4bd052d95f1..7806727b3e8 100644 --- a/hugo/content/en/llm_observability/instrument/multimodal.md +++ b/hugo/content/en/llm_observability/instrument/multimodal.md @@ -1,6 +1,6 @@ --- title: Multimodal Support -description: Attach audio and images to LLM spans in Agent Observability and render them inline in the trace view, automatically or through manual annotation. +description: Attach audio and images to LLM spans in Agent Observability and render them inline in the trace view, automatically or through manual instrumentation. further_reading: - link: '/llm_observability/instrument/sdk/' tag: 'Documentation' @@ -30,10 +30,10 @@ Media is attached per message using the typed fields `image_parts` and `audio_pa ## Requirements -| Capability | Python (`ddtrace`) | Node.js (`dd-trace`) | -| ---------- | ------------------ | -------------------- | -| Audio on messages | >= 4.12.0 | >= 6.7.0, or >= 5.118.0 on the v5 line | -| Images on messages | >= 4.13.0 | >= 6.10.0, or >= 5.121.0 on the v5 line | +| Capability | Python | Node.js | +| ---------- | ------ | ------- | +| Audio on messages | >= 4.12.0 | >= 6.7.0, or >= 5.118.0 | +| Images on messages | >= 4.13.0 | >= 6.10.0, or >= 5.121.0 | The [Agent Observability HTTP API][1] and [OpenTelemetry instrumentation][2] have no version requirement. @@ -65,7 +65,7 @@ Audio in streamed chat completions (`stream=True`) is not captured. | OpenAI chat completions | Input `input_audio` content and output audio, including the model's audio transcript | >= 6.7.0, or >= 5.118.0 | | OpenAI Agents | Input and output audio, through the OpenAI chat completions format | >= 6.7.0, or >= 5.118.0 | -Audio in streamed chat completions is not captured. The Node.js SDK does not capture images automatically, and does not instrument the OpenAI Realtime API. To attach images, or to trace a Realtime API conversation, use [manual instrumentation](#manual-instrumentation). +Audio in streamed chat completions is not captured. {{% /tab %}} {{< /tabs >}} @@ -83,7 +83,7 @@ Each media part takes the following fields: | `mime_type` | string | Yes | The media type of the attachment, such as `image/png` or `audio/wav`. | | `content` | string | Yes | The base64-encoded media. | -Datadog moves inline media out of the span payload into attachment storage during ingestion, and the trace view fetches it on demand. The `attachment_key` field described in the [HTTP API reference][4] holds the resulting reference and is populated by Datadog. Set `content`. +Datadog moves inline media out of the span payload during ingestion and the trace view fetches it on demand, so a large attachment does not weigh down the span itself. {{< tabs >}} {{% tab "Python" %}} @@ -142,7 +142,7 @@ def voice_turn(user_audio_bytes): return resp ``` -A media part that is missing `mime_type`, or that sets neither `content` nor `attachment_key`, raises a `TypeError`. +A media part that is missing `mime_type` or `content` raises a `TypeError`. {{% /tab %}} {{% tab "Node.js" %}} @@ -191,7 +191,7 @@ function voiceTurn (userAudioBytes) { voiceTurn = llmobs.wrap({ kind: 'llm', modelName: 'gpt-audio', modelProvider: 'openai' }, voiceTurn) ``` -A media part that is missing `mimeType`, or that sets neither `content` nor `attachmentKey`, is dropped. The rest of the message is still recorded. +A media part that is missing `mimeType` or `content` is dropped. The rest of the message is still recorded. {{% /tab %}} {{% tab "API" %}} @@ -211,29 +211,29 @@ Submit `audio_parts` and `image_parts` on the message objects you send to the [S } ``` -See [Message][2], [AudioPart][3], and [ImagePart][4] in the HTTP API reference. +For the full message schema, see [Message][2] in the [Agent Observability HTTP API reference][3]. [1]: /llm_observability/instrument/api/#spans-api [2]: /llm_observability/instrument/api/#message -[3]: /llm_observability/instrument/api/#audiopart -[4]: /llm_observability/instrument/api/#imagepart +[3]: /llm_observability/instrument/api/ {{% /tab %}} {{% tab "OpenTelemetry" %}} -If you use [OpenTelemetry instrumentation][1], no code change is required. Datadog extracts media from message parts that follow the OpenTelemetry GenAI semantic conventions: +If you use [OpenTelemetry instrumentation][1], no code change is required. Datadog extracts media from message parts that follow the [OpenTelemetry GenAI semantic conventions for message parts][2]: - A `blob` part with a `mime_type` and inline bytes becomes an image or audio part. When the part omits `modality`, Datadog infers it from the MIME type. - A `uri` part carrying a base64 image data URI, such as `data:image/png;base64,...`, becomes an image part. -Audio reaches the trace view through `blob` parts only. An audio data URI on a `uri` part is recorded as text. +Audio reaches the trace view through `blob` parts only. An audio data URI on a `uri` part is recorded as text. The conventions also specify `blob` as the part type for base64 data, so prefer `blob` for both audio and images. [1]: /llm_observability/instrument/otel_instrumentation/ +[2]: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/model/gen-ai/gen-ai-input-messages.json {{% /tab %}} {{< /tabs >}} ## View media in the trace view -Open a trace in [Trace Explorer][5] and select an LLM span. Media attached to the span's messages renders inside the message it belongs to. +Open a trace in [Trace Explorer][4] and select an LLM span. Media attached to the span's messages renders inside the message it belongs to. Images appear as thumbnails. Select a thumbnail to open the image at full size. Up to 12 images render per message. @@ -245,18 +245,17 @@ Audio appears as a player, with the message text shown alongside it. For a voice Images render when the MIME type is one of `image/png`, `image/jpeg`, `image/webp`, or `image/gif`, and the declared MIME type matches the encoded bytes. SVG images do not render. The trace view makes no outbound requests, so an image referenced by a remote URL is shown as text rather than fetched. -Audio plays through the browser's native audio player, so use a container the browser can decode, such as `audio/wav`, `audio/mpeg`, `audio/ogg`, or `audio/webm`. Raw formats such as `pcm16`, `g711_ulaw`, and `g711_alaw` do not play. Convert raw audio to WAV before attaching it. The Python OpenAI Realtime API integration does this conversion for you. +Audio plays through the browser's native audio player. Supported containers include `audio/wav`, `audio/mpeg`, `audio/ogg`, and `audio/webm`. Raw formats such as `pcm16`, `g711_ulaw`, and `g711_alaw` do not play. Convert raw audio to WAV before attaching it. The Python OpenAI Realtime API integration does this conversion for you. ## Limits and behavior -- **Span event size.** A span event is capped at 5 MB. When a span exceeds the cap, its input and output are replaced with a placeholder and `dropped_io` is added to the span's `collection_errors` attribute. The span itself is kept. In Python, configure the cap with `DD_LLMOBS_EVENT_SIZE_BYTES`. +- **Span event size.** A span event is capped at 5 MB. When a span exceeds the cap, its input and output are replaced with a placeholder and `dropped_io` is added to the span's `collection_errors` attribute. The span itself is kept. - **Per-part size.** Automatic instrumentation in Python caps a single inline media part at 4 MiB. Above the cap, the integration records a text marker such as `[audio]` or `[image omitted: too large]` and preserves the surrounding message text and model response. -- **Sensitive Data Scanner.** Media content is not scanned by [Sensitive Data Scanner][6]. Message text, tool arguments, and tool results are scanned as usual. +- **Sensitive Data Scanner.** Media content is not scanned by [Sensitive Data Scanner][5]. Message text, tool arguments, and tool results are scanned as usual. ## Best practices -- **Use the typed media fields.** Attach media with `image_parts` and `audio_parts` rather than embedding base64 strings in a message's `content`. Only typed fields render in the trace view, and content that reaches `content` as raw base64 is scanned by Sensitive Data Scanner. -- **Keep individual media parts small.** Several media parts that each fit under the per-part cap can still add up to more than the 5 MB span event limit, which drops the span's entire input and output. Downsample or trim media before attaching it. +- **Use the typed media fields.** Attach media with `image_parts` and `audio_parts` rather than embedding base64 strings in a message's `content`. Only typed fields render in the trace view. - **Attach the transcript with the audio.** Set the message `content` to the turn transcript so that the span is readable, searchable, and usable by evaluations even before the audio is played. ## Further Reading @@ -266,6 +265,5 @@ Audio plays through the browser's native audio player, so use a container the br [1]: /llm_observability/instrument/api/ [2]: /llm_observability/instrument/otel_instrumentation/ [3]: /llm_observability/instrument/auto_instrumentation/ -[4]: /llm_observability/instrument/api/#audiopart -[5]: https://app.datadoghq.com/llm/traces -[6]: /security/sensitive_data_scanner/ +[4]: https://app.datadoghq.com/llm/traces +[5]: /security/sensitive_data_scanner/ diff --git a/hugo/content/en/llm_observability/instrument/sdk.md b/hugo/content/en/llm_observability/instrument/sdk.md index 666ae7f8120..1d635d24686 100644 --- a/hugo/content/en/llm_observability/instrument/sdk.md +++ b/hugo/content/en/llm_observability/instrument/sdk.md @@ -1307,11 +1307,11 @@ The `LLMObs.annotate()` method accepts the following arguments: `input_data` : optional - _JSON serializable type or list of dictionaries_ -
Either a JSON serializable type (for non-LLM spans) or a list of dictionaries with this format: `{"content": "...", "role": "...", "tool_calls": ..., "tool_results": ..., "audio_parts": ..., "image_parts": ...}`, where `"tool_calls"` are an optional list of tool call dictionaries with required keys: `"name"`, `"arguments"`, and optional keys: `"tool_id"`, `"type"`, and `"tool_results"` are an optional list of tool result dictionaries with required key: `"result"`, and optional keys: `"name"`, `"tool_id"`, `"type"` for function calling scenarios. `"audio_parts"` and `"image_parts"` are optional lists of media dictionaries for multimodal spans, each with a required `"mime_type"` and exactly one of `"content"` (base64-encoded media, carried inline) or `"attachment_key"`. **Note**: Embedding spans are a special case and require a string or a dictionary (or a list of dictionaries) with this format: `{"text": "..."}`. +
Either a JSON serializable type (for non-LLM spans) or a list of dictionaries with this format: `{"content": "...", "role": "...", "tool_calls": ..., "tool_results": ..., "audio_parts": ..., "image_parts": ...}`, where `"tool_calls"` are an optional list of tool call dictionaries with required keys: `"name"`, `"arguments"`, and optional keys: `"tool_id"`, `"type"`, and `"tool_results"` are an optional list of tool result dictionaries with required key: `"result"`, and optional keys: `"name"`, `"tool_id"`, `"type"` for function calling scenarios. `"audio_parts"` and `"image_parts"` are optional lists of media dictionaries for multimodal spans, each with a required `"mime_type"` and `"content"` (base64-encoded media, carried inline). **Note**: Embedding spans are a special case and require a string or a dictionary (or a list of dictionaries) with this format: `{"text": "..."}`. `output_data` : optional - _JSON serializable type or list of dictionaries_ -
Either a JSON serializable type (for non-LLM spans) or a list of dictionaries with this format: `{"content": "...", "role": "...", "tool_calls": ..., "audio_parts": ..., "image_parts": ...}`, where `"tool_calls"` are an optional list of tool call dictionaries with required keys: `"name"`, `"arguments"`, and optional keys: `"tool_id"`, `"type"` for function calling scenarios. `"audio_parts"` and `"image_parts"` are optional lists of media dictionaries for multimodal spans, each with a required `"mime_type"` and exactly one of `"content"` (base64-encoded media, carried inline) or `"attachment_key"`. **Note**: Retrieval spans are a special case and require a string or a dictionary (or a list of dictionaries) with this format: `{"text": "...", "name": "...", "score": float, "id": "..."}`. +
Either a JSON serializable type (for non-LLM spans) or a list of dictionaries with this format: `{"content": "...", "role": "...", "tool_calls": ..., "audio_parts": ..., "image_parts": ...}`, where `"tool_calls"` are an optional list of tool call dictionaries with required keys: `"name"`, `"arguments"`, and optional keys: `"tool_id"`, `"type"` for function calling scenarios. `"audio_parts"` and `"image_parts"` are optional lists of media dictionaries for multimodal spans, each with a required `"mime_type"` and `"content"` (base64-encoded media, carried inline). **Note**: Retrieval spans are a special case and require a string or a dictionary (or a list of dictionaries) with this format: `{"text": "...", "name": "...", "score": float, "id": "..."}`. `tool_definitions` : optional - _list of dictionaries_ @@ -1457,11 +1457,11 @@ The `annotationOptions` object can contain the following: `inputData` : optional - _JSON serializable type or list of objects_ -
Either a JSON serializable type (for non-LLM spans) or a list of dictionaries with this format: `{role: "...", content: "...", audioParts: [...], imageParts: [...]}` (for LLM spans). `audioParts` and `imageParts` are optional lists of media objects for multimodal spans, each with a required `mimeType` and exactly one of `content` (base64-encoded media, carried inline) or `attachmentKey`. **Note**: Embedding spans are a special case and require a string or an object (or a list of objects) with this format: `{text: "..."}`. +
Either a JSON serializable type (for non-LLM spans) or a list of dictionaries with this format: `{role: "...", content: "...", audioParts: [...], imageParts: [...]}` (for LLM spans). `audioParts` and `imageParts` are optional lists of media objects for multimodal spans, each with a required `mimeType` and `content` (base64-encoded media, carried inline). **Note**: Embedding spans are a special case and require a string or an object (or a list of objects) with this format: `{text: "..."}`. `outputData` : optional - _JSON serializable type or list of objects_ -
Either a JSON serializable type (for non-LLM spans) or a list of objects with this format: `{role: "...", content: "...", audioParts: [...], imageParts: [...]}` (for LLM spans). `audioParts` and `imageParts` are optional lists of media objects for multimodal spans, each with a required `mimeType` and exactly one of `content` (base64-encoded media, carried inline) or `attachmentKey`. **Note**: Retrieval spans are a special case and require a string or an object (or a list of objects) with this format: `{text: "...", name: "...", score: number, id: "..."}`. +
Either a JSON serializable type (for non-LLM spans) or a list of objects with this format: `{role: "...", content: "...", audioParts: [...], imageParts: [...]}` (for LLM spans). `audioParts` and `imageParts` are optional lists of media objects for multimodal spans, each with a required `mimeType` and `content` (base64-encoded media, carried inline). **Note**: Retrieval spans are a special case and require a string or an object (or a list of objects) with this format: `{text: "...", name: "...", score: number, id: "..."}`. `metadata` : optional - _object_ From fd72f178b593b326767af2288ebcc132781ef1bb Mon Sep 17 00:00:00 2001 From: Zachary Groves Date: Thu, 3 Sep 2026 11:09:40 -0400 Subject: [PATCH 6/8] Document multimodal coverage per OpenTelemetry semantic convention Add a Multimodal support section to the OpenTelemetry instrumentation page summarizing which conventions carry audio and images, plus a media subsection under each convention's message mapping: - OpenTelemetry 1.37+: blob parts for audio and images, uri parts for base64 image data URIs, text references for remote URLs and file IDs. - OpenInference: ordered image content, images only, base64 data URIs only. - Langfuse: image_url blocks, images only, base64 data URIs only. - OpenLLMetry: text content only, no media extracted. Replace the OpenTelemetry tab on the Multimodal Support page with a coverage table linking to those sections, and reword the overview to lead with what the feature lets you do. Co-Authored-By: Claude --- .../instrument/multimodal.md | 20 ++++--- .../instrument/otel_instrumentation.md | 52 +++++++++++++++++++ 2 files changed, 66 insertions(+), 6 deletions(-) diff --git a/hugo/content/en/llm_observability/instrument/multimodal.md b/hugo/content/en/llm_observability/instrument/multimodal.md index 7806727b3e8..190fd03a5ef 100644 --- a/hugo/content/en/llm_observability/instrument/multimodal.md +++ b/hugo/content/en/llm_observability/instrument/multimodal.md @@ -15,7 +15,7 @@ further_reading: ## Overview -Multimodal support renders media attached to LLM spans directly in the trace view, alongside the rest of the span's data. Voice agents and agents that reason over images handle this content as part of normal operation, and multimodal support makes it visible in traces. +You can attach media (images and audio) to LLM spans and render it directly in the trace view, alongside the rest of the span's data. This lets you troubleshoot **voice agents** and **agents that reason over images** with their full context. Multimodal support enables you to: @@ -219,15 +219,23 @@ For the full message schema, see [Message][2] in the [Agent Observability HTTP A {{% /tab %}} {{% tab "OpenTelemetry" %}} -If you use [OpenTelemetry instrumentation][1], no code change is required. Datadog extracts media from message parts that follow the [OpenTelemetry GenAI semantic conventions for message parts][2]: +If you use [OpenTelemetry instrumentation][1], no code change is required. Datadog extracts media from messages at ingestion, and how much it can extract depends on the semantic convention your instrumentation emits: -- A `blob` part with a `mime_type` and inline bytes becomes an image or audio part. When the part omits `modality`, Datadog infers it from the MIME type. -- A `uri` part carrying a base64 image data URI, such as `data:image/png;base64,...`, becomes an image part. +| Convention | Images | Audio | +|------------|--------|-------| +| [OpenTelemetry 1.37+][2] | Inline `blob` parts, and `uri` parts carrying a base64 data URI | Inline `blob` parts | +| [OpenInference][3] | Ordered image content carrying a base64 data URI | Not extracted | +| [Langfuse][4] | `image_url` blocks carrying a base64 data URI | Not extracted | +| [OpenLLMetry][5] | Not extracted | Not extracted | -Audio reaches the trace view through `blob` parts only. An audio data URI on a `uri` part is recorded as text. The conventions also specify `blob` as the part type for base64 data, so prefer `blob` for both audio and images. +Only media carried inline as base64 bytes is rendered. A remote URL is recorded as a text reference and is never fetched. The [OpenTelemetry GenAI conventions][6] specify `blob` as the part type for inline base64 data, so prefer `blob` for both audio and images. [1]: /llm_observability/instrument/otel_instrumentation/ -[2]: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/model/gen-ai/gen-ai-input-messages.json +[2]: /llm_observability/instrument/otel_instrumentation/#media-in-messages +[3]: /llm_observability/instrument/otel_instrumentation/#media-in-openinference-messages +[4]: /llm_observability/instrument/otel_instrumentation/#media-in-langfuse-messages +[5]: /llm_observability/instrument/otel_instrumentation/#media-in-openllmetry-messages +[6]: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/model/gen-ai/gen-ai-input-messages.json {{% /tab %}} {{< /tabs >}} diff --git a/hugo/content/en/llm_observability/instrument/otel_instrumentation.md b/hugo/content/en/llm_observability/instrument/otel_instrumentation.md index 68f3a4a123d..3bb07cf5f11 100644 --- a/hugo/content/en/llm_observability/instrument/otel_instrumentation.md +++ b/hugo/content/en/llm_observability/instrument/otel_instrumentation.md @@ -29,6 +29,19 @@ For information on using Prompt Tracking with OpenTelemetry spans, see [Prompt T You can use OpenTelemetry spans inside [Agent Observability Experiments](/llm_observability/improve/experiments/setup#using-opentelemetry-spans-inside-experiments). By setting `DD_TRACE_OTEL_ENABLED=1`, OTel spans created inside an experiment task automatically appear as children of the experiment span. +### Multimodal support + +Audio and images on OpenTelemetry messages are rendered in the trace view. Coverage depends on the semantic convention your instrumentation emits: + +| Convention | Images | Audio | +|------------|--------|-------| +| [OpenTelemetry 1.37+](#media-in-messages) | Inline `blob` parts, and `uri` parts carrying a base64 data URI | Inline `blob` parts | +| [OpenInference](#media-in-openinference-messages) | Ordered image content carrying a base64 data URI | Not extracted | +| [Langfuse](#media-in-langfuse-messages) | `image_url` blocks carrying a base64 data URI | Not extracted | +| [OpenLLMetry](#media-in-openllmetry-messages) | Not extracted | Not extracted | + +Only media carried inline as base64 bytes is rendered. A remote URL is recorded as a text reference and is never fetched. For the fields, formats, and size limits that apply once media reaches a span, see [Multimodal Support](/llm_observability/instrument/multimodal/). + ### Span links Use [OpenTelemetry span links][9] on your GenAI spans to express non-parent-child relationships, such as when one span's output feeds another span's input. When two linked spans are in the same trace, the link appears as an edge in that trace's **Execution Graph**, so you can see how data flows between sibling spans (for example, a tool's output feeding a downstream LLM call). @@ -521,6 +534,23 @@ Input and output messages are extracted from the following sources, in priority | `gen_ai.output.messages` | `meta.output.messages` (llm) / `meta.output.value` (others) | | | `gen_ai.system_instructions` | Prepended to input | Added as system role messages | +##### Media in messages + +Message parts that carry media are extracted into the typed `audio_parts` and `image_parts` fields on the message: + +| Part type | Behavior | +|-----------|----------| +| `blob` with `mime_type` and inline bytes | Extracted to `image_parts` or `audio_parts`. When the part omits `modality`, it is inferred from the MIME type. | +| `uri` carrying a base64 image data URI, such as `data:image/png;base64,...` | Extracted to `image_parts`. | +| `uri` carrying a remote URL | Recorded as the text reference `[: ]`. The URL is not fetched. | +| `file` with a `file_id` | Recorded as the text reference `[ file: ]`. | + +A positional marker such as `[image blob: image/png]` is also added to the message text so that media keeps its place among the other parts. + +Audio reaches `audio_parts` through `blob` parts only. An audio data URI on a `uri` part is recorded as text, and the conventions specify `blob` as the part type for inline base64 data, so prefer `blob` for both audio and images. + +For the formats the trace view renders and the size limits that apply, see [Multimodal Support](/llm_observability/instrument/multimodal/). + ##### Embedding spans | OTel Source | Agent Observability Field | @@ -617,6 +647,10 @@ Messages are converted to OTel-compatible format and processed normally: | `gen_ai.prompt.*` | `meta.input.messages` (llm) / `meta.input.value` (others) | | `gen_ai.completion.*` | `meta.output.messages` (llm) / `meta.output.value` (others) | +##### Media in OpenLLMetry messages + +OpenLLMetry prompt and completion attributes carry text content only, so no audio or images are extracted from them. To render media on a span, emit it as [OpenTelemetry 1.37+ message parts](#media-in-messages) alongside the OpenLLMetry attributes. OTel standard attributes take priority, so both can be present on the same span. + #### Tool calls Tool calls are nested within completion attributes. @@ -745,6 +779,15 @@ The following indexed message attributes are supported: Image content maps to an image URI while preserving its position among other message content. +##### Media in OpenInference messages + +Ordered image content (`message.contents..message_content.image.image.url`) maps to an image URI. Because OpenInference carries images as a URL, what happens next depends on the URL: + +- A base64 data URI, such as `data:image/png;base64,...`, is extracted to `image_parts` and rendered in the trace view. +- A remote URL is recorded as the text reference `[image: ]`. The URL is not fetched. + +Audio is not extracted. The OpenInference specification defines `message_content.type: "audio"` along with the span-level attributes `audio.url`, `audio.mime_type`, and `audio.transcript`, but these are not mapped to Agent Observability fields. To render audio on a span, emit it as an [OpenTelemetry 1.37+ `blob` part](#media-in-messages) alongside the OpenInference attributes. OTel standard attributes take priority, so both can be present on the same span. + #### Embedding spans | OpenInference Source | Agent Observability Field | @@ -820,6 +863,15 @@ Each message is converted to the parts-based message shape: - A `tool_calls` array on a message becomes `tool_call` parts. - A message with `role: tool` and a `tool_call_id` becomes a `tool_result` part. +##### Media in Langfuse messages + +An `image_url` content block maps to an image URI. Because Langfuse carries images as a URL, what happens next depends on the URL: + +- A base64 data URI, such as `data:image/png;base64,...`, is extracted to `image_parts` and rendered in the trace view. +- A remote URL is recorded as the text reference `[image: ]`. The URL is not fetched. + +Audio is not extracted from Langfuse content blocks. To render audio on a span, emit it as an [OpenTelemetry 1.37+ `blob` part](#media-in-messages) alongside the Langfuse attributes. OTel standard attributes take priority, so both can be present on the same span. + ##### Tool spans For `tool`, `agent`, and `workflow` spans, `langfuse.observation.input`/`langfuse.observation.output` are used directly as `input.value`/`output.value`, after the standard `gen_ai.tool.call.*` fallback. From d17e8ba30b15c3ab252680215b14704b583127fc Mon Sep 17 00:00:00 2001 From: Zachary Groves Date: Thu, 3 Sep 2026 12:25:54 -0400 Subject: [PATCH 7/8] Scope OpenTelemetry media docs to the GenAI conventions Remove the per-convention media coverage for OpenInference, Langfuse, and OpenLLMetry. Their extraction behavior is being improved first, and the docs will follow in a separate PR rather than describing limitations that are about to change. Keeps the OpenTelemetry GenAI 1.37+ media mapping, which is unaffected. Co-Authored-By: Claude --- .../instrument/multimodal.md | 21 +++++-------- .../instrument/otel_instrumentation.md | 31 +------------------ 2 files changed, 9 insertions(+), 43 deletions(-) diff --git a/hugo/content/en/llm_observability/instrument/multimodal.md b/hugo/content/en/llm_observability/instrument/multimodal.md index 190fd03a5ef..9cb25103d77 100644 --- a/hugo/content/en/llm_observability/instrument/multimodal.md +++ b/hugo/content/en/llm_observability/instrument/multimodal.md @@ -219,23 +219,18 @@ For the full message schema, see [Message][2] in the [Agent Observability HTTP A {{% /tab %}} {{% tab "OpenTelemetry" %}} -If you use [OpenTelemetry instrumentation][1], no code change is required. Datadog extracts media from messages at ingestion, and how much it can extract depends on the semantic convention your instrumentation emits: +If you use [OpenTelemetry instrumentation][1], no code change is required. Datadog extracts media at ingestion from message parts that follow the [OpenTelemetry GenAI semantic conventions for message parts][2]: -| Convention | Images | Audio | -|------------|--------|-------| -| [OpenTelemetry 1.37+][2] | Inline `blob` parts, and `uri` parts carrying a base64 data URI | Inline `blob` parts | -| [OpenInference][3] | Ordered image content carrying a base64 data URI | Not extracted | -| [Langfuse][4] | `image_url` blocks carrying a base64 data URI | Not extracted | -| [OpenLLMetry][5] | Not extracted | Not extracted | +- A `blob` part with a `mime_type` and inline bytes becomes an image or audio part. When the part omits `modality`, Datadog infers it from the MIME type. +- A `uri` part carrying a base64 image data URI, such as `data:image/png;base64,...`, becomes an image part. -Only media carried inline as base64 bytes is rendered. A remote URL is recorded as a text reference and is never fetched. The [OpenTelemetry GenAI conventions][6] specify `blob` as the part type for inline base64 data, so prefer `blob` for both audio and images. +Only media carried inline as base64 bytes is rendered. A remote URL is recorded as a text reference and is never fetched. Audio reaches the trace view through `blob` parts only, and the conventions specify `blob` as the part type for inline base64 data, so prefer `blob` for both audio and images. + +For the full mapping, see [Media in messages][3]. [1]: /llm_observability/instrument/otel_instrumentation/ -[2]: /llm_observability/instrument/otel_instrumentation/#media-in-messages -[3]: /llm_observability/instrument/otel_instrumentation/#media-in-openinference-messages -[4]: /llm_observability/instrument/otel_instrumentation/#media-in-langfuse-messages -[5]: /llm_observability/instrument/otel_instrumentation/#media-in-openllmetry-messages -[6]: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/model/gen-ai/gen-ai-input-messages.json +[2]: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/model/gen-ai/gen-ai-input-messages.json +[3]: /llm_observability/instrument/otel_instrumentation/#media-in-messages {{% /tab %}} {{< /tabs >}} diff --git a/hugo/content/en/llm_observability/instrument/otel_instrumentation.md b/hugo/content/en/llm_observability/instrument/otel_instrumentation.md index 3bb07cf5f11..a413617e62d 100644 --- a/hugo/content/en/llm_observability/instrument/otel_instrumentation.md +++ b/hugo/content/en/llm_observability/instrument/otel_instrumentation.md @@ -31,14 +31,7 @@ You can use OpenTelemetry spans inside [Agent Observability Experiments](/llm_ob ### Multimodal support -Audio and images on OpenTelemetry messages are rendered in the trace view. Coverage depends on the semantic convention your instrumentation emits: - -| Convention | Images | Audio | -|------------|--------|-------| -| [OpenTelemetry 1.37+](#media-in-messages) | Inline `blob` parts, and `uri` parts carrying a base64 data URI | Inline `blob` parts | -| [OpenInference](#media-in-openinference-messages) | Ordered image content carrying a base64 data URI | Not extracted | -| [Langfuse](#media-in-langfuse-messages) | `image_url` blocks carrying a base64 data URI | Not extracted | -| [OpenLLMetry](#media-in-openllmetry-messages) | Not extracted | Not extracted | +Audio and images on OpenTelemetry messages are rendered in the trace view. Datadog extracts media from message parts that follow the OpenTelemetry GenAI semantic conventions, as described in [Media in messages](#media-in-messages). Only media carried inline as base64 bytes is rendered. A remote URL is recorded as a text reference and is never fetched. For the fields, formats, and size limits that apply once media reaches a span, see [Multimodal Support](/llm_observability/instrument/multimodal/). @@ -647,10 +640,6 @@ Messages are converted to OTel-compatible format and processed normally: | `gen_ai.prompt.*` | `meta.input.messages` (llm) / `meta.input.value` (others) | | `gen_ai.completion.*` | `meta.output.messages` (llm) / `meta.output.value` (others) | -##### Media in OpenLLMetry messages - -OpenLLMetry prompt and completion attributes carry text content only, so no audio or images are extracted from them. To render media on a span, emit it as [OpenTelemetry 1.37+ message parts](#media-in-messages) alongside the OpenLLMetry attributes. OTel standard attributes take priority, so both can be present on the same span. - #### Tool calls Tool calls are nested within completion attributes. @@ -779,15 +768,6 @@ The following indexed message attributes are supported: Image content maps to an image URI while preserving its position among other message content. -##### Media in OpenInference messages - -Ordered image content (`message.contents..message_content.image.image.url`) maps to an image URI. Because OpenInference carries images as a URL, what happens next depends on the URL: - -- A base64 data URI, such as `data:image/png;base64,...`, is extracted to `image_parts` and rendered in the trace view. -- A remote URL is recorded as the text reference `[image: ]`. The URL is not fetched. - -Audio is not extracted. The OpenInference specification defines `message_content.type: "audio"` along with the span-level attributes `audio.url`, `audio.mime_type`, and `audio.transcript`, but these are not mapped to Agent Observability fields. To render audio on a span, emit it as an [OpenTelemetry 1.37+ `blob` part](#media-in-messages) alongside the OpenInference attributes. OTel standard attributes take priority, so both can be present on the same span. - #### Embedding spans | OpenInference Source | Agent Observability Field | @@ -863,15 +843,6 @@ Each message is converted to the parts-based message shape: - A `tool_calls` array on a message becomes `tool_call` parts. - A message with `role: tool` and a `tool_call_id` becomes a `tool_result` part. -##### Media in Langfuse messages - -An `image_url` content block maps to an image URI. Because Langfuse carries images as a URL, what happens next depends on the URL: - -- A base64 data URI, such as `data:image/png;base64,...`, is extracted to `image_parts` and rendered in the trace view. -- A remote URL is recorded as the text reference `[image: ]`. The URL is not fetched. - -Audio is not extracted from Langfuse content blocks. To render audio on a span, emit it as an [OpenTelemetry 1.37+ `blob` part](#media-in-messages) alongside the Langfuse attributes. OTel standard attributes take priority, so both can be present on the same span. - ##### Tool spans For `tool`, `agent`, and `workflow` spans, `langfuse.observation.input`/`langfuse.observation.output` are used directly as `input.value`/`output.value`, after the standard `gen_ai.tool.call.*` fallback. From 948aa76d36f24c09d3b009089b17d1329ac5a923 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Fri, 4 Sep 2026 15:33:40 -0600 Subject: [PATCH 8/8] Clarify multimodal instrumentation guidance --- .../instrument/multimodal.md | 84 ++++++++++--------- 1 file changed, 44 insertions(+), 40 deletions(-) diff --git a/hugo/content/en/llm_observability/instrument/multimodal.md b/hugo/content/en/llm_observability/instrument/multimodal.md index 9cb25103d77..d29692b8655 100644 --- a/hugo/content/en/llm_observability/instrument/multimodal.md +++ b/hugo/content/en/llm_observability/instrument/multimodal.md @@ -15,27 +15,22 @@ further_reading: ## Overview -You can attach media (images and audio) to LLM spans and render it directly in the trace view, alongside the rest of the span's data. This lets you troubleshoot **voice agents** and **agents that reason over images** with their full context. - -Multimodal support enables you to: - -- Attach images to LLM span messages and see them rendered inline in the trace view -- Attach audio to LLM span messages and play it back from an inline audio player -- See per-turn transcripts alongside the audio for voice conversations -- Inspect the exact image a model was shown, and hear the exact audio it received and produced, without reconstructing it from logs or a separate storage system +Attach images and audio to LLM spans to troubleshoot voice agents and agents that process images. Inspect the images a model received and play its input and output audio directly in the trace view, alongside other span data. You don't need to reconstruct the interaction from logs or separate storage. For voice conversations, view each turn's transcript alongside its audio. {{< img src="llm_observability/instrumentation/image_example.png" alt="An LLM span in the Agent Observability trace view. The input USER message shows the prompt 'What is in this image?' with an inline photo of a black puppy, and the output ASSISTANT message describes it as a black Labrador Retriever puppy on a wooden surface." style="width:100%;" >}} -Media is attached per message using the typed fields `image_parts` and `audio_parts`. [Automatic instrumentation](#automatic-instrumentation) populates these fields for supported integrations, so media from those libraries appears in traces without any code change. For anything else, [manual instrumentation](#manual-instrumentation) covers how to attach media to a span. +Media belongs to individual messages within an LLM span. [Automatic instrumentation](#automatic-instrumentation) captures media from supported libraries. For other libraries, use [manual instrumentation](#manual-instrumentation) to attach media. For OpenTelemetry spans, see [OpenTelemetry instrumentation](#opentelemetry-instrumentation). ## Requirements -| Capability | Python | Node.js | -| ---------- | ------ | ------- | -| Audio on messages | >= 4.12.0 | >= 6.7.0, or >= 5.118.0 | -| Images on messages | >= 4.13.0 | >= 6.10.0, or >= 5.121.0 | +The following Datadog SDK versions support attaching media to messages. For automatic capture, see the [integration-specific requirements](#automatic-instrumentation). + +| Capability | Python SDK (`ddtrace`) | Node.js SDK (`dd-trace`) | +| ---------- | --------------------- | ----------------------- | +| Audio on messages | >= 4.12.0 | v6: >= 6.7.0; v5: >= 5.118.0 | +| Images on messages | >= 4.13.0 | v6: >= 6.10.0; v5: >= 5.121.0 | -The [Agent Observability HTTP API][1] and [OpenTelemetry instrumentation][2] have no version requirement. +No minimum Datadog SDK version applies to the [Agent Observability HTTP API][1] or [OpenTelemetry instrumentation][2]. OpenTelemetry instrumentation must emit the [supported GenAI message parts](#opentelemetry-instrumentation). ## Automatic instrumentation @@ -44,8 +39,8 @@ Datadog's [LLM integrations][3] populate media parts for the following libraries {{< tabs >}} {{% tab "Python" %}} -| Integration | Media captured | Tracer version | -| ----------- | -------------- | -------------- | +| Integration | Media captured | Datadog SDK version | +| ----------- | -------------- | ------------------- | | OpenAI chat completions | Input `input_audio` content and output audio, including the model's audio transcript | >= 4.12.0 | | OpenAI Realtime API | Per-turn input and output audio, transcripts, token usage, and tool calls | >= 4.13.0 | | LiteLLM | Input and output audio, through the OpenAI chat completions format | >= 4.12.0 | @@ -60,17 +55,28 @@ Audio in streamed chat completions (`stream=True`) is not captured. {{% /tab %}} {{% tab "Node.js" %}} -| Integration | Media captured | Tracer version | -| ----------- | -------------- | -------------- | -| OpenAI chat completions | Input `input_audio` content and output audio, including the model's audio transcript | >= 6.7.0, or >= 5.118.0 | -| OpenAI Agents | Input and output audio, through the OpenAI chat completions format | >= 6.7.0, or >= 5.118.0 | +| Integration | Media captured | Datadog SDK version | +| ----------- | -------------- | ------------------- | +| OpenAI chat completions | Input `input_audio` content and output audio, including the model's audio transcript | v6: >= 6.7.0; v5: >= 5.118.0 | +| OpenAI Agents | Input and output audio, through the OpenAI chat completions format | v6: >= 6.7.0; v5: >= 5.118.0 | Audio in streamed chat completions is not captured. {{% /tab %}} {{< /tabs >}} -Images referenced by a remote URL or a file ID are not fetched. Only images the application sends inline as base64 data are captured. Where an integration finds media it cannot capture, it records a text marker such as `[image]` or `[audio]` in the message content so that the media is still accounted for. +Integrations don't fetch images referenced by a remote URL or a file ID. They capture only images that the application sends inline as base64 data. When an integration cannot capture media, it records a text marker such as `[image]` or `[audio]` in the message. + +## OpenTelemetry instrumentation + +Datadog extracts media from [OpenTelemetry spans][2] that contain the following [GenAI message parts][7]. No additional instrumentation is needed if your application already emits these parts: + +- A `blob` part with a `mime_type` and inline bytes becomes an image or audio part. When the part omits `modality`, Datadog infers it from the MIME type. +- A `uri` part carrying a base64 image data URI, such as `data:image/png;base64,...`, becomes an image part. + +Only media carried inline as base64 bytes is rendered. A remote URL is recorded as a text reference and is never fetched. For audio playback, send audio in `blob` parts. The conventions specify `blob` for inline base64 data, so prefer `blob` for both audio and images. + +For the full mapping, see [Media in messages][8]. ## Manual instrumentation @@ -83,11 +89,15 @@ Each media part takes the following fields: | `mime_type` | string | Yes | The media type of the attachment, such as `image/png` or `audio/wav`. | | `content` | string | Yes | The base64-encoded media. | -Datadog moves inline media out of the span payload during ingestion and the trace view fetches it on demand, so a large attachment does not weigh down the span itself. +Datadog stores media separately during ingestion and loads it on demand in the trace view. Inline media still counts toward the span event size limit. Before attaching media, check the [supported formats](#supported-formats) and [size limits](#limits-and-behavior). + +The examples focus on annotation and omit SDK setup and model calls. For SDK setup, see the [Agent Observability SDK reference][6]. {{< tabs >}} {{% tab "Python" %}} +**Attach an image** + ```python import base64 from ddtrace.llmobs import LLMObs @@ -111,6 +121,8 @@ def describe_image(image_bytes): return resp ``` +**Attach audio and a transcript** + ```python import base64 from ddtrace.llmobs import LLMObs @@ -142,13 +154,15 @@ def voice_turn(user_audio_bytes): return resp ``` -A media part that is missing `mime_type` or `content` raises a `TypeError`. +If a media part is missing `mime_type` or `content`, annotation fails. {{% /tab %}} {{% tab "Node.js" %}} Node.js uses the camelCase field names `imageParts` and `audioParts`, with `mimeType` on each part. +**Attach an image** + ```javascript function describeImage (imageBytes) { const resp = ... // multimodal (vision) LLM call @@ -167,6 +181,8 @@ function describeImage (imageBytes) { describeImage = llmobs.wrap({ kind: 'llm', modelName: 'gpt-4o', modelProvider: 'openai' }, describeImage) ``` +**Attach audio and a transcript** + ```javascript function voiceTurn (userAudioBytes) { const resp = ... // multimodal (audio) LLM call @@ -217,24 +233,9 @@ For the full message schema, see [Message][2] in the [Agent Observability HTTP A [2]: /llm_observability/instrument/api/#message [3]: /llm_observability/instrument/api/ {{% /tab %}} -{{% tab "OpenTelemetry" %}} - -If you use [OpenTelemetry instrumentation][1], no code change is required. Datadog extracts media at ingestion from message parts that follow the [OpenTelemetry GenAI semantic conventions for message parts][2]: - -- A `blob` part with a `mime_type` and inline bytes becomes an image or audio part. When the part omits `modality`, Datadog infers it from the MIME type. -- A `uri` part carrying a base64 image data URI, such as `data:image/png;base64,...`, becomes an image part. - -Only media carried inline as base64 bytes is rendered. A remote URL is recorded as a text reference and is never fetched. Audio reaches the trace view through `blob` parts only, and the conventions specify `blob` as the part type for inline base64 data, so prefer `blob` for both audio and images. - -For the full mapping, see [Media in messages][3]. - -[1]: /llm_observability/instrument/otel_instrumentation/ -[2]: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/model/gen-ai/gen-ai-input-messages.json -[3]: /llm_observability/instrument/otel_instrumentation/#media-in-messages -{{% /tab %}} {{< /tabs >}} -## View media in the trace view +## View images and play audio Open a trace in [Trace Explorer][4] and select an LLM span. Media attached to the span's messages renders inside the message it belongs to. @@ -261,7 +262,7 @@ Audio plays through the browser's native audio player. Supported containers incl - **Use the typed media fields.** Attach media with `image_parts` and `audio_parts` rather than embedding base64 strings in a message's `content`. Only typed fields render in the trace view. - **Attach the transcript with the audio.** Set the message `content` to the turn transcript so that the span is readable, searchable, and usable by evaluations even before the audio is played. -## Further Reading +## Further reading {{< partial name="whats-next/whats-next.html" >}} @@ -270,3 +271,6 @@ Audio plays through the browser's native audio player. Supported containers incl [3]: /llm_observability/instrument/auto_instrumentation/ [4]: https://app.datadoghq.com/llm/traces [5]: /security/sensitive_data_scanner/ +[6]: /llm_observability/instrument/sdk/ +[7]: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/model/gen-ai/gen-ai-input-messages.json +[8]: /llm_observability/instrument/otel_instrumentation/#media-in-messages