diff --git a/hugo/config/_default/menus/main.en.yaml b/hugo/config/_default/menus/main.en.yaml index 39e9cd6b59f..1840400f923 100644 --- a/hugo/config/_default/menus/main.en.yaml +++ b/hugo/config/_default/menus/main.en.yaml @@ -5644,6 +5644,11 @@ menu: parent: llm_obs_instrumentation identifier: llm_obs_prompt_tracking weight: 311 + - name: Multimodal + url: llm_observability/instrument/multimodal + parent: llm_obs_instrumentation + identifier: llm_obs_multimodal + weight: 312 - name: Investigate url: llm_observability/investigate/ parent: llm_obs diff --git a/hugo/content/en/llm_observability/instrument/api.md b/hugo/content/en/llm_observability/instrument/api.md index d8275086e97..363417a5cf1 100644 --- a/hugo/content/en/llm_observability/instrument/api.md +++ b/hugo/content/en/llm_observability/instrument/api.md @@ -206,23 +206,21 @@ If the request is successful, the API responds with a 202 network code and an em #### AudioPart -An audio segment on a message. Provide either `content` or `attachment_key`. +An audio segment on a message. For supported formats and size limits, see [Multimodal Support][6]. | Field | Type | Description | |-------|------|-------------| -| mime_type [*required*] | string | The media type of the audio, such as `audio/wav` or `audio/pcm`. | -| content | string | The base64-encoded audio, carried inline with the message. | -| attachment_key | string | A reference to audio stored outside the span payload, instead of inline `content`. | +| mime_type [*required*] | string | The media type of the audio, such as `audio/wav` or `audio/mpeg`. | +| content [*required*] | string | The base64-encoded audio, carried inline with the message. | #### ImagePart -An image on a message. Provide either `content` or `attachment_key`. +An image on a message. For supported formats and size limits, see [Multimodal Support][6]. | Field | Type | Description | |-------|------|-------------| | mime_type [*required*] | string | The media type of the image, such as `image/png` or `image/jpeg`. | -| content | string | The base64-encoded image, carried inline with the message. | -| attachment_key | string | A reference to an image stored outside the span payload, instead of inline `content`. | +| content [*required*] | string | The base64-encoded image, carried inline with the message. | #### ToolDefinition @@ -704,3 +702,4 @@ For feedback events, provide exactly one of `span_id`, `trace_id`, `session_id`, [3]: /getting_started/tagging/ [4]: /llm_observability/investigate/evaluations/end_user_feedback [5]: /llm_observability/instrument/sdk/?tab=python#enriching-spans +[6]: /llm_observability/instrument/multimodal/ diff --git a/hugo/content/en/llm_observability/instrument/multimodal.md b/hugo/content/en/llm_observability/instrument/multimodal.md new file mode 100644 index 00000000000..d29692b8655 --- /dev/null +++ b/hugo/content/en/llm_observability/instrument/multimodal.md @@ -0,0 +1,276 @@ +--- +title: Multimodal Support +description: Attach audio and images to LLM spans in Agent Observability and render them inline in the trace view, automatically or through manual instrumentation. +further_reading: + - link: '/llm_observability/instrument/sdk/' + tag: 'Documentation' + text: 'Agent Observability SDK Reference' + - link: '/llm_observability/instrument/auto_instrumentation/' + tag: 'Documentation' + text: 'Automatic Instrumentation for Agent Observability' + - link: '/llm_observability/instrument/api/' + tag: 'Documentation' + text: 'Agent Observability HTTP API Reference' +--- + +## Overview + +Attach images and audio to LLM spans to troubleshoot voice agents and agents that process images. Inspect the images a model received and play its input and output audio directly in the trace view, alongside other span data. You don't need to reconstruct the interaction from logs or separate storage. For voice conversations, view each turn's transcript alongside its audio. + +{{< img src="llm_observability/instrumentation/image_example.png" alt="An LLM span in the Agent Observability trace view. The input USER message shows the prompt 'What is in this image?' with an inline photo of a black puppy, and the output ASSISTANT message describes it as a black Labrador Retriever puppy on a wooden surface." style="width:100%;" >}} + +Media belongs to individual messages within an LLM span. [Automatic instrumentation](#automatic-instrumentation) captures media from supported libraries. For other libraries, use [manual instrumentation](#manual-instrumentation) to attach media. For OpenTelemetry spans, see [OpenTelemetry instrumentation](#opentelemetry-instrumentation). + +## Requirements + +The following Datadog SDK versions support attaching media to messages. For automatic capture, see the [integration-specific requirements](#automatic-instrumentation). + +| Capability | Python SDK (`ddtrace`) | Node.js SDK (`dd-trace`) | +| ---------- | --------------------- | ----------------------- | +| Audio on messages | >= 4.12.0 | v6: >= 6.7.0; v5: >= 5.118.0 | +| Images on messages | >= 4.13.0 | v6: >= 6.10.0; v5: >= 5.121.0 | + +No minimum Datadog SDK version applies to the [Agent Observability HTTP API][1] or [OpenTelemetry instrumentation][2]. OpenTelemetry instrumentation must emit the [supported GenAI message parts](#opentelemetry-instrumentation). + +## Automatic instrumentation + +Datadog's [LLM integrations][3] populate media parts for the following libraries, with no code change on your part. + +{{< tabs >}} +{{% tab "Python" %}} + +| Integration | Media captured | Datadog SDK version | +| ----------- | -------------- | ------------------- | +| OpenAI chat completions | Input `input_audio` content and output audio, including the model's audio transcript | >= 4.12.0 | +| OpenAI Realtime API | Per-turn input and output audio, transcripts, token usage, and tool calls | >= 4.13.0 | +| LiteLLM | Input and output audio, through the OpenAI chat completions format | >= 4.12.0 | +| OpenAI chat completions and Responses API | Inline base64-encoded images | >= 4.15.0 | +| OpenAI Agents | Inline base64-encoded images | >= 4.15.0 | +| Anthropic | Inline base64-encoded images | >= 4.15.0 | + +Each turn of an OpenAI Realtime API conversation produces its own LLM span. Raw `pcm16` and G.711 audio is converted to WAV so that it plays in the browser. To turn off Realtime API instrumentation, set `DD_OPENAI_REALTIME_ENABLED=false`. + +Audio in streamed chat completions (`stream=True`) is not captured. + +{{% /tab %}} +{{% tab "Node.js" %}} + +| Integration | Media captured | Datadog SDK version | +| ----------- | -------------- | ------------------- | +| OpenAI chat completions | Input `input_audio` content and output audio, including the model's audio transcript | v6: >= 6.7.0; v5: >= 5.118.0 | +| OpenAI Agents | Input and output audio, through the OpenAI chat completions format | v6: >= 6.7.0; v5: >= 5.118.0 | + +Audio in streamed chat completions is not captured. + +{{% /tab %}} +{{< /tabs >}} + +Integrations don't fetch images referenced by a remote URL or a file ID. They capture only images that the application sends inline as base64 data. When an integration cannot capture media, it records a text marker such as `[image]` or `[audio]` in the message. + +## OpenTelemetry instrumentation + +Datadog extracts media from [OpenTelemetry spans][2] that contain the following [GenAI message parts][7]. No additional instrumentation is needed if your application already emits these parts: + +- A `blob` part with a `mime_type` and inline bytes becomes an image or audio part. When the part omits `modality`, Datadog infers it from the MIME type. +- A `uri` part carrying a base64 image data URI, such as `data:image/png;base64,...`, becomes an image part. + +Only media carried inline as base64 bytes is rendered. A remote URL is recorded as a text reference and is never fetched. For audio playback, send audio in `blob` parts. The conventions specify `blob` for inline base64 data, so prefer `blob` for both audio and images. + +For the full mapping, see [Media in messages][8]. + +## Manual instrumentation + +Attach media to an LLM span by adding `image_parts` or `audio_parts` to a message in `input_data` or `output_data`. + +Each media part takes the following fields: + +| Field | Type | Required | Description | +| ----- | ---- | -------- | ----------- | +| `mime_type` | string | Yes | The media type of the attachment, such as `image/png` or `audio/wav`. | +| `content` | string | Yes | The base64-encoded media. | + +Datadog stores media separately during ingestion and loads it on demand in the trace view. Inline media still counts toward the span event size limit. Before attaching media, check the [supported formats](#supported-formats) and [size limits](#limits-and-behavior). + +The examples focus on annotation and omit SDK setup and model calls. For SDK setup, see the [Agent Observability SDK reference][6]. + +{{< tabs >}} +{{% tab "Python" %}} + +**Attach an image** + +```python +import base64 +from ddtrace.llmobs import LLMObs +from ddtrace.llmobs.decorators import llm + +@llm(model_name="gpt-4o", model_provider="openai") +def describe_image(image_bytes): + resp = ... # multimodal (vision) LLM call + LLMObs.annotate( + input_data=[ + { + "role": "user", + "content": "What is in this image?", + "image_parts": [ + {"mime_type": "image/png", "content": base64.b64encode(image_bytes).decode("utf-8")} + ], + } + ], + output_data=[{"role": "assistant", "content": "The image shows a golden retriever puppy."}], + ) + return resp +``` + +**Attach audio and a transcript** + +```python +import base64 +from ddtrace.llmobs import LLMObs +from ddtrace.llmobs.decorators import llm + +@llm(model_name="gpt-realtime", model_provider="openai") +def voice_turn(user_audio_bytes): + resp = ... # multimodal (audio) LLM call + LLMObs.annotate( + input_data=[ + { + "role": "user", + "content": "Hey, how are you?", # transcript of the input audio + "audio_parts": [ + {"mime_type": "audio/wav", "content": base64.b64encode(user_audio_bytes).decode("utf-8")} + ], + } + ], + output_data=[ + { + "role": "assistant", + "content": "Hey! I'm doing great, thanks for asking. How about you?", + "audio_parts": [ + {"mime_type": "audio/wav", "content": base64.b64encode(resp.audio_bytes).decode("utf-8")} + ], + } + ], + ) + return resp +``` + +If a media part is missing `mime_type` or `content`, annotation fails. + +{{% /tab %}} +{{% tab "Node.js" %}} + +Node.js uses the camelCase field names `imageParts` and `audioParts`, with `mimeType` on each part. + +**Attach an image** + +```javascript +function describeImage (imageBytes) { + const resp = ... // multimodal (vision) LLM call + llmobs.annotate({ + inputData: [ + { + role: 'user', + content: 'What is in this image?', + imageParts: [{ mimeType: 'image/png', content: imageBytes.toString('base64') }] + } + ], + outputData: [{ role: 'assistant', content: 'The image shows a golden retriever puppy.' }] + }) + return resp +} +describeImage = llmobs.wrap({ kind: 'llm', modelName: 'gpt-4o', modelProvider: 'openai' }, describeImage) +``` + +**Attach audio and a transcript** + +```javascript +function voiceTurn (userAudioBytes) { + const resp = ... // multimodal (audio) LLM call + llmobs.annotate({ + inputData: [ + { + role: 'user', + content: 'Hey, how are you?', // transcript of the input audio + audioParts: [{ mimeType: 'audio/wav', content: userAudioBytes.toString('base64') }] + } + ], + outputData: [ + { + role: 'assistant', + content: "Hey! I'm doing great, thanks for asking. How about you?", + audioParts: [{ mimeType: 'audio/wav', content: resp.audioBuffer.toString('base64') }] + } + ] + }) + return resp +} +voiceTurn = llmobs.wrap({ kind: 'llm', modelName: 'gpt-audio', modelProvider: 'openai' }, voiceTurn) +``` + +A media part that is missing `mimeType` or `content` is dropped. The rest of the message is still recorded. + +{{% /tab %}} +{{% tab "API" %}} + +Submit `audio_parts` and `image_parts` on the message objects you send to the [Spans API endpoint][1]. The field names match the Python SDK. + +```json +{ + "role": "user", + "content": "What is in this image?", + "image_parts": [ + { + "mime_type": "image/png", + "content": "iVBORw0KGgoAAAANSUhEUgAA..." + } + ] +} +``` + +For the full message schema, see [Message][2] in the [Agent Observability HTTP API reference][3]. + +[1]: /llm_observability/instrument/api/#spans-api +[2]: /llm_observability/instrument/api/#message +[3]: /llm_observability/instrument/api/ +{{% /tab %}} +{{< /tabs >}} + +## View images and play audio + +Open a trace in [Trace Explorer][4] and select an LLM span. Media attached to the span's messages renders inside the message it belongs to. + +Images appear as thumbnails. Select a thumbnail to open the image at full size. Up to 12 images render per message. + +Audio appears as a player, with the message text shown alongside it. For a voice conversation, that text is the turn transcript. Select the player to load and play the audio. + +{{< img src="llm_observability/instrumentation/audio_example.png" alt="An LLM span in the Agent Observability trace view. The input message from the USER shows an inline audio player with the transcript 'Hey, how are you?', and the output ASSISTANT message shows a 'Click to play audio' control with the transcript 'Hey! I'm doing great, thanks for asking. How about you?'." style="width:100%;" >}} + +### Supported formats + +Images render when the MIME type is one of `image/png`, `image/jpeg`, `image/webp`, or `image/gif`, and the declared MIME type matches the encoded bytes. SVG images do not render. The trace view makes no outbound requests, so an image referenced by a remote URL is shown as text rather than fetched. + +Audio plays through the browser's native audio player. Supported containers include `audio/wav`, `audio/mpeg`, `audio/ogg`, and `audio/webm`. Raw formats such as `pcm16`, `g711_ulaw`, and `g711_alaw` do not play. Convert raw audio to WAV before attaching it. The Python OpenAI Realtime API integration does this conversion for you. + +## Limits and behavior + +- **Span event size.** A span event is capped at 5 MB. When a span exceeds the cap, its input and output are replaced with a placeholder and `dropped_io` is added to the span's `collection_errors` attribute. The span itself is kept. +- **Per-part size.** Automatic instrumentation in Python caps a single inline media part at 4 MiB. Above the cap, the integration records a text marker such as `[audio]` or `[image omitted: too large]` and preserves the surrounding message text and model response. +- **Sensitive Data Scanner.** Media content is not scanned by [Sensitive Data Scanner][5]. Message text, tool arguments, and tool results are scanned as usual. + +## Best practices + +- **Use the typed media fields.** Attach media with `image_parts` and `audio_parts` rather than embedding base64 strings in a message's `content`. Only typed fields render in the trace view. +- **Attach the transcript with the audio.** Set the message `content` to the turn transcript so that the span is readable, searchable, and usable by evaluations even before the audio is played. + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: /llm_observability/instrument/api/ +[2]: /llm_observability/instrument/otel_instrumentation/ +[3]: /llm_observability/instrument/auto_instrumentation/ +[4]: https://app.datadoghq.com/llm/traces +[5]: /security/sensitive_data_scanner/ +[6]: /llm_observability/instrument/sdk/ +[7]: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/model/gen-ai/gen-ai-input-messages.json +[8]: /llm_observability/instrument/otel_instrumentation/#media-in-messages diff --git a/hugo/content/en/llm_observability/instrument/otel_instrumentation.md b/hugo/content/en/llm_observability/instrument/otel_instrumentation.md index 68f3a4a123d..a413617e62d 100644 --- a/hugo/content/en/llm_observability/instrument/otel_instrumentation.md +++ b/hugo/content/en/llm_observability/instrument/otel_instrumentation.md @@ -29,6 +29,12 @@ For information on using Prompt Tracking with OpenTelemetry spans, see [Prompt T You can use OpenTelemetry spans inside [Agent Observability Experiments](/llm_observability/improve/experiments/setup#using-opentelemetry-spans-inside-experiments). By setting `DD_TRACE_OTEL_ENABLED=1`, OTel spans created inside an experiment task automatically appear as children of the experiment span. +### Multimodal support + +Audio and images on OpenTelemetry messages are rendered in the trace view. Datadog extracts media from message parts that follow the OpenTelemetry GenAI semantic conventions, as described in [Media in messages](#media-in-messages). + +Only media carried inline as base64 bytes is rendered. A remote URL is recorded as a text reference and is never fetched. For the fields, formats, and size limits that apply once media reaches a span, see [Multimodal Support](/llm_observability/instrument/multimodal/). + ### Span links Use [OpenTelemetry span links][9] on your GenAI spans to express non-parent-child relationships, such as when one span's output feeds another span's input. When two linked spans are in the same trace, the link appears as an edge in that trace's **Execution Graph**, so you can see how data flows between sibling spans (for example, a tool's output feeding a downstream LLM call). @@ -521,6 +527,23 @@ Input and output messages are extracted from the following sources, in priority | `gen_ai.output.messages` | `meta.output.messages` (llm) / `meta.output.value` (others) | | | `gen_ai.system_instructions` | Prepended to input | Added as system role messages | +##### Media in messages + +Message parts that carry media are extracted into the typed `audio_parts` and `image_parts` fields on the message: + +| Part type | Behavior | +|-----------|----------| +| `blob` with `mime_type` and inline bytes | Extracted to `image_parts` or `audio_parts`. When the part omits `modality`, it is inferred from the MIME type. | +| `uri` carrying a base64 image data URI, such as `data:image/png;base64,...` | Extracted to `image_parts`. | +| `uri` carrying a remote URL | Recorded as the text reference `[: ]`. The URL is not fetched. | +| `file` with a `file_id` | Recorded as the text reference `[ file: ]`. | + +A positional marker such as `[image blob: image/png]` is also added to the message text so that media keeps its place among the other parts. + +Audio reaches `audio_parts` through `blob` parts only. An audio data URI on a `uri` part is recorded as text, and the conventions specify `blob` as the part type for inline base64 data, so prefer `blob` for both audio and images. + +For the formats the trace view renders and the size limits that apply, see [Multimodal Support](/llm_observability/instrument/multimodal/). + ##### Embedding spans | OTel Source | Agent Observability Field | diff --git a/hugo/content/en/llm_observability/instrument/sdk.md b/hugo/content/en/llm_observability/instrument/sdk.md index 50261dea7ee..1d635d24686 100644 --- a/hugo/content/en/llm_observability/instrument/sdk.md +++ b/hugo/content/en/llm_observability/instrument/sdk.md @@ -1307,11 +1307,11 @@ The `LLMObs.annotate()` method accepts the following arguments: `input_data` : optional - _JSON serializable type or list of dictionaries_ -
Either a JSON serializable type (for non-LLM spans) or a list of dictionaries with this format: `{"content": "...", "role": "...", "tool_calls": ..., "tool_results": ..., "audio_parts": ..., "image_parts": ...}`, where `"tool_calls"` are an optional list of tool call dictionaries with required keys: `"name"`, `"arguments"`, and optional keys: `"tool_id"`, `"type"`, and `"tool_results"` are an optional list of tool result dictionaries with required key: `"result"`, and optional keys: `"name"`, `"tool_id"`, `"type"` for function calling scenarios. `"audio_parts"` and `"image_parts"` are optional lists of media dictionaries for multimodal spans, each with a required `"mime_type"` and exactly one of `"content"` (base64-encoded media, carried inline) or `"attachment_key"`. **Note**: Embedding spans are a special case and require a string or a dictionary (or a list of dictionaries) with this format: `{"text": "..."}`. +
Either a JSON serializable type (for non-LLM spans) or a list of dictionaries with this format: `{"content": "...", "role": "...", "tool_calls": ..., "tool_results": ..., "audio_parts": ..., "image_parts": ...}`, where `"tool_calls"` are an optional list of tool call dictionaries with required keys: `"name"`, `"arguments"`, and optional keys: `"tool_id"`, `"type"`, and `"tool_results"` are an optional list of tool result dictionaries with required key: `"result"`, and optional keys: `"name"`, `"tool_id"`, `"type"` for function calling scenarios. `"audio_parts"` and `"image_parts"` are optional lists of media dictionaries for multimodal spans, each with a required `"mime_type"` and `"content"` (base64-encoded media, carried inline). **Note**: Embedding spans are a special case and require a string or a dictionary (or a list of dictionaries) with this format: `{"text": "..."}`. `output_data` : optional - _JSON serializable type or list of dictionaries_ -
Either a JSON serializable type (for non-LLM spans) or a list of dictionaries with this format: `{"content": "...", "role": "...", "tool_calls": ..., "audio_parts": ..., "image_parts": ...}`, where `"tool_calls"` are an optional list of tool call dictionaries with required keys: `"name"`, `"arguments"`, and optional keys: `"tool_id"`, `"type"` for function calling scenarios. `"audio_parts"` and `"image_parts"` are optional lists of media dictionaries for multimodal spans, each with a required `"mime_type"` and exactly one of `"content"` (base64-encoded media, carried inline) or `"attachment_key"`. **Note**: Retrieval spans are a special case and require a string or a dictionary (or a list of dictionaries) with this format: `{"text": "...", "name": "...", "score": float, "id": "..."}`. +
Either a JSON serializable type (for non-LLM spans) or a list of dictionaries with this format: `{"content": "...", "role": "...", "tool_calls": ..., "audio_parts": ..., "image_parts": ...}`, where `"tool_calls"` are an optional list of tool call dictionaries with required keys: `"name"`, `"arguments"`, and optional keys: `"tool_id"`, `"type"` for function calling scenarios. `"audio_parts"` and `"image_parts"` are optional lists of media dictionaries for multimodal spans, each with a required `"mime_type"` and `"content"` (base64-encoded media, carried inline). **Note**: Retrieval spans are a special case and require a string or a dictionary (or a list of dictionaries) with this format: `{"text": "...", "name": "...", "score": float, "id": "..."}`. `tool_definitions` : optional - _list of dictionaries_ @@ -1435,11 +1435,7 @@ def describe_image(image_bytes): {{< /code-block >}} -Messages annotated with `audio_parts` or `image_parts` render as inline audio players and images in the trace view: - -{{< img src="llm_observability/instrumentation/audio_example.png" alt="An LLM span in the Agent Observability trace view. The input message from the USER shows an inline audio player with the transcript 'Hey, how are you?', and the output ASSISTANT message shows a 'Click to play audio' control with the transcript 'Hey! I'm doing great, thanks for asking. How about you?'." style="width:100%;" >}} - -{{< img src="llm_observability/instrumentation/image_example.png" alt="An LLM span in the Agent Observability trace view. The input USER message shows the prompt 'What is in this image?' with an inline photo of a black puppy, and the output ASSISTANT message describes it as a black Labrador Retriever puppy on a wooden surface." style="width:100%;" >}} +Messages annotated with `audio_parts` or `image_parts` render as inline audio players and images in the trace view. For rendered examples, supported formats, size limits, and the integrations that populate these fields automatically, see [Multimodal Support](/llm_observability/instrument/multimodal/). {{% /tab %}} @@ -1461,11 +1457,11 @@ The `annotationOptions` object can contain the following: `inputData` : optional - _JSON serializable type or list of objects_ -
Either a JSON serializable type (for non-LLM spans) or a list of dictionaries with this format: `{role: "...", content: "...", audioParts: [...], imageParts: [...]}` (for LLM spans). `audioParts` and `imageParts` are optional lists of media objects for multimodal spans, each with a required `mimeType` and exactly one of `content` (base64-encoded media, carried inline) or `attachmentKey`. **Note**: Embedding spans are a special case and require a string or an object (or a list of objects) with this format: `{text: "..."}`. +
Either a JSON serializable type (for non-LLM spans) or a list of dictionaries with this format: `{role: "...", content: "...", audioParts: [...], imageParts: [...]}` (for LLM spans). `audioParts` and `imageParts` are optional lists of media objects for multimodal spans, each with a required `mimeType` and `content` (base64-encoded media, carried inline). **Note**: Embedding spans are a special case and require a string or an object (or a list of objects) with this format: `{text: "..."}`. `outputData` : optional - _JSON serializable type or list of objects_ -
Either a JSON serializable type (for non-LLM spans) or a list of objects with this format: `{role: "...", content: "...", audioParts: [...], imageParts: [...]}` (for LLM spans). `audioParts` and `imageParts` are optional lists of media objects for multimodal spans, each with a required `mimeType` and exactly one of `content` (base64-encoded media, carried inline) or `attachmentKey`. **Note**: Retrieval spans are a special case and require a string or an object (or a list of objects) with this format: `{text: "...", name: "...", score: number, id: "..."}`. +
Either a JSON serializable type (for non-LLM spans) or a list of objects with this format: `{role: "...", content: "...", audioParts: [...], imageParts: [...]}` (for LLM spans). `audioParts` and `imageParts` are optional lists of media objects for multimodal spans, each with a required `mimeType` and `content` (base64-encoded media, carried inline). **Note**: Retrieval spans are a special case and require a string or an object (or a list of objects) with this format: `{text: "...", name: "...", score: number, id: "..."}`. `metadata` : optional - _object_ @@ -1576,13 +1572,9 @@ describeImage = llmobs.wrap({ kind: 'llm', modelName: 'gpt-4o', modelProvider: ' {{< /code-block >}} -Messages annotated with `audioParts` or `imageParts` render as inline audio players and images in the trace view: - -{{< img src="llm_observability/instrumentation/audio_example.png" alt="An LLM span in the Agent Observability trace view. The input message from the USER shows an inline audio player with the transcript 'Hey, how are you?', and the output ASSISTANT message shows a 'Click to play audio' control with the transcript 'Hey! I'm doing great, thanks for asking. How about you?'." style="width:100%;" >}} - -{{< img src="llm_observability/instrumentation/image_example.png" alt="An LLM span in the Agent Observability trace view. The input USER message shows the prompt 'What is in this image?' with an inline photo of a black puppy, and the output ASSISTANT message describes it as a black Labrador Retriever puppy on a wooden surface." style="width:100%;" >}} +Messages annotated with `audioParts` or `imageParts` render as inline audio players and images in the trace view. For rendered examples, supported formats, size limits, and the integrations that populate these fields automatically, see [Multimodal Support](/llm_observability/instrument/multimodal/). -For OpenAI audio chat completions, `audioParts` are also captured automatically by [Datadog's LLM integrations](/llm_observability/instrument/auto_instrumentation/)—no manual annotation required. Unlike `audioParts`, `imageParts` are not currently captured automatically and must be annotated manually; automatic capture is planned for a future release. +For OpenAI audio chat completions, `audioParts` are also captured automatically by [Datadog's LLM integrations](/llm_observability/instrument/auto_instrumentation/), with no manual annotation required. The Node.js SDK does not capture `imageParts` automatically. Annotate them as shown above. {{% /tab %}} {{% tab "Java" %}}