[HDX-5162] LLM observability dashboard, span chat view, and sessions - #2990
Open
wrn14897 wants to merge 11 commits into
Open
[HDX-5162] LLM observability dashboard, span chat view, and sessions#2990wrn14897 wants to merge 11 commits into
wrn14897 wants to merge 11 commits into
Conversation
…s, and /llm dashboard
Adds read-time, schema-agnostic LLM observability on top of existing trace
and log data. No ingestion changes: everything derives from span/log
attribute maps at query time, so it works retroactively on already-ingested
telemetry.
- Normalization lib (packages/app/src/llm/lib): detects LLM spans and
normalizes model, provider, token usage (incl. cached/reasoning), cost,
session ids, TTFT, and chat messages across four instrumentation dialects:
OTel GenAI semconv (attribute- and event-based), OpenLLMetry,
OpenInference, and the Vercel AI SDK. Includes real-world variants
observed from opencode and Claude Code telemetry (whole-string
llm.input_messages, camelCase ai.usage.*, flat token/cost keys,
bracketed model ids like claude-opus-5[1m]).
- Span side panel: an LLM tab renders the normalized conversation (roles,
markdown, tool calls) with a usage/cost summary; the Overview tab gains an
LLM section; trace waterfall labels LLM spans with model + token count.
- Cost estimation: bundled model price catalog (adapted from Langfuse's
MIT-licensed price list) with regex matching for provider/Bedrock/Vertex
id flavors; an instrumentation-provided cost attribute always wins.
- /llm preset dashboard (Overview | Sessions | Search):
- Overview: KPI tiles (calls, tokens, est. cost, avg cost/call, cache hit
rate, error rate), calls/error trends, token split
(uncached/cached/output/reasoning), cost by model, finish reasons,
models/services/users/error tables, latency heatmap, p95 by model,
TTFT, and tool analytics.
- Sessions: activity grouped by the cross-dialect session id
(gen_ai.conversation.id, session.id, ai.telemetry.metadata.sessionId)
with a drawer timeline; span attributes load lazily per expanded call
since agent SDKs stamp full conversation history on every span
(~48 MiB -> ~20 KiB list payload).
- Search: side-by-side LLM trace span and log event tables, correlating
signals for instrumentations that emit session ids without trace
context.
- Aggregations gate token/cost sums on authoritative usage reporters so SDK
wrapper spans don't double count.
Known limitations: bundled prices go stale between releases, and apps that
double-instrument (e.g. opencode emitting both OpenInference and Vercel AI
spans per call) still double count in sums.
🦋 Changeset detectedLatest commit: 8eae348 The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
github-actions
Bot
added
the
review/tier-4
Contributor
🔴 Tier 4 — CriticalTouches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI. Why this tier:
Review process: Deep review from a domain expert. Synchronous walkthrough may be required. Stats
|
Contributor
Greptile SummaryThe PR adds a schema-agnostic LLM observability dashboard that derives normalized model, usage, cost, latency, tool, agent, and conversation data from telemetry at query time.
Confidence Score: 4/5The PR does not appear safe to merge until session entries without trace context can be looked up without selecting an unrelated colliding span. Session rows normalize absent trace IDs to an empty string, and the lazy detail query uses that value with span ID and timestamp under an unordered LIMIT 1, leaving the previously reported wrong-conversation lookup reachable. Files Needing Attention: packages/app/src/llm/dashboard/SessionSpanDetail.tsx and packages/app/src/llm/dashboard/LLMSessionPanel.tsx
|
| Filename | Overview |
|---|---|
| packages/app/src/llm/dashboard/SessionSpanDetail.tsx | Adds lazy attribute loading for a selected session span, but rows lacking trace context remain ambiguously identified. |
| packages/app/src/llm/dashboard/LLMSessionPanel.tsx | Queries session summaries and lazily expands individual spans while forwarding normalized trace, span, and timestamp identity. |
| packages/app/src/llm/hooks/useLLMDashboardExpressions.ts | Builds source-aware normalized SQL expressions consumed throughout the LLM dashboard. |
| packages/app/src/llm/lib/expressions.ts | Defines cross-dialect ClickHouse expressions for LLM detection, usage, cost, sessions, and attribution. |
| packages/app/src/components/DBDeltaChart.tsx | Adds configurable sampling projections and stable priority-property partitioning for domain-specific delta views. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
Telemetry[Span and log attributes] --> Normalize[Cross-dialect LLM expressions]
Normalize --> Queries[ClickHouse read-time queries]
Queries --> Overview[Overview charts]
Queries --> Latency[Latency and delta analysis]
Queries --> Sessions[Session timeline]
Sessions --> Detail[Lazy span attribute lookup]
Detail --> Chat[Normalized chat view]
Queries --> Search[Span and log search]
Reviews (11): Last reviewed commit: "perf(app): trim oversized attribute valu..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| @@ -0,0 +1,316 @@ | |||
| import { useCallback, useEffect, useState } from 'react'; | |||
Contributor
There was a problem hiding this comment.
This new component is 316 lines, while LLMSessionPanel.tsx is also 302 lines. Both exceed the repository's 300-line maximum, increasing maintenance cost; split them into smaller focused components.
Context Used: AGENTS.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Comment on lines
+77
to
+81
| !isLoading && ( | ||
| <Text size="sm" c="dimmed"> | ||
| No LLM messages found on this span. Prompt and completion capture | ||
| may be disabled in the instrumentation. | ||
| </Text> |
Contributor
There was a problem hiding this comment.
This no-message branch and the no-session-results branch in LLMSessionPanel.tsx render plain Text elements instead of the required shared EmptyState, bypassing the repository's consistent empty-state presentation and behavior. Use @/components/EmptyState for both branches.
Context Used: AGENTS.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Contributor
E2E Test Results✅ All tests passed • 307 passed • 1 skipped • 1241s
Tests ran across 4 shards in parallel. |
Contributor
Deep ReviewRead-time, schema-agnostic LLM observability feature (~7.1k added lines, 58 files) under ✅ No critical issues found. 🟡 P2 — recommended
🔵 P3 nitpicks (3)
Reviewers (3 completed of 13 dispatched): security, reliability, previous-comments — corroborated by direct verification of the diff (file sizes, chart titles, Testing gaps:
|
wrn14897
changed the title
…ion lookup hardening Post-review fixes for the LLM observability branch, driven by a live comparison against opencode's self-reported session cost: the /llm dashboard showed ~$22.72 for a session opencode itself priced at $9.16. Cost accuracy (verified exact against opencode's cost_usd on 111 calls): - Provided-cost election: apps that stamp their own per-call cost (cost_usd / llm.cost.total / gen_ai.usage.cost) are treated as the authoritative reporters, and all token/cost/call aggregations sum only those rows when any exist in scope (llmGatedSumExpr / llmGatedCountExpr, rendered as raw select aggregates). This dedupes dual-instrumented apps — opencode emits OpenInference spans (with cost) AND Vercel AI SDK spans (with gen_ai.usage.*) for every call, in separate traces, so no row-local gate could catch it. - Cache-aware estimation: the SQL cost expression now prices uncached input, cache reads (discounted), cache writes (Anthropic's 1.25x premium, new catalog rate + attribute keys incl. OpenInference prompt_details.cache_write, Vercel inputTokenDetails.cacheWriteTokens, and flat cache_creation_tokens), and output separately, matching the TS-side computeCostUsd. The inclusive/exclusive input-token heuristic now accounts for writes, and totalTokens reports effective context. - Query-size guard: the enriched cost expression embeds the price catalog per token term; the first live run exceeded ClickHouse's 256 KiB max_query_size. Rates are now factored into per-term multiIfs and the whole expression is bound once per query as a WITH expression alias (LLM_COST_SQL_ALIAS), keeping dashboard queries at ~80 KiB. Session drawer correctness (review feedback): - The per-span attribute lookup now pins TraceId alongside SpanId + timestamp (span ids can be empty or collide across traces) and is bounded to the searched window for partition pruning. Also trims the llm module's public surface to what external consumers import (fixes 23 knip unused-export findings) and replaces the unused zod chat-message schemas with plain interfaces. Adds the missing changeset for the LLM observability feature.
The LLM observability branch added 6 eslint-disable comments, tripping the app/eslint-disable ratchet (150 > baseline 144). Remove the escapes by fixing the underlying patterns instead of suppressing them: - Chat messages get a stable `id` assigned in extractConversation (conversations are immutable once extracted), so message lists key on data instead of array indexes. - Session timeline rows carry their accordion `itemValue` in the row data built per fetch, replacing the index-derived key/value pair. - The session filter now lives in the URL only: SessionSelectControlled becomes a plain value/onChange SessionSelect wired straight to the nuqs param, deleting the two deliberately-under-depped form<->URL sync effects (the drawer's "Filter dashboard" action writes the same param). - The trace-source default adoption effect gets full dependencies — the select only offers usable trace sources, so a user selection always resolves to itself and the effect can never fight it.
Adds the standard beta badge (matching Service Map's nav badge) next to the LLM breadcrumb, plus an info hover card explaining what to expect: the dashboard is experimental, which instrumentations it understands, that costs are catalog estimates unless the instrumentation reports its own cost, and how dual-instrumented apps are counted.
Hide the passive LLM surfaces on event/trace views while the feature bakes, mirroring the alert-details flag pattern: default off, enabled in dev via .env.development, opt-in comment in docker-compose.yml. Gated under IS_LLM_PANELS_ENABLED: - the LLM tab in the event row side panel - the LLM section on the row Overview tab - the LLM tab in the trace span detail panel - the model + token-count suffix on waterfall span labels The /llm dashboard and its dashboards-list entry are intentionally not gated — the dashboard is opt-in by navigation and already labeled beta.
Drop all passive LLM surfaces on existing event/trace views so this PR ships only the opt-in dashboard; the side-panel integrations can return in a follow-up: - Restore DBRowSidePanel(+types), DBRowOverviewPanel, DBTracePanel, and DBTraceWaterfallChart to main (removes the LLM tab, overview section, span-detail tab, and waterfall label suffix). - Remove the now-moot IS_LLM_PANELS_ENABLED flag and its env/compose wiring. - Delete the pieces those surfaces orphaned: LLMConversationPanel, lib/rowData (row-data extraction glue), asLLMEvents, their tests, and the @/llm barrels (dashboard code imports defining modules directly). - Reword the changeset to describe the dashboard only. Outside src/llm the branch now only touches AppNav (dashboards-group active state for /llm) and the dashboards-list preset entry.
- Recognize current-registry dotted semconv usage keys (gen_ai.usage.cache_read.input_tokens, gen_ai.usage.reasoning.output_tokens) emitted by GitHub Copilot Chat and newer SDKs, plus copilot_chat.time_to_first_token (ms) for TTFT. - Expand the model price catalog with xAI, DeepSeek, Mistral, Cohere, Qwen, Meta Llama, and Amazon Nova families across bare, OpenRouter, HuggingFace, and Bedrock id flavors. - Add agent attribution (gen_ai.agent.name / agent.name): per-agent calls, tokens, est. cost, and error rate charts on the dashboard and an agent badge in the span subpanel.
Keeps the MIT attribution header in modelPrices.ts intact.
Moves the latency heatmap out of Overview into a dedicated Latency tab that works like the search page's delta mode: drag a region on the duration heatmap to compare the selected spans against the rest and see which attributes differ. Include/exclude clicks on a breakdown value append the condition to the dashboard's where input, scoping every tab. Reuses DBSearchHeatmapChart (URL-persisted selection, settings drawer, delta chart) with the LLM span scope and no cost WITH alias — the sampling queries never reference cost. Overview keeps the p95-by-model and TTFT trend charts.
…kdown DBDeltaChart gains an opt-in isPriorityProperty predicate (threaded through DBSearchHeatmapChart) that pins matching visible properties above the rest while preserving score order within each group. The LLM Latency tab passes isLLMAttributeKey, built from the dashboard's attribute key registries plus the gen_ai/llm/ai/openinference/ copilot_chat namespaces, so model, token, cost, tool, and agent attributes always lead the breakdown. Search page ranking unchanged.
…pling The delta breakdown's sampling queries are row-capped (LIMIT 1000 + stable hash order + PartIds indexHint, inherited from search's delta mode), but they SELECT * — and agent SDKs stamp full conversation histories on every span, so a 1000-row LLM-span sample measured ~412 MiB of SpanAttributes (~800 MiB per selection across the outlier + inlier queries). DBDeltaChart gains an opt-in selectExpression (default '*', threaded through DBSearchHeatmapChart as deltaSelectExpression). The LLM Latency tab passes a trimmed select that drops attribute values longer than 256 chars server-side via mapFilter (~500x smaller: 0.11 MiB vs 55 MiB measured on the same sample) — long values are exactly what the breakdown hides as high-cardinality anyway. Falls back to SELECT * for JSON-typed or derived attribute columns, and holds queries until the JSON-column lookup resolves. Search page behavior unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linear
https://linear.app/clickhouse/issue/HDX-5162/llm-observability-span-chat-view-cost-tracking-sessions-and-llm
Why
Teams running LLM apps and coding agents (OpenAI/Anthropic SDKs, Vercel AI SDK, LangChain via OpenLLMetry/OpenInference, opencode, Claude Code, GitHub Copilot Chat) already send their telemetry to HyperDX — but the product had zero LLM awareness: no chat rendering, no token/cost concepts, no model analytics. This PR adds LLM observability at feature parity with dedicated LLM observability tools while staying HyperDX-native.
Approach: read-time, schema-agnostic
Unlike dedicated LLM observability tools that rely on ingest-time processing and dedicated tables, everything here derives from span/log attribute maps at query time:
Mapand JSON-typed attribute columns via the source'seventAttributesExpressionWhat's included
Normalization lib (
packages/app/src/llm/lib) — pure, unit-tested TS:gen_ai.usage.cache_read.input_tokens), OpenLLMetry, OpenInference, Vercel AI SDK — plus real-world variants captured as fixtures from opencode, Claude Code, and GitHub Copilot Chat telemetry (whole-stringllm.input_messages, camelCaseai.usage.*, flatinput_tokens/cost_usdkeys, bracketed model ids likeclaude-opus-5[1m],copilot_chat.time_to_first_token)Cost estimation (
lib/modelPrices.ts,lib/cost.ts):gen_ai.usage.cost,llm.cost.total,cost_usd) always wins; catalog is the fallbackmultiIfgenerator for dashboard aggregation, bound once per query as aWITHalias to stay well under ClickHouse'smax_query_size/llmpreset dashboard (Overview | Latency | Sessions | Search), listed on the Dashboards page:gen_ai.agent.name): per-agent calls, tokens, est. cost, and error rategen_ai.conversation.id→session.id→ai.telemetry.metadata.sessionId) — the correlation surface for instrumentations that stamp session ids but don't propagate trace context. Row click opens a timeline drawer; each call expands into a normalized chat view with role badges, markdown, collapsible tool calls, and a usage/cost summaryCorrectness & performance notes
gen_ai.usage.*/llm.token_count.*/ flat primary-reporter keys) so SDK wrapper spans (e.g. Vercel'sai.streamTextarounddoStream) don't double countstopvs["stop"])SELECT *sample measured ~412 MiB — attribute values over 256 chars are dropped server-side viamapFilter(~500× smaller), which loses nothing since long values are hidden as high-cardinality anywayKnown limitations
Testing
src/llm/__tests__(per-dialect fixtures lifted from real opencode/Claude Code/Copilot telemetry, cost math and price-catalog matching, SQL expression generation, lazy-loading regression guards)tsc --noEmitclean; eslint 0 errors; knip cleanScreenshots