Skip to content

[HDX-5090] Support creating alerts without saved searches or dashboard tiles (backend) - #3010

Merged
kodiakhq[bot] merged 7 commits into
mainfrom
warren/HDX-5090-detached-alerts-backend
Aug 28, 2026
Merged

[HDX-5090] Support creating alerts without saved searches or dashboard tiles (backend)#3010
kodiakhq[bot] merged 7 commits into
mainfrom
warren/HDX-5090-detached-alerts-backend

Conversation

@wrn14897

@wrn14897 wrn14897 commented Aug 26, 2026

Copy link
Copy Markdown
Member

Summary

Backend foundation for detached alerts (HDX-5090): a new inline alert source that persists its own chart config directly on the alert document, so alerts no longer require a saved search (logs) or a dashboard tile (metrics). This unblocks customers (e.g. Epidemic Sound, migrating 1000+ Grafana alert rules) for whom creating a saved search or dashboard tile per alert does not scale.

The persisted config is the exact shape a dashboard tile stores (SavedChartConfig minus the embedded alert field), so inline alerts evaluate through the same battle-tested code path as tile alerts:

  • common-utils: AlertSource.INLINE, exported AlertChartConfigSchema (builder + raw SQL, no PromQL), zInlineAlert, new AlertSchema union member, and optional chartConfig on AlertsPageItemSchema.
  • Model: chartConfig (Mixed) on the Alert document; makeAlert persists it for inline alerts and clears it when the source changes (mirrors savedSearch/dashboard reference clearing). No migration needed.
  • Internal API: new internalAlertSchema accepts the inline source and validates metric formulas on builder configs (expression parse, series references, ratio exclusivity); validateAlertInput enforces supported display types (Line/Stacked Bar/Number), validates raw SQL templates, and checks team-scoped source/connection ownership — including that a raw-SQL config's source belongs to its connection (compared as ObjectIds). The single-alert response includes chartConfig for edit surfaces; the unpaginated list omits it.
  • check-alerts task: new AlertTaskType.INLINE; the tile-alert config assembly is factored into a shared buildAlertChartConfigFromSavedConfig used by both tile and inline alerts, so group-by, multi-window, formulas, ratio mode, and raw SQL behavior are identical. A raw-SQL source that was moved to a different connection after alert creation has its metadata dropped with a warning (also closes the identical pre-existing tile-alert exposure). Notifications link to the chart explorer seeded with the alert's config over the alerting window, and default their title to the config's name.

Deliberately out of scope (follow-ups):

  • Creation/edit UI (chart explorer create flow; full chart editor on the alert details page)
  • External API v2 + MCP write support — the v2 API keeps the narrower alertSchema and rejects source: 'inline' (guarded by a test) until its OpenAPI/Terraform contract is extended. v2 GETs echo inline alerts read-only.
  • Terraform/IaC: inline alerts are already excluded by isImportableAlert.

How to test on Vercel preview

N/A — non-UI change (backend only; no UI creates inline alerts yet).

Testing done:

  • make ci-lint, make ci-unit — pass
  • Integration: routers/api/alerts.int (62 tests, incl. inline-alert CRUD + validation), full checkAlerts.int (176 tests, incl. end-to-end inline alert evaluation + grouped notification + template link/title), external-api/alerts.int (54 tests, incl. the v2 rejection guard), checkAlerts/providers/default.int (36 tests, incl. stale source-connection scenarios)

References

  • Linear Issue: HDX-5090
  • Related PRs: N/A

@wrn14897 wrn14897 added the ai-generated AI-generated content; review carefully before merging. label Aug 26, 2026
@changeset-bot

changeset-bot Bot commented Aug 26, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 1e7155c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Minor
@hyperdx/api Minor
@hyperdx/app Minor
@hyperdx/otel-collector Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Aug 26, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hyperdx-oss Ready Ready Preview Aug 28, 2026 6:28am
hyperdx-storybook Ready Ready Preview Aug 28, 2026 6:28am

Request Review

@greptile-apps

greptile-apps Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR adds detached inline alerts whose chart configuration is persisted directly on the alert and evaluated through the existing chart-alert pipeline.

  • Adds shared schemas, persistence, internal CRUD support, and source/connection validation for inline builder and raw-SQL alerts.
  • Extends the alert worker and notification links to evaluate and navigate to inline chart configurations.
  • Keeps external API writes on the existing narrower alert contract.

Confidence Score: 4/5

The PR should not merge until equivalent uppercase ObjectIds are accepted consistently by the raw-SQL source/connection validation.

The prior reply claims the ObjectId.equals change accepts uppercase hexadecimal IDs, but BSON 4.7.2 compares a 24-character string against the lowercase hexadecimal representation, so the unchanged raw string remains a concrete counterexample and valid inline-alert writes are rejected.

Files Needing Attention: packages/api/src/controllers/alerts.ts and packages/api/src/tasks/checkAlerts/providers/default.ts

Important Files Changed

Filename Overview
packages/common-utils/src/types.ts Adds the inline alert source and shared chart-config schemas used across the API and worker.
packages/api/src/utils/zod.ts Adds the internal inline-alert request union and formula validation while preserving the narrower external schema.
packages/api/src/controllers/alerts.ts Persists and validates inline alert configurations, but its attempted ObjectId normalization still rejects equivalent uppercase connection IDs.
packages/api/src/tasks/checkAlerts/providers/default.ts Loads inline alert query dependencies and guards stale raw-SQL source metadata, while repeating the noncanonical ObjectId comparison issue.
packages/api/src/tasks/checkAlerts/index.ts Reuses saved chart assembly and evaluation behavior for inline alert tasks.
packages/api/src/routers/api/alerts.ts Enables internal inline-alert CRUD and returns chartConfig from the single-alert response.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Client[Internal alerts API] --> Validate[Validate inline chart config]
  Validate --> Mongo[(Persist alert and chartConfig)]
  Mongo --> Worker[check-alerts worker]
  Worker --> Resolve[Resolve source and connection]
  Resolve --> Query[Render and execute chart query]
  Query --> Evaluate[Evaluate threshold and history]
  Evaluate --> Notify[Send notification with chart explorer link]
Loading

Fix all with Greploop Fix All in Claude Code Fix All in Conductor Fix All in Cursor Fix All in Codex

Reviews (9): Last reviewed commit: "Merge branch 'main' into warren/HDX-5090..." | Re-trigger Greptile

Comment thread packages/common-utils/src/types.ts
Comment thread packages/api/src/controllers/alerts.ts
@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

E2E Test Results

All tests passed • 323 passed • 1 skipped • 1366s

Status Count
✅ Passed 323
❌ Failed 0
⚠️ Flaky 0
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@wrn14897
wrn14897 marked this pull request as ready for review August 27, 2026 05:41
@github-actions github-actions Bot added the review/tier-4 Critical — deep review + domain expert sign-off label Aug 27, 2026
@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Background tasks or delivery pipeline substantially modified — 513 lines (bar: 30):
    • packages/api/src/tasks/checkAlerts/index.ts
    • packages/api/src/tasks/checkAlerts/providers/default.ts
    • packages/api/src/tasks/checkAlerts/providers/index.ts
    • packages/api/src/tasks/checkAlerts/template.ts
  • Cross-layer change: touches backend (packages/api) + shared utils (packages/common-utils)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 10
  • Production lines changed: 803 (+ 979 in test files, excluded from tier calculation)
  • Critical-path lines changed: 513
  • Branch: warren/HDX-5090-detached-alerts-backend
  • Author: wrn14897

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Deep Review

Backend foundation for detached "inline" alerts (AlertSource.INLINE) that persist their own chart config on the alert document and evaluate through the existing tile-alert path. Ten reviewers ran against the diff vs base 0558f77e. All five prior review threads (formula-validation bypass, source/connection divergence, ObjectId comparison, stale-connection worker guard, list-endpoint config leak) were independently verified as genuinely resolved in the current diff. No P0 issues. One P1 external-contract gap; the rest are recommendations and nits.

🔴 P0/P1 — must fix

  • packages/api/src/utils/externalApi.ts:359translateAlertDocumentToExternalAlert copies source: alertObj.source verbatim and never emits chartConfig, so an inline alert (creatable through the live internal API today) is echoed by the v2 GET /alerts/GET /alerts/:id and the MCP getAlert tool with an inline source value that is absent from the shipped OpenAPI AlertSource enum and with all reference fields (dashboardId/savedSearchId/tileId/config) missing.
    • Fix: exclude inline alerts from v2 read responses and their meta.total/X-Total-Count counts until the v2 contract documents the inline source and its config shape, or extend the OpenAPI/ExternalAlert schema to cover it.
    • api-contract, adversarial

🟡 P2 — recommended

  • packages/api/src/tasks/checkAlerts/providers/default.ts:262getInlineAlertDetails duplicates ~70–80 lines of connection/source resolution from getTileDetails, and the copies have already drifted: the raw-SQL skip log reads line/bar in getTileDetails but line/bar/number in getInlineAlertDetails.
    • Fix: extract one shared resolver (parallel to buildAlertChartConfigFromSavedConfig) so future fixes and messages stay in sync across both call sites.
    • maintainability, kieran-typescript
  • packages/api/src/tasks/checkAlerts/providers/default.ts:321 — when a builder inline alert's referenced source (or a raw-SQL inline alert's pinned connection) is deleted after creation, getInlineAlertDetails returns [], loadAlert throws, and the error is only logged; executionErrors/state are never updated, so the alert silently stops evaluating while the UI shows its last state. Detached alerts have no tile/search to remind a user of the dependency, raising the stakes versus the pre-existing tile behavior.
    • Fix: record an executionError (or a distinct orphaned state) on the alert when detail loading fails, or guard source/connection deletion against referencing alerts.
    • adversarial
  • packages/api/src/controllers/alerts.ts:176 — the inline builder branch of validateAlertInput never applies the source-kind formula gate; validateInlineAlertFormulas only checks expression validity, ratio exclusivity, and number-chart count, so a formula on a non-formula source kind persists and then errors on every evaluation tick — the exact failure the formula validation was added to prevent.
    • Fix: in the inline builder branch, reject configs carrying formulas when the resolved source kind does not support formulas.
    • correctness
  • packages/api/src/tasks/checkAlerts/__tests__/checkAlerts.int.test.ts:2866 — new inline behavior lacks coverage: no end-to-end evaluation test for a raw-SQL inline config, and the PUT /:id inline update path exercises only success cases, so a regression that skipped or misapplied validateAlertInput on update would go uncaught.
    • Fix: add an end-to-end raw-SQL inline evaluation test and a PUT test asserting an invalid inline chartConfig returns 400 and leaves the stored alert unchanged.
    • testing
🔵 P3 nitpicks (5)
  • packages/api/src/tasks/checkAlerts/index.ts:725buildAlertChartConfigFromSavedConfig logs Source not found for builder inline alert for both the tile and inline call sites, mislabeling a missing-source tile alert as inline.
    • Fix: include the taskType/source in the log context or pass a call-site label.
  • packages/api/src/controllers/alerts.ts:129 — a raw-SQL inline config whose sqlTemplate uses $__sourceTable but omits the interval macro passes write-time validation, because validateRawSqlForAlert throws during macro expansion (no from present) and skips the interval check; the query then evaluates without bucketing at run time. Pre-existing; also affects raw-SQL tile alerts.
    • Fix: validate macros against a representative from, or run the interval check independently of macro-expansion success.
  • packages/api/src/models/alert.ts:72AlertSource is redeclared here with string values matching @hyperdx/common-utils, kept in sync only by convention (this PR had to edit both). Pre-existing.
    • Fix: import and re-export AlertSource from common-utils, as AlertThresholdType already is.
  • packages/api/src/controllers/alerts.ts:240 — the source-connection ObjectId consistency check is duplicated near-verbatim between write-time validation and the worker's getRawSqlSourceMetadata.
    • Fix: extract a shared sourceBelongsToConnection(source, connectionId) predicate.
  • packages/api/src/tasks/checkAlerts/index.ts:1782 — the AlertChartConfigSavedChartConfig assignability the shared builder relies on holds only because zod infers the alert key as optional, an unstated inference detail a future schema change could silently break.
    • Fix: add a compile-time assignability assertion (e.g. a .test-d.ts) or a comment pinning the invariant.
    • kieran-typescript, maintainability

Reviewers (10): correctness, security, testing, maintainability, project-standards, api-contract, reliability, adversarial, kieran-typescript, previous-comments.

Testing gaps:

  • getInlineAlertDetails defensive branches (null chartConfig, missing connection, unsupported raw-SQL display type, deleted builder source) are all untested silent-skip paths.
  • template.ts inline branches — the null-chartConfig throw in buildAlertMessageTemplateHdxLink and the alert.chartConfig?.name ?? 'chart' title fallback — are unexercised.
  • No test asserts the connection password loaded via .select('+password') is absent from the notification payload and the GET /alerts/:id response; no worker-side test covers a cross-team connection/source reference.
  • No round-trip fidelity test for chartConfig through the Mongoose Mixed field, and no direct coverage of alertHasGroupBy for a raw-SQL inline config.

Comment thread packages/api/src/controllers/alerts.ts Outdated
Comment thread packages/api/src/controllers/alerts.ts
Add a 'chart' alert source that persists its own chart config directly on
the alert document, so alerts no longer require a saved search (logs) or a
dashboard tile (metrics). Builder configs on log/trace/metric sources plus
raw SQL (Line/StackedBar/Number) are supported; PromQL is rejected.

- common-utils: AlertSource.CHART, AlertChartConfigSchema, zChartAlert,
  AlertSchema union member, chartConfig on AlertsPageItemSchema
- model: chartConfig (Mixed) on Alert; makeAlert persists/clears it like the
  other source references
- internal API: internalAlertSchema accepts the new source (external v2
  keeps the narrower alertSchema until its contract is extended);
  validateAlertInput checks display type, raw SQL template, and team-scoped
  source/connection ownership; responses include chartConfig
- check-alerts: new CHART task type evaluated through the same code path as
  tile alerts (shared buildAlertChartConfigFromSavedConfig), including
  group-by and multi-window behavior; notifications link to the chart
  explorer seeded with the alert's config and default their title to the
  config's name

Backend only; the creation/edit UI and external API v2 support land
separately.
… consistency (HDX-5090)

Two write-path gaps in the new chart alert source, both of which would
otherwise persist configs that fail on every evaluation tick:

- Builder configs skipped validateChartConfigFormulas (dashboards get it
  from the editor and the external tile refinement, but chart alerts are
  authored through this API directly). internalAlertSchema now rejects
  malformed formulas, references to nonexistent series, and formulas
  combined with seriesReturnType: 'ratio' (mapped onto the helper's
  external-shape asRatio).

- Raw-SQL configs accepted a team-owned source on a different team-owned
  connection. The worker executes through chartConfig.connection while
  expanding $__sourceTable/metricTables from the source, so a divergent
  pair yields wrong values or repeated query failures. validateAlertInput
  now requires the source to belong to the configured connection.
objectIdSchema admits every representation ObjectId.isValid accepts
(uppercase hex, 12-byte strings) and the Mongo lookups cast them, but the
raw-SQL source/connection consistency check compared strings — so an
equivalent non-canonical connection ID resolved both documents and was
then rejected as a mismatch. Compare via ObjectId.equals instead.
…ves connections (HDX-5090)

Write-path validation ties a raw-SQL chart alert's source to its
connection at save time, but a source can be moved to a different
connection afterwards. The worker executes through the config's pinned
connection while expanding $__sourceTable/metricTables from the source,
so a moved source silently targeted the wrong database (when the same
table exists there) or failed confusingly.

Extract the optional source lookup shared by the tile and chart raw-SQL
branches into getRawSqlSourceMetadata, which now drops (with a warning)
a source whose connection no longer matches the config's. Templates that
don't use source macros keep evaluating correctly; templates that do
fail with a visible query error recorded on the alert instead of
returning wrong values.

This intentionally also covers raw-SQL tile alerts, which had the
identical pre-existing exposure.
'chart' collided with existing vocabulary: ChartAlertBaseSchema and the
alerts-page copy ("created from dashboard charts") both already use
"chart" to mean tile alerts, and the enum's other values name the
referenced entity while the new kind's defining property is that the
definition lives inline on the alert with no referenced entity.

Wire value 'chart' -> 'inline' (unreleased, so no migration), plus the
matching identifiers: AlertSource.INLINE, AlertTaskType.INLINE,
zInlineAlert, getInlineAlertDetails, validateInlineAlertFormulas, and the
test fixtures. Shape-named identifiers keep their names (chartConfig,
AlertChartConfigSchema, buildChartExplorerLink) since they describe the
config's content or link target, not the source kind.
@wrn14897
wrn14897 force-pushed the warren/HDX-5090-detached-alerts-backend branch from b4c939f to 14175ec Compare August 27, 2026 17:00
Comment thread packages/api/src/routers/api/alerts.ts Outdated
GET /alerts is unpaginated, so attaching every inline alert's full chart
config (raw SQL templates included) would bloat every alerts-page load at
the 1000+-rule scale this feature targets, and bake in a collection
contract that couldn't be paginated away later. formatAlertResponse gains
an includeChartConfig flag; only GET /alerts/:id sets it, keeping the full
config on the detail response for edit surfaces. A slim list-side summary
(name/display type) is deferred to the UI PR that will consume it.
Comment on lines +165 to +169
if (
!new Types.ObjectId(String(source.connection)).equals(
chartConfig.connection,
)
) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Uppercase ObjectIds remain rejected

When a raw-SQL inline alert supplies its connection as an equivalent uppercase 24-character ObjectId, ObjectId.equals receives the original uppercase string and compares it against the lowercase hexadecimal representation, causing the API to reject a valid source/connection pair. The same comparison in the worker can discard valid source metadata from an already persisted noncanonical config.

Knowledge Base Used: API services and background tasks

Fix in Claude Code Fix in Conductor Fix in Cursor Fix in Codex

@kodiakhq
kodiakhq Bot merged commit df4a7a5 into main Aug 28, 2026
29 checks passed
@kodiakhq
kodiakhq Bot deleted the warren/HDX-5090-detached-alerts-backend branch August 28, 2026 06:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-generated AI-generated content; review carefully before merging. automerge review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants