Track dropped extension telemetry reports - #9783
Open
Rick Winter (RickWinter) wants to merge 3 commits into
Open
Track dropped extension telemetry reports#9783Rick Winter (RickWinter) wants to merge 3 commits into
Rick Winter (RickWinter) wants to merge 3 commits into
Conversation
Rick Winter (RickWinter)
requested review from
JeffreyCA,
Marina He (hemarina),
Richard Park (richardpark-msft),
tg-msft and
Victor Vazquez (vhvb1989)
as code owners
August 28, 2026 15:39
|
Azure Pipelines: Successfully started running 1 pipeline(s). 20 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Copilot review overview
Review tier: Balanced
Findings: 2
New issues introduced by this change (3)
| Severity | Finding |
|---|---|
 |
cli/azd/internal/grpcserver/telemetry_service.go — This branch cannot observe real unauthenticated RPCs: the server-wide tokenAuthInterceptor… |
|
cli/azd/internal/grpcserver/telemetry_service.go — This records the raw extension ID before the official-source gate, including for… |
 |
cli/azd/internal/grpcserver/telemetry_service.go — This count goes into the process-global usage bag. For azd up,… |
What changed in this PR
Adds command-span telemetry for extension usage reports rejected by the host.
Changes:
- Adds bounded drop reasons and counts.
- Tests validation and operational drop paths.
- Documents telemetry contracts, diagnostics, and privacy.
| File | Description |
|---|---|
docs/specs/metrics-audit/telemetry-schema.md |
Defines the new telemetry fields. |
docs/specs/metrics-audit/feature-telemetry-matrix.md |
Updates extension telemetry coverage. |
docs/reference/telemetry-data.md |
Documents public telemetry fields. |
docs/architecture/adr-001-extension-telemetry-events.md |
Records the aggregation design. |
cli/azd/internal/tracing/fields/fields.go |
Declares drop attributes. |
cli/azd/internal/grpcserver/telemetry_service.go |
Records drop reasons and counts. |
cli/azd/internal/grpcserver/telemetry_service_test.go |
Tests drop behavior. |
cli/azd/docs/extensions/extension-telemetry.md |
Adds diagnostic guidance and queries. |
cli/azd/cmd/telemetry_test.go |
Verifies field keys. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
There was a problem hiding this comment.
Copilot review overview
Review tier: Balanced
Findings: 2
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
 |
docs/reference/telemetry-data.md — The preceding sentence says unofficial and over-budget reports “record nothing,” but this new… |
|
docs/specs/metrics-audit/telemetry-schema.md — The Eligibility and Volume rows immediately above still say these calls are dropped “without… |
Issues resolved since last review (3)
| Severity | Finding |
|---|---|
|
cli/azd/internal/grpcserver/telemetry_service.go — This count goes into the process-global usage bag. For azd up,… View resolved comment |
|
cli/azd/internal/grpcserver/telemetry_service.go — This records the raw extension ID before the official-source gate, including for… View resolved comment |
|
cli/azd/internal/grpcserver/telemetry_service.go — This branch cannot observe real unauthenticated RPCs: the server-wide tokenAuthInterceptor… View resolved comment |
Contributor
There was a problem hiding this comment.
Copilot review overview
Review tier: Balanced
Findings: 1
New issues introduced by this change (3)
| Severity | Finding |
|---|---|
|
cli/azd/internal/cmd/up_graph.go — Filtering only the synthetic azd up spans does not keep these fields command-span-only.… |
|
docs/specs/metrics-audit/telemetry-schema.md — This table now includes drop observability, but the Purpose rule at line 256 still says every… |
|
docs/specs/metrics-audit/feature-telemetry-matrix.md — The matrix says every drop records an extension ID, but the implementation deliberately uses… |
Issues resolved since last review (2)
| Severity | Finding |
|---|---|
|
docs/specs/metrics-audit/telemetry-schema.md — The Eligibility and Volume rows immediately above still say these calls are dropped “without… View resolved comment |
|
docs/reference/telemetry-data.md — The preceding sentence says unofficial and over-budget reports “record nothing,” but this new… View resolved comment |
Comment on lines
+807
to
+810
| return slices.DeleteFunc(tracing.GetUsageAttributes(), func(attr attribute.KeyValue) bool { | ||
| return attr.Key == fields.ExtensionUsageDropped.Key || | ||
| attr.Key == fields.ExtensionUsageDroppedCount.Key | ||
| }) |
| | Drop observability | Rejected and dropped calls append one unique `<extension-id-or-unattributed>@<reason>` value to `extension.usage.dropped` on the hosting command span and increment `extension.usage.dropped.count`. IDs appear only after official-source admission; earlier failures use fixed `unattributed`. Synthetic `azd up` phase spans do not copy these fields. Reasons are fixed by the host, and no caller-controlled event or attribute content is copied | | ||
| | Values | Not enumerated or pattern-checked. The extension author owns what a value means and is responsible for keeping it low cardinality and free of customer content | | ||
| | Classification | Always `SystemMetadata` | | ||
| | Purpose | Always `FeatureInsight` | |
| | **Up-graph performance** | `up` (graph execution) | (none — enriches the `up` command span) | `perf.provision_duration_ms`, `perf.deploy_duration_ms`, `perf.total_duration_ms` | Emitted from `internal/cmd/up_graph.go` after the graph completes; provision/deploy durations set only when those phases run | | ||
| | **VS RPC** | `vs-server` long-running session | `vsrpc.*` (event prefix) | Per-RPC attributes documented in `telemetry-schema.md` | Long-running RPC server for VS integration | | ||
| | **Extension telemetry service** | Extension calls `ReportUsage` over the extension gRPC API | `ext.usage` | `extension.id`, `extension.version`, `extension.source`, `extension.event`, plus one `ext.<key>` attribute per entry in the caller's attribute map | Telemetry requires no separate capability or declaration. Only extensions whose configured source matches the verified official `azd` registry name, type, and normalized URL are recorded — a call from any other source succeeds but is dropped, as is any call past 100 recorded events per `azd` invocation. Identity fields are derived from host-signed claims and the installed record, never from the request. Every caller key is prefixed with `ext.` so it cannot overwrite a host field, and the host bounds count and length only — it does not enumerate or pattern-check values | | ||
| | **Extension telemetry service** | Extension calls `ReportUsage` over the extension gRPC API | `ext.usage`; drop aggregates enrich the command span | Accepted reports: `extension.id`, `extension.version`, `extension.source`, `extension.event`, plus one `ext.<key>` per caller attribute. Dropped reports: `extension.usage.dropped`, `extension.usage.dropped.count` | Telemetry requires no separate capability or declaration. Only extensions whose configured source matches the verified official `azd` registry name, type, and normalized URL are recorded. Calls from other sources and calls past the 100-event budget are dropped. All rejected and dropped paths append a unique `<extension-id>@<reason>` entry and increment a total count on the command span. Reasons are a fixed host enum, so caller content never enters the drop signal. Identity fields are derived from host-signed claims and the installed record, never from the request. Every caller key is prefixed with `ext.` so it cannot overwrite a host field, and the host bounds count and length only; it does not enumerate or pattern-check values | |
Contributor
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|
Victor Vazquez (vhvb1989)
approved these changes
Aug 31, 2026
Marina He (hemarina)
requested changes
Sep 1, 2026
Marina He (hemarina)
left a comment
Marina He (hemarina)
left a comment
Contributor
There was a problem hiding this comment.
Thanks for adding bounded observability for dropped extension telemetry. The overall privacy design looks sound. Pre-admission failures use the fixed unattributed identity, extension IDs enter the signal only after official-source admission, and reasons are fixed host-defined enums. Rejected caller-provided event names, keys, values, sources, and unverified IDs are not copied.
The new extension.usage.dropped and extension.usage.dropped.count properties should not create Unclassified entries in the GDPR telemetry catalog. Both are exported AttributeKey definitions classified as SystemMetadata / PerformanceAndHealth, and the count is marked as a measurement. This assumes official extension IDs remain registry-controlled system metadata. The existing extension admission and privacy-review process must also continue to prevent accepted dynamic ext.* attributes from carrying PII or Customer Content.
The unresolved VS RPC accounting issue should be fixed before merging. The drop aggregate is stored in the process-global usage bag, and the pre-existing vsrpc/server.go path copies that bag onto every completed vsrpc.* span. Because the bag persists for the lifetime of azd vs-server and the count is cumulative, a single drop is repeated on later unrelated RPC spans.
Since every span is exported to the requests table—and VS RPC spans have separate operation_Id values—this inflates both documented queries:
- Summing
extension.usage.dropped.countovercounts dropped reports. - Counting distinct
operation_Idvalues overcounts affected invocations.
This contradicts the documentation stating that the fields appear only on the hosting command span. The PR already filters them from synthetic azd up spans to prevent equivalent duplication. Please apply the same exclusion to VS RPC spans, preferably through a shared filter, and add sequential and concurrent RPC regression coverage.
Please also resolve the remaining documentation inconsistencies:
- The schema says extension telemetry is always
FeatureInsight, while these fields are declared asPerformanceAndHealth. - The feature matrix uses
<extension-id>@<reason>, although pre-admission failures use<extension-id-or-unattributed>@<reason>.
The VS RPC issue affects telemetry accuracy rather than GDPR handling. With that duplication fixed and the documentation synchronized, the approach looks sound.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extension telemetry reports can be rejected without leaving a signal that explains the missing data. This change records those drops as a bounded aggregate on the command span.
<extension-id-or-unattributed>@<reason>values inextension.usage.droppedextension.usage.dropped.countmeasurementazd upphase spansThe aggregate stays bounded to admitted extension IDs, the fixed
unattributedidentity, and the fixed reason set. Repeated failures increase the count without adding duplicate dimension values. Accepted report behavior and the existing event budget remain unchanged.Fixes #9527.
Testing
mage preflightTelemetry Change Checklist
New Fields
fields/fields.godocs/specs/metrics-audit/telemetry-schema.mdunattributedidentity, and fixed host-defined reasonsNew Events
Privacy
CustomerContentemitted in telemetryTesting
azd upphase spans exclude drop fieldsDownstream
Documentation