Skip to content

Add azure.ai.loom extension for Foundry experiment tracking - #9795

Open
HarshaVardhanBabu (babu-namburi) wants to merge 8 commits into
Azure:mainfrom
babu-namburi:feature/experiment-tracking-cli
Open

Add azure.ai.loom extension for Foundry experiment tracking#9795
HarshaVardhanBabu (babu-namburi) wants to merge 8 commits into
Azure:mainfrom
babu-namburi:feature/experiment-tracking-cli

Conversation

@babu-namburi

@babu-namburi HarshaVardhanBabu (babu-namburi) commented Aug 31, 2026

Copy link
Copy Markdown

Fixes #9796

Summary

  • Add a standalone first-party azure.ai.loom extension with all 18 Foundry experiment-tracking API surfaces under azd ai loom run.
  • Organize operations into run inspection, trace, span, ingest, and wandb command groups.
  • Preserve the existing azure.ai.projects command and provider behavior by removing experiment commands from that extension.
  • Resolve the Foundry project endpoint from an explicit flag, the active azd environment, compatible projects context, or host environment variables.
  • Use Azure Developer CLI bearer authentication by default, with optional AZURE_AI_PROJECT_API_KEY authentication.
  • Add extension metadata, documentation, CODEOWNERS, lint workflow, Azure DevOps release pipeline wiring, and an end-to-end PowerShell smoke-test script.

CLI hierarchy

azd ai loom run
├── list
├── history-keys
├── summary
├── metrics
├── system-metrics
├── logs
├── log-records
├── compare
├── trace
│   ├── list
│   ├── show
│   └── chat
├── span
│   └── query
├── ingest
│   ├── metrics
│   ├── logs
│   ├── traces
│   └── agent-traces
└── wandb
    ├── graphql
    └── file-stream

Validation

  • go build ./... and go test ./... from cli/azd/extensions/azure.ai.loom.
  • go test ./... from cli/azd/extensions/azure.ai.projects.
  • golangci-lint run ./... and extension cspell checks.
  • mage checkDependencyVersions.
  • azd x build for azure.ai.loom, including artifact packaging and local installation.
  • Exercised all 18 APIs against the fdp-command-job-westus2-proj project using project-key authentication.
  • Verified bearer-token request handling through unit tests.

Smoke testing

Run the PowerShell smoke-test script from the Loom extension directory:

.\test-all.ps1 `
  -ProjectEndpoint "https://<account>.services.ai.azure.com/api/projects/<project-id>" `
  -RunId "<run-id>" `
  -SecondRunId "<second-run-id>" `
  -TraceId "<trace-id>"

The script builds and installs the extension, generates temporary synthetic OTLP protobuf, agent-trace, GraphQL, and W&B file-stream payloads, exercises every CLI surface, and removes the fixtures afterward. Use -SkipWriteOperations to run only inspection, trace, and span checks.

Release handoff

This PR adds repository build and release wiring for azure.ai.loom. Publishing the extension to the official registry remains a separate release/registry step.

Add authenticated CLI coverage for run inspection, agent traces, OTLP ingestion, and W&B compatibility APIs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
7 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@microsoft-github-policy-service

Copy link
Copy Markdown
Contributor

Thank you for your contribution HarshaVardhanBabu (@babu-namburi)! We will review the pull request and get back to you soon.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Foundry experiment-tracking operations to the azure.ai.projects extension.

Changes:

  • Adds 18 run, ingestion, trace, span, and W&B API commands.
  • Adds bearer/API-key authentication and Foundry request handling.
  • Adds tests and user documentation.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
README.md Documents experiment-tracking usage.
internal/exterrors/codes.go Adds experiment error codes.
internal/experimenttracking/client.go Implements the authenticated HTTP client.
internal/experimenttracking/client_test.go Tests client behavior.
internal/cmd/root.go Registers new command groups.
internal/cmd/experiment.go Implements experiment commands.
internal/cmd/experiment_test.go Tests command helpers and registration.
docs/environment-variables.md Documents API-key authentication.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread cli/azd/extensions/azure.ai.projects/internal/cmd/experiment.go Outdated
Comment thread cli/azd/extensions/azure.ai.projects/internal/cmd/experiment.go
Comment thread cli/azd/docs/environment-variables.md Outdated
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move experiment tracking commands under azd ai loom run while preserving the existing azure.ai.projects command surface.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babu-namburi HarshaVardhanBabu (babu-namburi) changed the title Add Foundry experiment tracking CLI commands Add azure.ai.loom extension for Foundry experiment tracking Sep 1, 2026
Exercise every azd ai loom run surface with configurable project, run, trace, and payload inputs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Create temporary OTLP protobuf and JSON fixtures so the smoke test exercises ingestion and W&B APIs without caller-provided payload files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babu-namburi

Copy link
Copy Markdown
Author
image

@babu-namburi
HarshaVardhanBabu (babu-namburi) marked this pull request as ready for review September 1, 2026 10:16
Copilot AI review requested due to automatic review settings September 1, 2026 10:16
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
5 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated 5 comments.

Comment thread cli/azd/extensions/azure.ai.loom/internal/cmd/run.go Outdated
Comment thread cli/azd/extensions/azure.ai.loom/internal/cmd/run.go Outdated
Comment thread cli/azd/extensions/azure.ai.loom/internal/experimenttracking/client.go Outdated
Comment thread cli/azd/extensions/azure.ai.loom/internal/cmd/project_resolver.go Outdated
Comment thread cli/azd/docs/environment-variables.md
Preserve authoritative run targeting, surface OTLP partial success, escape dot segments, and complete endpoint fallback documentation and coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings September 1, 2026 10:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated 3 comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

cli/azd/extensions/azure.ai.loom/internal/cmd/project_endpoint.go:28

  • url.Parse errors include the original URL, so a malformed endpoint containing user info or a sensitive query value can echo that secret in this structured error. Return a generic parse failure here; the existing suggestion already tells the user how to correct it.
			fmt.Sprintf("invalid project endpoint URL: %s", err),

cli/azd/extensions/azure.ai.loom/internal/cmd/run.go:337

  • strconv.ParseFloat accepts NaN and infinities, and each bypasses this ordering check. The request then always fails in json.Marshal as an unsupported value and is reported as an internal request error; reject non-finite --min/--max values as invalid parameters before building the body.
			if maxStep < minStep {
				return invalidExperimentParameter("max", "--max must be greater than or equal to --min")
			}

Comment thread cli/azd/extensions/azure.ai.loom/internal/experimenttracking/client.go Outdated
Comment thread cli/azd/extensions/azure.ai.loom/README.md Outdated
Comment thread cli/azd/extensions/azure.ai.loom/internal/cmd/run.go
Prevent credential-bearing redirects, harden validation errors, and add command-level HTTP contract coverage for every experiment tracking surface.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings September 2, 2026 04:52
@babu-namburi

Copy link
Copy Markdown
Author

Addressed the latest review feedback in c067118:

  • disabled redirects for authenticated Loom HTTP clients to prevent credential forwarding;
  • documented the host-shell AZURE_AI_PROJECT_ENDPOINT compatibility fallback;
  • added command-level HTTP contract coverage for all 18 API surfaces;
  • made malformed endpoint parse errors non-disclosing; and
  • rejected NaN and infinite comparison bounds before JSON serialization.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Endpoint path safety and cancellation classification issues remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

cli/azd/extensions/azure.ai.loom/internal/exterrors/errors.go:72

  • A canceled request reaches this fallback and is converted to an internal structured error, which removes the context.Canceled chain and makes Ctrl+C appear as an unexpected failure. Check cancellation before this fallback and return a user-cancellation error, as azure.ai.routines/internal/exterrors/errors.go:93-95 does.
  • Files reviewed: 32/33 changed files
  • Comments generated: 2
  • Review effort level: Balanced

Comment on lines +55 to +56
path := strings.TrimRight(endpoint.EscapedPath(), "/")
return fmt.Sprintf("https://%s%s", host, path), nil
return exterrors.Validation(
exterrors.CodeInvalidExperimentPayload,
message,
"provide a valid JSON object using the documented request schema",
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

customer-reported identify a customer issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add azure.ai.loom CLI support for Foundry experiment tracking APIs

2 participants