Onboard build failure analysis workflow

[YuliiaKovalova]

Summary

• onboard Arcade's build-failure analysis agent and automatic/manual agentic workflows
• adapt Azure Pipelines discovery to Runtime's runtime definition (129) and download only exact failed/canceled-job Logs_Build_* artifacts
• use Runtime's PAT pool, structured safe-output metadata, deterministic PR targeting, and a digest-pinned binlog-mcp container

Validation

• compiled both workflows with Runtime's pinned gh-aw v0.86.2 using --validate
• exercised failed-job/artifact matching against public Runtime build 1561838; selected 9 exact failed-job build artifacts without the successful NativeAOT_Libraries prefix collisions

Example

The example of the output given by the workflow:
dotnet/arcade#17348 (comment)
microsoft/testfx#10637 (comment)

Full fork E2E run

• Build Failure Analysis #32471218479 completed successfully: binlog fetch, activation, Copilot agent, threat detection, safe outputs, and conclusion.
• The workflow posted a root-cause analysis and an inline code suggestion on the mirrored fork PR.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
15 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR adds an agentic GitHub Actions workflow pair for analyzing Azure Pipelines (runtime, definitionId 129) build failures by downloading the failed/canceled legs’ Logs_Build_* artifacts, extracting *.binlog, and delegating analysis to a repository-scoped build-failure analyst agent that reads the binlogs via the binlog-mcp MCP server and posts results via safe-outputs.

Changes:

• Introduces a shared prompt body and an agent playbook for consistent build-failure analysis behavior.
• Adds two workflows: an automatic check_run-triggered workflow and a maintainer /analyze-build-failure slash-command workflow, both reusing failed-job binlogs instead of rebuilding.
• Pins the binlog-mcp container digest in the actions lock data and includes the compiled .lock.yml output.

Reviewed changes

Copilot reviewed 5 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
.github/workflows/shared/build-failure-analysis-shared.md	Shared agent prompt body imported by the two workflows.
.github/workflows/build-failure-analysis.md	Automatic workflow that fetches failed/canceled job binlogs from ADO and runs the analysis agent with safe-outputs.
.github/workflows/build-failure-analysis.lock.yml	gh-aw compiled lock workflow (generated) with pinned actions/containers and emitted jobs.
.github/workflows/build-failure-analysis-command.md	Slash-command workflow to rerun analysis on a PR’s latest failed build.
.github/aw/actions-lock.json	Adds a pinned container digest entry for the binlog-mcp image.
.github/agents/build-failure-analyst.agent.md	Adds the reusable build-failure analyst agent playbook used by the workflows.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/workflows/build-failure-analysis.md:130

• mcp-servers.binlog-mcp.allowed: ["*"] grants the agent access to every tool the binlog-mcp MCP server exposes. Even with a digest-pinned container, this is a broader capability surface than necessary and makes future binlog-mcp tool additions automatically callable by the agent. Consider narrowing this allowlist to only the specific binlog_* tools the analyst is expected to use (and/or a supported wildcard that only matches binlog_*).

  binlog-mcp:
    container: "mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-binlog-mcp-amd64"
    mounts:
      - "/tmp/binlogs:/data/binlogs:ro"
    allowed: ["*"]

[Copilot]

Pull request overview

Copilot reviewed 5 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (2)

.github/workflows/build-failure-analysis.md:519

• The per-artifact binlog staging counter i is only incremented on a successful cp. If one cp fails and the artifact contains multiple *.binlog files, the next file will reuse the same destination name (e.g. ${ai}_0_...) and can overwrite a previously staged binlog, contradicting the "destinations unique" comment and potentially hiding the real failing leg.

# on `needs.fetch-binlog.outputs.binlog-found == 'true'`, these only run once
# binlogs have been retrieved from the failed Azure DevOps build.
steps:
  - name: Download analysis artifact
    uses: actions/download-artifact@v8.0.1

.github/workflows/build-failure-analysis-command.md:460

• The per-artifact binlog staging counter i is only incremented when cp succeeds. If an artifact contains multiple *.binlog files and an early copy fails, subsequent files can reuse the same ${ai}_${i}_... destination name and overwrite staged data, which can make the analysis incomplete or misleading.

            # Info-ZIP prepends warnings on STDOUT for a recoverable archive,
            # and a multi-line value would still pass the `grep -qE` check
            # below, since `grep -q` matches if ANY line matches. `timeout`
            # bounds a hostile archive; pipefail + fail-closed because a killed
            # probe's partial output can end in a numeric column and undercount.

[YuliiaKovalova]

Addressed both review comments in b785d9c by using sanitized artifact names for every workflow-command warning. A subsequent fork E2E run also exposed and fixed the generated safe-output target expression in 0e84cd0. Full E2E proof: https://github.com/YuliiaKovalova/runtime/actions/runs/32471218479

[Copilot]

Pull request overview

Copilot reviewed 5 out of 7 changed files in this pull request and generated 2 comments.

[YuliiaKovalova]

🤖 This update was generated with GitHub Copilot.

Addressed the Runtime artifact-layout feedback in e3024a72fdb:

• Keeps binlog-mcp as the primary analyzer and adds digest-pinned lewing/helix.mcp v0.8.0 as a read-only Azure DevOps compile-log fallback. Only azdo_timeline, azdo_search_timeline, azdo_search_log, azdo_log, and azdo_artifacts are exposed; Helix/test diagnosis remains out of scope.
• Matches Logs_Build_* artifacts exact-first, then accepts a prefix only when it identifies one timeline job. This covers Runtime's display-only monointerpreter, minijit, and llvmaot suffixes without confusing similarly named jobs.
• Runs analysis even when a failed compile task has no matching binlog, inventories failed timeline tasks, and uses bounded hlx evidence for anything not explained by binlogs or complete timeline diagnostics.
• Makes per-artifact staging all-or-nothing, fixes destination reuse after copy failures, and revalidates both PR head and merge revisions before analysis and immediately before safe-output writes.

Fork E2E: https://github.com/YuliiaKovalova/runtime/actions/runs/33157134057

The run replayed Runtime build 1561838 and selected 15 of 68 build-log artifacts (previous exact matching selected 9), including all six display-only suffix cases. It staged 74 binlogs from all 15 artifacts, used hlx timeline evidence for the unmatched Build managed test components/compile failures, and posted exactly one summary plus one review suggestion pinned to commit d297327fa0b20bbabed8b4ee203885303dadab70:

• Summary: E2E target for build failure analysis YuliiaKovalova/runtime#1 (comment)
• Pinned suggestion: E2E target for build failure analysis YuliiaKovalova/runtime#1 (comment)

Both source workflows compile cleanly with gh aw compile --strict --validate on gh-aw v0.86.2. The two follow-up review comments about stale activation documentation are fixed and resolved in the same head.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

.github/workflows/build-failure-analysis.md:487

• The PIPESTATUS checks after unzip -Z1 | grep -qE ... can misclassify the "suspicious path" case as an unzip failure. When grep -q matches early it closes the pipe and unzip may exit non-zero due to SIGPIPE; checking zscan_rc[0] first will then hit the "could not list" branch instead of the intended "suspicious entry path" branch.

            if [ "${zscan_rc[0]}" -ne 0 ]; then
              echo "::warning::Skipping ${safe_name}: could not list archive entries (unzip -Z1 rc=${zscan_rc[0]})."; continue
            fi
            if [ "${zscan_rc[1]}" -eq 0 ]; then
              echo "::warning::Skipping ${safe_name}: archive has a suspicious (absolute or ..) entry path."; continue

.github/workflows/build-failure-analysis-command.md:541

• The PIPESTATUS checks after unzip -Z1 | grep -qE ... can misclassify the "suspicious path" case as an unzip failure. When grep -q matches early it closes the pipe and unzip may exit non-zero due to SIGPIPE; checking zscan_rc[0] first will then hit the "could not list" branch instead of the intended "suspicious entry path" branch.

            if [ "${zscan_rc[0]}" -ne 0 ]; then
              echo "::warning::Skipping ${safe_name}: could not list archive entries (unzip -Z1 rc=${zscan_rc[0]})."; continue
            fi
            if [ "${zscan_rc[1]}" -eq 0 ]; then
              echo "::warning::Skipping ${safe_name}: archive has a suspicious (absolute or ..) entry path."; continue

[Copilot]

Pull request overview

Copilot reviewed 5 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

.github/workflows/build-failure-analysis.md:56

• This workflow currently has no fork guard. Other agentic workflows in this repo use a root-level if: (!github.event.repository.fork) to avoid running (and attempting to use the PAT pool / safe-outputs writeback) in forks. Adding the fork guard here prevents unexpected runs/costs/failures when someone enables the workflow in a fork.

# A Runtime compile failure does not always publish a `Logs_Build_*` artifact.
# Activate once the failed build is verified; `binlog-mcp` handles available
# binlogs and `hlx` provides the bounded Azure DevOps task-log fallback.
if: needs.fetch-binlog.outputs.analysis-ready == 'true'

[Copilot]

Pull request overview

Copilot reviewed 5 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

.github/workflows/build-failure-analysis-command.md:224

• The comment says to keep this logic in sync with on.command.name, but this workflow’s trigger config uses on.slash_command.name (and it’s in the frontmatter, not “below”). This mismatch makes future maintenance error-prone when renaming the command.

          # KEEP IN SYNC with `on.command.name` below.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 7 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

.github/workflows/build-failure-analysis-command.md:131

• This comment says activation requires binlog-found == 'true', but the generated workflow actually gates activation on analysis-ready == 'true' (binlog download can be absent, with hlx as the fallback). This is misleading for troubleshooting why the agent ran without staged binlogs.

    # `pre_activation` remains the authoritative role + command-position check,
    # and `activation` additionally requires `binlog-found == 'true'`.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 7 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

.github/workflows/build-failure-analysis.md:482

• The comment says anything left in /tmp/binlogs from an earlier run would otherwise be uploaded and attributed to this build, but the cleanup only removes *.binlog. Any other leftover files would still be uploaded with the artifact. Consider clearing the directory contents instead of only binlogs to match the stated invariant.

          # Only binlogs extracted by this run may be analyzed. Anything left in
          # the directory by an earlier run on the same runner would otherwise be
          # uploaded and attributed to this build.
          rm -f /tmp/binlogs/*.binlog

.github/workflows/build-failure-analysis-command.md:537

• The comment says anything left in /tmp/binlogs from an earlier run would otherwise be uploaded and attributed to this build, but the cleanup only removes *.binlog. Any other leftover files would still be uploaded with the artifact. Consider clearing the directory contents instead of only binlogs to match the stated invariant.

          # Only binlogs extracted by this run may be analyzed. Anything left in
          # the directory by an earlier run on the same runner would otherwise be
          # uploaded and attributed to this build.
          rm -f /tmp/binlogs/*.binlog

[PureWeen]

This silently disables automatic analysis for fork-origin PRs. GitHub check runs for fork heads can have check_run.pull_requests=[], so this produces an empty PR number, analysis-ready=false, and no safe-output target even though the workflow declares roles: all. Please derive the PR number from the fetched Azure Pipelines metadata (triggerInfo["pr.number"] or refs/pull/<N>/merge), bind it to the event-owned check_run.head_sha via triggerInfo["pr.sourceSha"], retain the current PR/head validation, expose it from fetch-binlog, and use that output for both safe-output targets.

Note

This review comment was generated by GitHub Copilot.

[YuliiaKovalova]

Fixed in face01d.

fetch-binlog now falls back to the build's own PR metadata when check_run.pull_requests is empty: triggerInfo["pr.number"], and failing that the refs/pull/<N>/merge sourceBranch. That value is not event-owned, so it is bound to the one value the event does own — the build must report triggerInfo["pr.sourceSha"] == check_run.head_sha, otherwise the job refuses. Since BUILD_ID itself comes from the event's details_url, a build can only ever resolve to the PR whose head the event is already about. The existing sourceBranch == refs/pull/<PR>/merge, definition-id, and head/merge staleness checks all still run on top of it.

Both safe-output targets now read needs.fetch-binlog.outputs.pr-number instead of re-deriving check_run.pull_requests[0], which is what left fork PRs with an empty target. Both the agent and safe_outputs jobs already declare needs: fetch-binlog, so the expression resolves in each.

Verified by extracting the generated block from the lock and driving it through 7 scenarios: pr.number match, pr.number-absent merge-ref fallback, sourceSha mismatch, missing triggerInfo, empty check head SHA, non-merge sourceBranch, and same-repo (event PR still wins, no ADO lookup).

Note

This reply was generated by GitHub Copilot.

[PureWeen]

This concurrency group is acquired for every created or edited issue comment before the slash command is matched. As a result, an unrelated comment can cancel an active /analyze-build-failure run; the new run then fails command activation and leaves no replacement analysis. Please set cancel-in-progress: false here and handle genuine duplicate commands through idempotency instead of broad event-level cancellation.

Note

This review comment was generated by GitHub Copilot.

[YuliiaKovalova]

Fixed in face01d, though cancel-in-progress: false on its own turned out not to be enough.

GitHub evicts a pending run from a concurrency group regardless of that setting, so with every comment still sharing the PR-scoped group an unrelated comment would discard a queued /analyze-build-failure instead of a running one — the same silent no-result outcome. The workflow's own add-comment safe output, posted near the end of its run, was enough to trigger it.

So the group is narrowed as well: non-command comments are routed to build-failure-analysis-cmd-run-<run_id>, which collides with nothing — the same trick build-failure-analysis.md already uses for non-runtime check runs. Only comments that look like the command share the PR group, and cancel-in-progress is false so a later invocation can't abort an analysis in flight.

Genuine duplicates are handled by idempotency as suggested: add-comment is capped at 1 with hide-older-comments, so a rerun replaces the previous summary rather than stacking on it.

Note

This reply was generated by GitHub Copilot.

[PureWeen]

This pre-gate does not match the generated activation predicate. awk 'NF {print $1; exit}' skips leading spaces and blank lines, but the pinned gh-aw matcher tests the untrimmed comment and requires the command at byte zero. Such a comment passes this gate and pays for the ADO artifact download, then is guaranteed to be rejected during activation. Please make this predicate exactly match the generated slash-command predicate and update the explanatory comment accordingly.

Note

This review comment was generated by GitHub Copilot.

[YuliiaKovalova]

Confirmed and fixed in face01d.

Checked against the pinned sources (github/gh-aw-actions@6aab9e5): check_command_position.cjs passes the raw context.payload.comment.body to resolveMatchedCommand, and its text.trimStart() only builds the log/denial message — it is not part of the match. So the command really does have to sit at byte zero.

The awk 'NF {print $1; exit}' pre-gate is replaced with a case over the unmodified body (no tr -d '\r', no leading-whitespace skip), and the job-level if: moves from contains() to startsWith() so a mere mention no longer spins up a runner. The explanatory comment is rewritten accordingly.

The remaining [[:space:]] in the pattern is a deliberate, documented superset: gh-aw's effective predicate is that regex intersected with the generated activation expression (exact / + ' ' / + '\n'), and a hand-written exact copy would start silently rejecting real commands if gh-aw ever widened its set — whereas a superset only ever costs a wasted download.

Verified by extracting the gate from the regenerated lock and running it against the real slash_command_matcher.cjs over 12 inputs (exact, +CRLF, +LF, with-arg, tab-arg, /analyze-build-failure-now, leading-space, leading-newline, mention, quoted, /other, empty). They agree on all 12; the old gate disagreed on leading-space and leading-newline.

Note

This reply was generated by GitHub Copilot.

[Copilot]

🔵 Needs a closer look

Multiple moderate workflow correctness, validation, fallback, and safe-output issues remain unresolved.

Review details

Suppressed comments (15)

Previously missed (4) — in code that hasn't changed since the last review.

.github/workflows/build-failure-analysis-command.md:471

• The job/artifact identity is made lossy before mapping: lowercasing, deleting all non-alphanumeric characters, and deduplicating collapses distinct timeline jobs such as Foo-Bar and Foo_Bar to foobar. The later artifact loop can then treat both artifacts as the same job, so if one is failed it may download the successful job's binlog too; preserve exact names and only use a normalized/prefix key when its identity is unique.
  .github/workflows/build-failure-analysis.md:407
• The job/artifact identity is made lossy before mapping: lowercasing, deleting all non-alphanumeric characters, and deduplicating collapses distinct timeline jobs such as Foo-Bar and Foo_Bar to foobar. The later artifact loop can then treat both artifacts as the same job, so if one is failed it may download the successful job's binlog too; preserve exact names and only use a normalized/prefix key when its identity is unique.
  .github/workflows/build-failure-analysis-command.md:154
• activation does not require binlog-found: the top-level if and generated lock gate on analysis-ready, while binlog-found only controls artifact upload/download. As written, this comment says the valid hlx-only path is disabled; update it to describe the actual gate.

This issue also appears on line 535 of the same file.
.github/workflows/build-failure-analysis.md:476

• The guard is now MAX_ZIP_BYTES=2147483648 (2 GB), but this explanation still says the per-artifact cap is 500 MB and that an ordinary build trips it. That is materially misleading for operators assessing disk/network exposure and for anyone deciding whether a skipped artifact should be expected; update the comment to describe the actual 2 GB limit and fallback behavior.

.github/workflows/build-failure-analysis-command.md:802

• This has the same failure mode as the automatic workflow: after analysis-ready=true, a transient artifact-service/upload failure fails fetch-binlog and suppresses the agent, rather than allowing the hlx-only path to run. Make the artifact handoff best-effort and let the agent proceed with no downloaded binlogs when the handoff is unavailable.

      - name: Upload analysis artifact

.github/workflows/build-failure-analysis-command.md:538

• The guard is now MAX_ZIP_BYTES=2147483648 (2 GB), but this explanation still says the per-artifact cap is 500 MB and that an ordinary build trips it. That is materially misleading for operators assessing disk/network exposure and for anyone deciding whether a skipped artifact should be expected; update the comment to describe the actual 2 GB limit and fallback behavior.

          # A 500 MB per-artifact cap is close enough to the size of a real
          # log artifact that an ordinary build trips it, and the job then
          # silently skips exactly the leg it exists to diagnose. Only one
          # archive is on disk at a time (each is deleted before the next

.github/workflows/build-failure-analysis-command.md:486

• The artifact-list request only consumes .value from one response. Azure DevOps paginates this endpoint via a continuation token, and this workflow explicitly expects roughly 150 Logs_Build_* artifacts; a failed leg on a later page will be missed and the agent may analyze incomplete evidence. Preserve and follow the continuation token (or otherwise prove the response is complete) before matching names.

          ado_get "artifact list" "${ADO_API}/build/builds/${BUILD_ID}/artifacts?api-version=7.1" || emit_none
          artifacts_json="${ADO_DOC}"
          mapfile -t all_names < <(printf '%s' "${artifacts_json}" | jq -r '.value // [] | map(select(.name | test("^Logs_Build_"))) | .[].name')

.github/workflows/build-failure-analysis-command.md:375

• This is another non-full-string numeric check: grep -q can accept the first all-digit line of a multi-line value. If the event/aw_context payload is malformed, PR_NUMBER can pass and then reach API paths and workflow-command logging. Use a Bash character-class check (or equivalent) that rejects any non-digit, including newlines.

          if ! printf '%s' "${PR_NUMBER}" | grep -qE '^[0-9]+$'; then

.github/workflows/build-failure-analysis-command.md:411

• This API-response validation has the same per-line grep -qE '^[0-9]+$' flaw: a malformed JSON field containing a numeric first line can pass, and BUILD_ID is then interpolated into later ::warning:: output and ADO URLs. Make the check reject every non-digit character rather than matching any line.

          if ! printf '%s' "${BUILD_ID}" | grep -qE '^[0-9]+$'; then

.github/workflows/build-failure-analysis-command.md:777

• The final gate only re-reads the PR head/merge SHA, so a newer runtime build can complete successfully for the same head while this command run is downloading/analyzing the older failed build; because this workflow sets cancel-in-progress: false, it will still post the stale failure analysis. Re-query the PR's latest runtime build and require the analyzed build to remain the newest failed build before allowing safe outputs, or otherwise invalidate this run when a newer build appears.

          LATEST_PR=$(gh api "repos/${GH_AW_REPO}/pulls/${PR_NUMBER}" 2>/dev/null)
          LATEST_HEAD=$(printf '%s' "${LATEST_PR}" | jq -r '.head.sha // empty')
          LATEST_MERGE=$(printf '%s' "${LATEST_PR}" | jq -r '.merge_commit_sha // empty')

.github/workflows/build-failure-analysis.md:739

• This upload is a hard step after the fetch script has emitted analysis-ready=true. A transient GitHub artifact-service/upload failure therefore marks fetch-binlog failed and prevents the downstream activation/agent jobs from running, so the documented hlx fallback is never reached even though binlogs are optional. Make the artifact handoff best-effort and have the agent continue with an empty binlog directory when upload/download fails.

        if: steps.fetch.outputs.binlog-found == 'true'

.github/workflows/build-failure-analysis.md:263

• This check is not actually full-string: grep applies ^...$ per line and -q succeeds if any line matches. A dispatch input such as 123\n456 therefore passes, after which BUILD_ID is used in ADO URLs and later ::warning:: output. Use a character-class check (or a null-delimited grep) that rejects every non-digit, including newlines.

          if ! printf '%s' "${BUILD_ID}" | grep -qE '^[0-9]+$'; then

.github/workflows/build-failure-analysis.md:320

• The same per-line grep -qE '^[0-9]+$' issue applies here: a malformed/free-form PR input containing a newline after a numeric line can pass validation, then be interpolated into GitHub API paths and ::warning:: messages. Replace this with a full-string character-class check that rejects newlines and all other non-digits.

          if ! printf '%s' "${PR_NUMBER}" | grep -qE '^[0-9]+$'; then

.github/workflows/build-failure-analysis.md:422

• The artifact-list request only consumes .value from one response. Azure DevOps paginates this endpoint via a continuation token, and this workflow explicitly expects roughly 150 Logs_Build_* artifacts; a failed leg on a later page will be missed and the agent may analyze incomplete evidence. Preserve and follow the continuation token (or otherwise prove the response is complete) before matching names.

          ado_get "artifact list" "${ADO_API}/build/builds/${BUILD_ID}/artifacts?api-version=7.1" || emit_none
          artifacts_json="${ADO_DOC}"
          mapfile -t all_names < <(printf '%s' "${artifacts_json}" | jq -r '.value // [] | map(select(.name | test("^Logs_Build_"))) | .[].name')

.github/workflows/shared/build-failure-analysis-shared.md:80

• The workflow declares the same required safe-outputs.data schema for both add_comment and create_pull_request_review_comment, but this instruction requires the structured object only for the summary. The agent's inline suggestion calls therefore have no required data payload and can be rejected by safe-output validation (or lose the artifact identity); require the object on every inline output as well.

   - Post exactly one summary via `add_comment` with structured data
     `{"workflow_artifact":"build-failure-analysis","artifact_kind":"analysis"}`
     and any inline
     `suggestion` blocks via `create_pull_request_review_comment`. Both

• Files reviewed: 5/7 changed files
• Comments generated: 0 new
• Review effort level: Lite