Add stress benchmark CI workflow by dheerajodha · Pull Request #3426 · conforma/cli

dheerajodha · 2026-07-21T12:23:33Z

What:

Add a report-only GitHub Actions workflow that runs the stress benchmark on PRs to main, surfacing peak memory and execution time in the job summary without blocking merges. Uses oras to pull pre-built benchmark data from Quay with upstream regeneration as fallback.

Also adds --allow-past-effective-time to the stress benchmark's CLI args, since the benchmark pins --effective-time 2024-12-10T00:00:00Z for reproducibility and the CLI now rejects past dates by default (EC-1993).

Why:

There is no automated way to catch memory or performance regressions before merge. This workflow gives reviewers a quick read on resource usage without gating PRs on benchmark results.

Tickets:

Ref: EC-1818

fullsend-ai-review · 2026-07-21T12:24:30Z

🤖 Finished Review · ✅ Success · Started 12:24 PM UTC · Completed 12:41 PM UTC
Commit: 87c4a29 · View workflow run →

coderabbitai · 2026-07-21T12:31:42Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a GitHub Actions “Stress Benchmark” workflow for pull requests targeting main and manual runs. It prepares tooling and data, builds and runs the benchmark, captures logs, and publishes parsed metrics or fallback messages to the job summary.

Changes

Stress benchmark workflow

Layer / File(s)	Summary
Workflow setup `.github/workflows/benchmark.yaml`	Defines triggers, read-only permissions, job settings, workload variables, runner hardening, tooling setup, and benchmark data preparation.
Benchmark execution `.github/workflows/benchmark.yaml`	Builds and runs the stress binary while capturing standard output and standard error.
Result reporting `.github/workflows/benchmark.yaml`	Processes logs, handles missing output, extracts and converts metrics, and writes a Markdown results table or fallback message to the job summary.

Estimated code review effort: 2 (Simple) | ~10 minutes

Sequence Diagram(s)

sequenceDiagram
  participant GitHubActions
  participant BenchmarkRunner
  participant StressBenchmark
  participant JobSummary
  GitHubActions->>BenchmarkRunner: Trigger stress benchmark workflow
  BenchmarkRunner->>BenchmarkRunner: Prepare tools and benchmark data
  BenchmarkRunner->>StressBenchmark: Build and execute benchmark
  StressBenchmark-->>BenchmarkRunner: Write stdout and stderr logs
  BenchmarkRunner->>JobSummary: Parse metrics and publish summary

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely identifies the primary change: adding a stress benchmark CI workflow.
Description check	✅ Passed	The description includes complete What, Why, and Tickets sections with relevant details about the workflow and its purpose.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

codecov · 2026-07-21T12:38:39Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag	Coverage Δ
acceptance	`54.42% <ø> (ø)`
generative	`16.35% <ø> (ø)`
integration	`27.56% <ø> (ø)`
unit	`72.13% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

fullsend-ai-review · 2026-07-21T12:41:49Z

Review

Findings

High

[protected-path] .github/workflows/benchmark.yaml — This PR adds a file under the protected .github/ path. No linked GitHub issue provides authorization for modifying governance/infrastructure files, and the PR description's "Why" section is blank. Human approval is always required for protected-path changes.
Remediation: Link a GitHub issue (or create one referencing Jira EC-1818) that explicitly authorizes this CI workflow addition, and fill in the "Why" section of the PR description.

Medium

[logic-error] benchmark/simple/simple.go:108 — The simple benchmark uses --effective-time 2024-12-10T00:00:00Z (a past date) without --allow-past-effective-time. This PR correctly adds the flag to the stress benchmark but does not update the simple benchmark. Since past effective times are now rejected by default (see internal/policy/policy.go), the simple benchmark will fail at runtime with: "effective time ... is in the past; use --allow-past-effective-time to override".
Remediation: Add "--allow-past-effective-time" to the argument list in benchmark/simple/simple.go at line 109 (after the --effective-time argument).

Low

[action-pin-consistency] .github/workflows/benchmark.yaml:43 — harden-runner is pinned to v2.20.1 (sha b09bb98e) while 5 of 6 other repo workflows pin to v2.20.0 (sha bf7454d0). Only release.yaml also uses v2.20.1.
Remediation: Align to the repo's majority pin (v2.20.0) or update all workflows to v2.20.1 in a single pass.

Next steps:

/fs-fix — agent addresses review findings automatically
/fs-fix <your instruction> — agent fixes with your specific guidance
Push commits directly — review re-runs automatically on push
/fs-fix-stop — disable automatic fix runs for this PR

Previous run

Review

Findings

High

[protected-path] .github/workflows/benchmark.yaml — This PR adds a file under the protected .github/ path. The PR has no linked GitHub issue to authorize the change (only a Jira reference EC-1818 which is not verifiable via the GitHub API). Human approval is always required for changes to governance and infrastructure files.
Remediation: Link a GitHub issue that documents the intent and authorization for adding this workflow.

Medium

[action-version-pinning] .github/workflows/benchmark.yaml:22 — The pinned action versions are older than those used in other repository workflows. step-security/harden-runner uses v2.15.1 while others use v2.20.0+; actions/checkout uses v6.0.2 vs v6.1.0; actions/cache/restore uses v5.0.4 vs v5.1.0; actions/setup-go uses v6.3.0 vs v6.5.0.
Remediation: Update action pins to match the versions used in other workflows: harden-runner@bf7454d0, checkout@d23441a4, cache/restore@caa29612, setup-go@924ae3a1.
[missing-authorization] .github/workflows/benchmark.yaml — The PR references Jira ticket EC-1818 but has no linked GitHub issue. Non-trivial changes (127+ lines, new CI workflow) should have a linked GitHub issue that documents the intent and acceptance criteria.
Remediation: Create a GitHub issue describing the goal, link it to the PR, and reference EC-1818.

Low

[edge-case] .github/workflows/benchmark.yaml:103 — The grep patterns use [\d.]+ which would match malformed numeric strings containing multiple dots (e.g., 1.2.3). In practice the Go benchmark driver produces well-formed numbers, so this is unlikely to cause issues, but a stricter regex like [0-9]+\.?[0-9]* would be more robust.
[scope-consistency] benchmark/stress/stress.go:168 — The PR description does not mention the --allow-past-effective-time flag addition and why it is needed (the hardcoded effective-time 2024-12-10T00:00:00Z is now in the past).
Remediation: Update the PR description to mention the supporting change.

Next steps:

/fs-fix — agent addresses review findings automatically
/fs-fix <your instruction> — agent fixes with your specific guidance
Push commits directly — review re-runs automatically on push
/fs-fix-stop — disable automatic fix runs for this PR

Previous run (2)

Review

Findings

Medium

[protected-path] .github/workflows/benchmark.yaml — This PR adds a file under .github/ which is a protected path. The PR references Jira ticket EC-1818 and the description explains the rationale for adding a benchmark CI workflow. Human approval is always required for protected-path changes, regardless of context.

Low

[permission-expansion] .github/workflows/benchmark.yaml:59 — The cache restore step uses path: '**' which restores cached content over the entire workspace. Combined with executing scripts from the workspace (./prepare_data.sh, the built ./stress binary), a poisoned cache from the main branch could overwrite these files before execution. This is a pre-existing repo-wide pattern (same convention in codeql.yaml, lint.yaml, checks-codecov.yaml, update-go-containerregistry.yaml) and not specific to this PR.
[incomplete-docs] benchmark/README.md — The benchmark README documents how to run benchmarks locally but does not mention that the stress benchmark is also run automatically via CI on PRs to main. Consider adding a brief note referencing .github/workflows/benchmark.yaml.

Previous run (3)

Review

Findings

Low

[edge-case] .github/workflows/benchmark.yaml:99 — The grep pattern grep -oP '[\d.]+ ns/op' could theoretically match multiple substrings if the benchmark output format ever adds additional metrics ending in ns/op with a digit-space prefix. Currently, the input $line is constrained to ^BenchmarkStress lines and standard Go benchmark output produces exactly one ns/op value per line, so this is not a practical concern today. Adding | head -1 after the grep would be a low-cost defensive measure.

Protected paths detected — this PR modifies files under one or more
protected paths. The review agent cannot approve PRs that touch these paths.
A human reviewer must approve this PR.

Protected files in this PR:

.github/workflows/benchmark.yaml

Previous run (4)

Review

Findings

Medium

[protected-path] .github/workflows/benchmark.yaml — This PR adds a file under the protected .github/ path. The PR references Jira ticket EC-1818 and explains the rationale for the change in the description. Human approval is always required for changes to governance and infrastructure files under .github/, regardless of context.

Low

[markdown-injection] .github/workflows/benchmark.yaml:91 — Stderr output from the benchmark is placed between Markdown code fences in GITHUB_STEP_SUMMARY without sanitization. If stderr contains a line of triple backticks, it could break out of the code fence and inject arbitrary Markdown into the job summary. Impact is limited since the pull_request trigger (not pull_request_target) means the attacker already controls the code producing stderr — no privilege escalation or data exfiltration is possible.
Remediation: Pipe stderr through a sanitization step that escapes triple-backtick sequences, or use a heredoc with a unique delimiter instead of code fences.

Previous run (5)

Review

Findings

Low

[injection] .github/workflows/benchmark.yaml:93 — Shell variables (ns_op, peak_rss, alloc, heap) are interpolated directly into awk program strings via double-quoted shell expansion (e.g., ${ns_op:-0}). Currently, the values are constrained to digits and dots by the upstream grep -oP '[\\d.]+ ns/op' extraction with the -o flag, so exploitation is not feasible. However, this pattern is fragile: if the grep patterns are later modified to be less restrictive, the awk interpolation becomes exploitable.
Remediation: Pass the values as awk variables instead of interpolating into the program string: secs=$(awk -v val="${ns_op:-0}" 'BEGIN {printf "%.1f", val / 1000000000}')

Note: This PR modifies files under .github/, which is a protected path. Human approval is required for changes to governance and infrastructure files, regardless of automated review outcome.

Previous run (6)

Review

Findings

Medium

[protected-path] .github/workflows/benchmark.yaml — This PR adds a file under .github/, which is a protected path requiring human approval. The PR references Jira ticket EC-1818 and explains the rationale in the description. Human approval is always required for protected-path changes, regardless of context.
[missing established step pattern] .github/workflows/benchmark.yaml:52 — Every existing workflow in this repo that sets up Go (checks-codecov, lint, codeql, release, update-go-containerregistry) includes a cache restore step using actions/cache/restore between checkout and setup-go. The benchmark workflow omits this step, which will result in slower CI runs as the Go module and build caches are not restored.
Remediation: Add a "Restore Cache" step using actions/cache/restore between "Checkout repository" and "Setup Go environment", matching the pattern in other workflows.

Low

[error-handling] .github/workflows/benchmark.yaml:73 — The pipeline ./stress 2>benchmark-stderr.txt | tee benchmark-output.txt does not use set -o pipefail. If the stress binary fails, tee still exits 0, so the step reports success. While continue-on-error: true and the summary step's guard clauses handle this gracefully, the step's green check could be misleading when investigating failures.
Remediation: Add set -o pipefail at the beginning of the run block, or check ${PIPESTATUS[0]} after the pipeline.

Labels: PR already has appropriate labels (github_actions, ci, size: M, requires-manual-review). Adding Review effort label for reviewer guidance.

Previous run (7)

Review

Findings

Medium

[protected-path] .github/workflows/benchmark.yaml — This PR adds a file under the .github/ protected path. The PR description and Jira reference (EC-1818) explain the rationale (adding a report-only benchmark CI workflow), providing sufficient context. However, human approval is always required for protected-path changes, regardless of context.
[naming-convention] .github/workflows/benchmark.yaml:32 — The job ID stress has no explicit name: field. All other workflows in the repo provide descriptive job names (e.g., Test, Lint, Analyze, Scorecard analysis). Without a name:, the GitHub Actions UI displays the raw job ID stress.
Remediation: Add name: Stress Benchmark to the job definition.

Low

[injection] .github/workflows/benchmark.yaml:97 — Shell variables (ns_op, peak_rss, alloc, heap) are interpolated into double-quoted awk command strings. The upstream grep -oP '[\d.]+' regex constrains values to digits and dots, making awk injection practically infeasible. However, the pattern is fragile; using awk -v to pass variables is a safer idiom at zero cost.
Remediation: Use secs=$(awk -v val="${ns_op:-0}" 'BEGIN {printf "%.1f", val / 1000000000}').
[step-pattern-consistency] .github/workflows/benchmark.yaml:32 — All other Go-based workflows in the repo include a Restore Cache step using actions/cache/restore after checkout. The benchmark workflow omits this step. This may be intentional (cold build for consistent results), but adding the cache would reduce CI wall time without affecting benchmark accuracy since only the Go build step would benefit, not the benchmark execution itself.
[scope-creep] .github/workflows/benchmark.yaml — The workflow triggers only on pull_request to main, while other CI workflows also trigger on push to main and release-* branches. This is a reasonable design choice for a report-only benchmark (job summaries are most useful on PRs), but confirming this is intentional is worthwhile since push-triggered runs could enable historical regression tracking.

Previous run (8)

Review

Findings

High

[protected-path] .github/workflows/benchmark.yaml — This PR adds a file under .github/, which is a protected path requiring human approval. No linked GitHub issue establishes authorization for this change. While the Jira ticket EC-1818 is referenced and the PR description explains the rationale, protected-path changes require a linked issue for scope verification.
Remediation: Link a GitHub issue (or create one mirroring EC-1818) that authorizes this CI workflow addition.

Low

[error-handling] .github/workflows/benchmark.yaml:72 — The "Run stress benchmark" step redirects stderr to /dev/null (./stress 2>/dev/null), silently discarding all error output including Go panics (the stress binary uses panic() in 6+ locations for invalid env vars, setup failures, and execution errors). Combined with continue-on-error: true, a failing benchmark produces no diagnostic output — the job summary would only show "No benchmark results found in output" with no indication of the root cause.
Remediation: Remove the 2>/dev/null redirect so errors appear in the CI step log, or redirect stderr to a file and include it in the job summary on failure.
[naming-convention] .github/workflows/benchmark.yaml:32 — The job ID Stress uses PascalCase, while the majority of workflows in the repo (11 of 15 job IDs) use lowercase (analyze, lint, auto-merge, release, info, dispatch, stop-fix). PascalCase is only used in checks-codecov.yaml.
Remediation: Consider renaming the job ID to stress for consistency with the dominant convention.

Previous run (9)

Review

Findings

Low

[edge-case] .github/workflows/benchmark.yaml:65 — go run . 2>/dev/null suppresses all stderr output, including Go compilation errors, runtime panics, and diagnostic messages from the benchmark driver. If the benchmark fails silently, the summary step reports "No benchmark results found" with no indication of what went wrong, making failures hard to diagnose in CI.
Consider redirecting stderr to a log file: go run . 2>benchmark-stderr.txt | tee benchmark-output.txt
[pattern-inconsistency] .github/workflows/benchmark.yaml:22 — The workflow triggers only on pull_request without workflow_dispatch, unlike other CI workflows in the repo (lint.yaml, checks-codecov.yaml) that include it for manual re-runs. While omitting push is reasonable for a non-blocking benchmark, adding workflow_dispatch would allow ad-hoc performance investigations.
[scope-alignment] .github/workflows/benchmark.yaml:35 — The workflow sets EC_STRESS_COMPONENTS=40 and EC_STRESS_WORKERS=10, which differ from the code defaults in stress.go (defaultComponents=10, defaultWorkers=35). The rationale for these CI-specific values (higher component count, lower parallelism — presumably to test memory pressure on CI runners) is not documented.

Note: This PR modifies files under .github/ (a protected path). Human approval is required for protected-path changes regardless of automated review findings.

Previous run (10)

Review

Findings

High

[protected-path] .github/workflows/benchmark.yaml — This PR modifies files under the protected .github/ path. The PR has no linked GitHub issue (only a Jira reference to EC-1818 which cannot be verified through the GitHub API). Human approval is always required for changes to governance and infrastructure files, regardless of the change's nature.
Remediation: Link a GitHub issue that authorizes the modification of .github/ workflow files, or ensure a human maintainer explicitly approves this change.

Low

[error-handling] .github/workflows/benchmark.yaml:64 — The benchmark step runs go run . 2>/dev/null | tee benchmark-output.txt, which discards all stderr output including Go compilation errors, runtime panics, and benchmark driver diagnostics. If the benchmark fails to compile or panics, the job summary silently reports "No benchmark results found" with no indication of the root cause.
Remediation: Consider separating compilation from execution so build errors remain visible: go build -o stress . && ./stress 2>/dev/null | tee benchmark-output.txt.
[scope-coherence] .github/workflows/benchmark.yaml:35 — The workflow sets EC_STRESS_WORKERS: "5" while the benchmark code defaults to 35 (defaultWorkers in benchmark/stress/stress.go). The rationale for this CI-specific override is not documented, creating a disconnect between the benchmark defaults and what CI exercises.
Remediation: Add a YAML comment explaining the override (e.g., # Reduced from default 35 to fit GitHub Actions runner memory).
[naming-coherence] .github/workflows/benchmark.yaml:18 — The workflow is named "Benchmark" generically but only runs the stress benchmark. The project has two benchmark types (benchmark/simple/ and benchmark/stress/).

Labels: PR adds a GitHub Actions CI workflow for benchmark automation

fullsend-ai-review · 2026-07-21T13:12:23Z

🤖 Finished Review · ✅ Success · Started 1:12 PM UTC · Completed 1:26 PM UTC
Commit: 87c4a29 · View workflow run →

Superseded by updated review

fullsend-ai-review · 2026-07-21T14:08:33Z

🤖 Finished Review · ✅ Success · Started 2:08 PM UTC · Completed 2:24 PM UTC
Commit: 87c4a29 · View workflow run →

fullsend-ai-review · 2026-07-22T11:58:57Z

🤖 Review · ⚠️ Cancelled · Started 11:58 AM UTC · Ended 12:01 PM UTC
Commit: 87c4a29 · View workflow run →

fullsend-ai-review · 2026-07-22T12:02:30Z

🤖 Finished Review · ✅ Success · Started 12:02 PM UTC · Completed 12:16 PM UTC
Commit: 87c4a29 · View workflow run →

Superseded by updated review

fullsend-ai-review · 2026-07-22T12:21:27Z

🤖 Finished Review · ✅ Success · Started 12:21 PM UTC · Completed 12:36 PM UTC
Commit: 87c4a29 · View workflow run →

fullsend-ai-review · 2026-07-22T13:08:09Z

🤖 Review · ❌ Terminated · Started 1:08 PM UTC · Ended 1:23 PM UTC
Commit: 87c4a29 · View workflow run →

qodo-for-conforma · 2026-07-22T13:08:11Z

PR Summary by Qodo

Add Stress Benchmark GitHub Actions workflow

✨ Enhancement ⚙️ Configuration changes 🕐 20-40 Minutes

AI Description

• Add a report-only “Stress Benchmark” GitHub Actions workflow for PRs to main.
• Pull prebuilt benchmark data via ORAS (fallback to regeneration), then build and run.
• Surface runtime and memory metrics in the GitHub job summary without blocking merges.

Diagram

graph TD
  pr{{"PR to main"}} --> wf["Stress Benchmark workflow"] --> runner["Ubuntu runner"] --> prep["Prepare data (prepare_data.sh)"] --> build["Build benchmark (go build)"] --> exec["Run benchmark (stress)"] --> sum["Job summary metrics"]
  prep --> quay[("Quay artifact (oras pull)")]

High-Level Assessment

The following are alternative approaches to this PR:

1. Upload raw outputs as artifacts

➕ Makes stdout/stderr and benchmark-output.txt easy to download for deeper debugging
➕ Preserves historical raw data for manual comparisons
➖ Adds artifact storage and retention considerations
➖ Slightly longer workflow and more moving parts

2. Baseline comparison (benchstat / previous main)

➕ Turns the report into actionable regression signals (delta vs baseline)
➕ Can later be promoted to a required check with thresholds
➖ Requires sourcing a baseline (e.g., main branch run) and managing variability/noise
➖ More complex logic and potential for flaky reporting

3. Scheduled runs only (nightly)

➕ Reduces PR noise and avoids extra CI load on every PR
➕ More stable environment for trend tracking
➖ Regressions are detected later than PR-time feedback
➖ Less contextual visibility for authors reviewing a change

Recommendation: The current report-only PR workflow is the best first step: it provides early visibility into time/RSS regressions without introducing merge friction. Consider a follow-up to upload benchmark output as an artifact and/or add optional baseline comparison once the benchmark’s variance and runtime characteristics are well understood.

Files changed (1) +117 / -0

Other (1) +117 / -0

benchmark.yamlAdd non-blocking stress benchmark workflow with job-summary reporting +117/-0

Add non-blocking stress benchmark workflow with job-summary reporting

• Introduces a new GitHub Actions workflow triggered on PRs to main and manual dispatch. The job prepares benchmark data (via ORAS + fallback script), builds and runs the stress benchmark, then parses stdout/stderr to publish execution time and memory metrics into the GitHub job summary while allowing failures via continue-on-error.

.github/workflows/benchmark.yaml

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/benchmark.yaml:
- Around line 32-33: Add a job-level timeout to the benchmark job in the
workflow containing runs-on and continue-on-error, setting it above the
documented 6–8 minute runtime while still bounding hung pulls or stress
processes. Preserve continue-on-error so benchmark failures remain non-blocking.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: 9e1158a9-cb54-49bb-999b-074a054f1b55

📥 Commits

Reviewing files that changed from the base of the PR and between 850319e and bb9a516.

📒 Files selected for processing (1)

.github/workflows/benchmark.yaml

qodo-for-conforma · 2026-07-22T13:13:02Z

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

1. ~~Zeroed benchmark metrics~~ ✗ Dismissed 🐞 Bug ≡ Correctness

Description

The job summary silently reports 0 for execution time/memory when parsing fails because empty
extracted values are coerced to 0 via ${var:-0}. This can produce a plausible-looking benchmark
table that masks parsing breakage or missing metrics in the benchmark output.

Code

.github/workflows/benchmark.yaml[R96-104]

+          ns_op=$(echo "$line" | grep -oP '[\d.]+ ns/op' | awk '{print $1}')
+          peak_rss=$(echo "$line" | grep -oP '[\d.]+ peak-RSS-bytes' | awk '{print $1}')
+          alloc=$(echo "$line" | grep -oP '[\d.]+ allocated-bytes/op' | awk '{print $1}')
+          heap=$(echo "$line" | grep -oP '[\d.]+ heap-bytes-from-system' | awk '{print $1}')
+
+          secs=$(awk "BEGIN {printf \"%.1f\", ${ns_op:-0} / 1000000000}")
+          rss_mb=$(awk "BEGIN {printf \"%.0f\", ${peak_rss:-0} / 1048576}")
+          alloc_mb=$(awk "BEGIN {printf \"%.0f\", ${alloc:-0} / 1048576}")
+          heap_mb=$(awk "BEGIN {printf \"%.0f\", ${heap:-0} / 1048576}")

Relevance

●● Moderate
No historical evidence about rejecting/accepting default-to-zero metric parsing in workflow
summaries; pattern not seen in review history.
ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The summary computation explicitly converts empty parsed values into zeros, which will misreport
metrics whenever the grep-based extraction doesn’t find a match (partial parsing failure).

.github/workflows/benchmark.yaml[96-104]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The workflow extracts `ns/op`, `peak-RSS-bytes`, `allocated-bytes/op`, and `heap-bytes-from-system` from the benchmark output, but if any extraction yields an empty string the summary falls back to `0` using `${var:-0}`. This can silently publish incorrect results (e.g., `0.0s`, `0 MB`).

### Issue Context
This is a report-only workflow; its primary value is accurate visibility. If parsing fails, the summary should clearly indicate missing/unknown values and ideally include the raw result line(s) used.

### Fix Focus Areas
- .github/workflows/benchmark.yaml[89-117]

### Suggested fix approach
- Detect missing extractions and render `N/A` (or similar) instead of defaulting to 0.
- Optionally append the matched `BenchmarkStress...` result line to the summary for debugging.
- Example pattern:
 - after extraction, check `[[ -z "$ns_op" ]]` etc.
 - if missing, set `secs="N/A"` / `rss_mb="N/A"` rather than computing numeric conversions.
 - keep the summary step non-failing and informative.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

2. go build missing CGO_ENABLED=0 ✗ Dismissed 📘 Rule violation ☼ Reliability

Description

The workflow builds the stress benchmark binary using go build without setting CGO_ENABLED=0,
which can produce cgo-linked binaries and violate the build requirement. This can reduce
portability/reproducibility of the CI-built binary.

Code

.github/workflows/benchmark.yaml[64]
+        run: go build -o benchmark/stress/stress ./benchmark/stress

Relevance

●● Moderate
No repo history found about enforcing CGO_ENABLED=0 in CI go build; rule referenced but no local
precedent.
ⓘ Recommendations generated based on similar findings in past PRs

Evidence
PR Compliance ID 111 requires Go build invocations in CI/build scripts to set CGO_ENABLED=0. The
new workflow runs go build -o benchmark/stress/stress ./benchmark/stress without CGO_ENABLED=0.
Rule 111: Build Go binaries with CGO disabled
.github/workflows/benchmark.yaml[63-65]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The CI workflow builds a Go binary without disabling CGO (`CGO_ENABLED=0`), which violates the compliance requirement for Go builds.

## Issue Context
This workflow step builds `benchmark/stress/stress` via `go build` and should explicitly set `CGO_ENABLED=0` (or document a justified exception if cgo is required).

## Fix Focus Areas
- .github/workflows/benchmark.yaml[63-65]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

3. ~~Unbounded BenchmarkStress grep~~ ✗ Dismissed 🐞 Bug ☼ Reliability

Description

line is assigned from grep '^BenchmarkStress' ... without limiting to a single match, so
multiple result lines will create a multiline line value that can yield malformed metric
extraction and numeric formatting. The workflow should either enforce exactly one match or select a
deterministic single line (e.g., the last match) before parsing.

Code

.github/workflows/benchmark.yaml[R89-99]

+          line=$(grep '^BenchmarkStress' benchmark/stress/benchmark-output.txt || true)
+          if [[ -z "$line" ]]; then
+            echo "## Stress Benchmark" >> "$GITHUB_STEP_SUMMARY"
+            echo "No benchmark results found in output." >> "$GITHUB_STEP_SUMMARY"
+            exit 0
+          fi
+
+          ns_op=$(echo "$line" | grep -oP '[\d.]+ ns/op' | awk '{print $1}')
+          peak_rss=$(echo "$line" | grep -oP '[\d.]+ peak-RSS-bytes' | awk '{print $1}')
+          alloc=$(echo "$line" | grep -oP '[\d.]+ allocated-bytes/op' | awk '{print $1}')
+          heap=$(echo "$line" | grep -oP '[\d.]+ heap-bytes-from-system' | awk '{print $1}')

Relevance

●● Moderate
No historical evidence of reviewers flagging unbounded grep/multiline parsing issues in GitHub
Actions workflows.
ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The workflow uses an unconstrained grep to populate line, then reuses it as if it were a single
benchmark record for subsequent metric extraction.

.github/workflows/benchmark.yaml[89-99]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The workflow assumes a single `BenchmarkStress...` result line, but `grep` can return multiple matches and the current code does not constrain or validate this. If multiple lines are present, downstream parsing can produce multiline numeric variables, which can break formatting or produce incorrect summary values.

### Issue Context
The job summary parser treats `line` as a single record for metric extraction. This should be made explicit by either:
- rejecting ambiguous output (multiple matches), or
- choosing one authoritative line.

### Fix Focus Areas
- .github/workflows/benchmark.yaml[89-99]

### Suggested fix approach
- Use a deterministic single-line selection, e.g.:
 - `line=$(grep '^BenchmarkStress' benchmark/stress/benchmark-output.txt | tail -n 1 || true)`
- Optionally validate count:
 - `count=$(grep -c '^BenchmarkStress' ... || true)` and if `count != 1`, write a note to `$GITHUB_STEP_SUMMARY` and skip metric parsing.
- Consider including the selected raw line in the summary for traceability.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Context used

✅ Compliance rules (platform): 27 rules

Tip of the day

💡 Did you know, you can group findings by type and pick your Finding display, from Minimal to Full

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

fullsend-ai-review · 2026-07-22T13:23:08Z

🤖 Finished Review · ✅ Success · Started 1:08 PM UTC · Completed 1:23 PM UTC
Commit: 87c4a29 · View workflow run →

fullsend-ai-review · 2026-07-27T11:58:19Z

🤖 Finished Review · ✅ Success · Started 11:58 AM UTC · Completed 12:13 PM UTC
Commit: 87c4a29 · View workflow run →

fullsend-ai-review · 2026-07-27T12:14:49Z

🤖 Finished Review · ✅ Success · Started 12:14 PM UTC · Completed 12:31 PM UTC
Commit: 87c4a29 · View workflow run →

simonbaird · 2026-07-31T21:28:59Z

Looks good I guess. Did you look at the fullsend suggestions?

fullsend-ai-review · 2026-08-10T17:40:02Z

🤖 Review · ⚠️ Cancelled · Started 5:40 PM UTC · Ended 5:50 PM UTC

Commit: 87c4a29 · View workflow run →

dheerajodha · 2026-08-10T17:41:05Z

Looks good I guess. Did you look at the fullsend suggestions?

I did, they were all outdated, weird. I was expecting the comment to get marked as outdated automatically, but that didn't work so I manually resolved them.

fullsend-ai-review · 2026-08-10T17:51:55Z

🤖 Finished Review · ✅ Success · Started 5:51 PM UTC · Completed 6:07 PM UTC

Commit: 87c4a29 · View workflow run →

Add a report-only GitHub Actions workflow that runs the stress benchmark on PRs to main, surfacing peak memory and execution time in the job summary without blocking merges. Uses oras to pull pre-built benchmark data from Quay with upstream regeneration as fallback. Ref: EC-1818 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Initial CI run with 10/5 completed in under a minute. Increase to 40/10 to target 5-6 minute total job time on CI runners. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Rename workflow to "Stress Benchmark", add workflow_dispatch trigger, separate build from execution to surface compilation errors, and document CI-specific env var overrides. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Doubles component count from 40 to 80 to increase peak RSS (~4 GB), making memory regressions more visible in CI job summaries. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Redirect stderr to a file instead of /dev/null so panics and errors are surfaced in the job summary. Rename job ID from Stress to stress to match the dominant lowercase convention in the repo. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

All components use the same image digest, so internal caches deduplicate after the first — higher counts don't add memory pressure. Use the code default to avoid misleading numbers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Add job name for GitHub Actions UI display - Add cache restore step matching repo convention - Use awk -v for variable passing instead of shell interpolation - Add set -o pipefail so benchmark failures are not masked by tee Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Adds a 15-minute timeout to prevent hung oras pulls or stuck benchmark processes from running for the default 6-hour limit. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The ec CLI emits harmless logrus ERRO messages during normal validation, which made every job summary start with a misleading Stderr section. Gate it on step outcome so it only appears when the benchmark actually fails. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The CLI now rejects past effective times by default. The benchmark uses a pinned date for reproducibility, so opt into the override. Resolves: EC-1818 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Align pinned action versions with the rest of the repository: - harden-runner v2.15.1 → v2.20.1 - checkout v6.0.2 → v6.1.0 - cache/restore v5.0.4 → v5.1.0 - setup-go v6.3.0 → v6.5.0 Resolves: EC-1818 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fullsend-ai-review · 2026-08-11T06:18:44Z

🤖 Review · ❌ Terminated · Started 6:18 AM UTC · Ended 6:35 AM UTC

Commit: 87c4a29 · View workflow run →

dheerajodha · 2026-08-11T06:19:59Z

Had to force push, to get the commits that fixed the vulnerability-related violations I was getting.

fullsend-ai-review

See the review comment for full details.

fullsend-ai-review · 2026-08-11T06:35:12Z

🤖 Finished Review · ✅ Success · Started 6:18 AM UTC · Completed 6:35 AM UTC

Commit: 87c4a29 · View workflow run →

github-actions Bot added the size: M label Jul 21, 2026

This comment was marked as outdated.

Sign in to view

fullsend-ai-review Bot added ci github_actions Pull requests that update GitHub Actions code labels Jul 21, 2026

This comment was marked as outdated.

Sign in to view

fullsend-ai-review Bot added the requires-manual-review Review requires human judgment label Jul 21, 2026

This comment was marked as outdated.

Sign in to view

fullsend-ai-review Bot removed the requires-manual-review Review requires human judgment label Jul 21, 2026

This comment was marked as outdated.

Sign in to view

fullsend-ai-review Bot added the requires-manual-review Review requires human judgment label Jul 22, 2026

This comment was marked as outdated.

Sign in to view

fullsend-ai-review Bot added the Review effort 2/5 label Jul 22, 2026

dheerajodha marked this pull request as ready for review July 22, 2026 13:07

coderabbitai Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread .github/workflows/benchmark.yaml

qodo-for-conforma Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread .github/workflows/benchmark.yaml

Comment thread .github/workflows/benchmark.yaml

Comment thread .github/workflows/benchmark.yaml

This comment was marked as outdated.

Sign in to view

st3penta mentioned this pull request Aug 10, 2026

Include PR comments in review count conforma/review-rot#249

Open

This comment was marked as outdated.

Sign in to view

fullsend-ai-review Bot removed the requires-manual-review Review requires human judgment label Aug 10, 2026

dheerajodha and others added 13 commits August 11, 2026 11:37

chore(EC-1818): bump stress benchmark to 40 components, 10 workers

6e5b77e

Initial CI run with 10/5 completed in under a minute. Increase to 40/10 to target 5-6 minute total job time on CI runners. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

chore(EC-1818): bump stress benchmark to 80 components

4c9d380

Doubles component count from 40 to 80 to increase peak RSS (~4 GB), making memory regressions more visible in CI job summaries. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fix(EC-1818): add job timeout to benchmark workflow

9003852

Adds a 15-minute timeout to prevent hung oras pulls or stuck benchmark processes from running for the default 6-hour limit. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fix(EC-1818): add descriptions to job summary metrics table

8545c00

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fix(EC-1818): update benchmark time estimate to match CI observations

c9c79eb

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fix: allow past effective time in stress benchmark

033e8bb

The CLI now rejects past effective times by default. The benchmark uses a pinned date for reproducibility, so opt into the override. Resolves: EC-1818 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

dheerajodha force-pushed the EC-1818 branch from 1592371 to fc37eb1 Compare August 11, 2026 06:17

fullsend-ai-review Bot suggested changes Aug 11, 2026

View reviewed changes

Comment thread .github/workflows/benchmark.yaml

Uh oh!

Conversation

dheerajodha commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What:

Why:

Tickets:

Uh oh!

fullsend-ai-review Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

codecov Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

fullsend-ai-review Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review

Findings

High

Medium

Low

Review

Findings

High

Medium

Low

Review

Findings

Medium

Low

Review

Findings

Low

Review

Findings

Medium

Low

Review

Findings

Low

Review

Findings

Medium

Low

Review

Findings

Medium

Low

Review

Findings

High

Low

Review

Findings

Low

Review

Findings

High

Low

Uh oh!

This comment was marked as outdated.

Uh oh!

fullsend-ai-review Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

fullsend-ai-review Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

dheerajodha commented Jul 21, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jul 21, 2026 •

edited

Loading

coderabbitai Bot commented Jul 21, 2026 •

edited

Loading

codecov Bot commented Jul 21, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jul 21, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jul 21, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jul 21, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jul 22, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jul 22, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jul 22, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jul 22, 2026 •

edited

Loading

qodo-for-conforma Bot commented Jul 22, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jul 27, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jul 27, 2026 •

edited

Loading

fullsend-ai-review Bot commented Aug 10, 2026 •

edited

Loading

fullsend-ai-review Bot commented Aug 10, 2026 •

edited

Loading

fullsend-ai-review Bot commented Aug 11, 2026 •

edited

Loading