Skip to content

ci: post the LOC diff as a sticky pull request comment - #1753

Merged
sbryngelson merged 5 commits into
masterfrom
ci/loc-diff-comment
Aug 23, 2026
Merged

ci: post the LOC diff as a sticky pull request comment#1753
sbryngelson merged 5 commits into
masterfrom
ci/loc-diff-comment

Conversation

@sbryngelson

@sbryngelson sbryngelson commented Aug 23, 2026

Copy link
Copy Markdown
Member

Written by Claude Code, driven by @sbryngelson. Every line of the diff, the commit messages, and this description were authored by Claude Code in an interactive session.


Description

Makes the line-of-code diff actually visible: github-actions[bot] now posts a single sticky comment on each PR showing which files changed size and by how much. Today ./mfc.sh count_diff exists but its output only ever reaches the CI log, where nobody reads it.

This is a port of tinygrad's LOC bot (sz.py + szdiff.yml), with three deliberate departures:

  • Counts against the merge base, not the tip of master. tinygrad compares PR head against current master, so a stale branch is charged for other people's commits; they need a second job to detect that and post "the line count difference bot is disabled" instead. Fetching refs/pull/N/head and diffing against git merge-base makes that failure mode impossible, and the extra job disappears.
  • No tokens/line column. tinygrad needs it because they enforce a hard line ceiling and have to catch statements crammed onto one line. ./mfc.sh format already makes that unavailable here.
  • The comment is taken back down when the diff returns to zero, rather than leaving a number that is no longer true.

pull_request_target is required so that PRs from forks can be commented on at all — under pull_request their GITHUB_TOKEN is read-only. The safety invariant, noted in the workflow: the toolchain runs from base/, and pr/ is only ever opened and line-counted, never executed. persist-credentials: false on the PR checkout, and no attacker-controlled string is interpolated into a run: block.

Two fixes to count_diff fall out of pointing it at an untrusted tree: non-regular files are skipped (a symlinked src/*.fpp pointing at /dev/zero would otherwise be read until the runner died) and reads use errors="replace" (one non-UTF-8 byte in a comment — an accented name — would otherwise fail the job).

Incidentally, the counter now recurses into src/*/include, which the old src/<dir>/*.*f* glob silently skipped — 13 files including the GPU macro headers.

Type of change

  • Other (CI tooling)

Testing

The workflow itself cannot run until it is on master, since pull_request_target reads the workflow file from the base branch. What was verified locally:

  • count_diff against a real merge-base worktree of this branch, and against master~5 to exercise the negative-diff path; both the terminal table and the Markdown render correctly.
  • Comparing a tree against itself produces a zero-byte file, which is what drives the delete path.
  • Both untrusted-input cases reproduced against the real counter and confirmed closed: a symlink to /dev/zero is skipped rather than read forever, and a Latin-1 byte in a comment counts normally instead of raising UnicodeDecodeError.
  • The changed=true/false step logic under bash -e, GitHub's default shell.
  • marocchino/sticky-pull-request-comment behavior read from source: delete: true with an empty body passes validation, and deleting a nonexistent comment returns cleanly rather than erroring.
  • ./mfc.sh format and ./mfc.sh precheck pass. No Fortran changed, so no golden files are affected.

Checklist

  • I added or updated tests for new behavior
  • I updated documentation if user-facing behavior changed

No test: this is CI plumbing whose only real integration point is GitHub Actions, which the test suite cannot reach. No docs: the CLI reference is generated and gitignored, and count_diff is a CI-only command.

One reviewer-facing note — count_diff changed signature from an MFC_PR environment variable to two positional tree paths (./mfc.sh count_diff <base> <pr>). Nothing outside the workflow called it.

count_diff now takes two explicit tree paths instead of the MFC_PR env var, reports only files whose count moved, adds a per-directory rollup, and with --markdown writes a GitHub-flavored table (empty when nothing changed). The workflow moves to pull_request_target so fork PRs can be commented on, and posts via sticky-pull-request-comment. It counts against the merge base rather than the tip of the base branch, so a PR is only charged for the lines it actually adds. The counter also recurses into src/*/include, which the old glob silently skipped.
Key the concurrency group on the PR number rather than github.head_ref, which is a branch name and collides across forks (two PRs from branches both named master would cancel each other, silently dropping one comment). Skip non-regular files and read with errors=replace, since under pull_request_target the counted tree is attacker-controlled: a symlinked src/*.fpp pointing at /dev/zero would otherwise read until the runner died, and one non-UTF-8 byte in a comment would fail the job. Take the comment back down when the diff returns to zero instead of leaving a stale number, which also means the workflow can no longer be filtered on src/**: a PR that stops touching src/ has to still run in order to be cleaned up.
Copilot AI lite review requested due to automatic review settings August 23, 2026 21:50
Comment thread .github/workflows/line-count.yml Dismissed
@github-actions

Copy link
Copy Markdown

Claude Code Review

Head SHA: 9827831

Files changed:

  • 4
  • .github/workflows/line-count.yml
  • toolchain/mfc/args.py
  • toolchain/mfc/cli/commands.py
  • toolchain/mfc/count.py

Findings:

  • toolchain/mfc/count.py:16-22: handle_dir now walks the PR's checkout tree recursively (**/*.*f*) and guards with os.path.isfile(filepath), with a comment claiming this skips "anything that is not a regular file." os.path.isfile() follows symlinks, so it returns True for a symlink whose target is a regular file anywhere on the runner's filesystem. Since line-count.yml now runs under pull_request_target and checks out the fork's PR content verbatim into pr/ (fully attacker-controlled), a malicious PR could add a symlink under src/<codedir>/ (e.g. src/common/x.f90 -> /etc/passwd or a path elsewhere on the runner) and have its line count silently included in the generated markdown table and posted back to the PR via sticky-pull-request-comment, leaking size/existence info about files outside the checked-out tree. The added regular-file check does not actually prevent this — it should use os.path.islink() (or compare os.path.realpath against the tree root) to reject symlinks, not just non-regular files.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a GitHub Actions workflow that posts a single sticky pull request comment showing per-file and per-directory line-of-code (LOC) deltas, making ./mfc.sh count_diff output visible directly on PRs (including from forks) by using pull_request_target and comparing against the merge base.

Changes:

  • Updated ./mfc.sh count_diff to compare two explicit source-tree paths and optionally emit GitHub-flavored Markdown for CI commenting.
  • Extended LOC counting to recurse into src/*/** (including include/) and harden reading behavior for CI by skipping non-files and using errors="replace".
  • Added a pull_request_target workflow that computes the merge-base diff and posts/removes a sticky PR comment accordingly.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
toolchain/mfc/count.py Adds two-tree diffing, Markdown output for CI, recursive counting, and untrusted-tree handling.
toolchain/mfc/cli/commands.py Updates the CLI schema for count_diff to accept positional tree paths and a --markdown output option.
toolchain/mfc/args.py Adjusts parser default-population list to exclude count_diff.
.github/workflows/line-count.yml Implements the PR sticky comment workflow using merge-base comparison under pull_request_target.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread toolchain/mfc/count.py
Comment thread toolchain/mfc/count.py Outdated
os.path.isfile follows symlinks, so the previous guard rejected /dev/zero but happily counted src/common/x.f90 -> /etc/passwd, leaking a file's existence and size into a public PR comment. Worse, glob's ** recurses into symlinked directories, so a single src/common/link -> /etc pulled in every file beneath it, none of which are themselves symlinks and none of which an islink check would catch. Require instead that each path resolve to a regular file inside the tree being counted. Read lazily rather than via f.read(), so a large file in the tree costs one line of memory instead of all of it. Materialize the PR tree with git worktree from refs/pull/N/head in the base clone rather than with actions/checkout, which removes the checkout of untrusted code that CodeQL flags in this privileged workflow. Also say comment lines are excluded in the status line, which they always were.
@sbryngelson
sbryngelson merged commit 17c16b6 into master Aug 23, 2026
83 of 86 checks passed
@sbryngelson
sbryngelson deleted the ci/loc-diff-comment branch August 23, 2026 22:19
@codecov

codecov Bot commented Aug 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.61%. Comparing base (e556597) to head (cfdfabc).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1753   +/-   ##
=======================================
  Coverage   61.61%   61.61%           
=======================================
  Files          84       84           
  Lines       21520    21520           
  Branches     3180     3180           
=======================================
  Hits        13259    13259           
  Misses       6082     6082           
  Partials     2179     2179           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants