Skip to content

ci: PR smoke test for a11y-scan SPM plugin (end-to-end scan) - #35

Draft
Crash0v3rrid3 wants to merge 2 commits into
mainfrom
chore/spm-pr-smoke-test
Draft

ci: PR smoke test for a11y-scan SPM plugin (end-to-end scan)#35
Crash0v3rrid3 wants to merge 2 commits into
mainfrom
chore/spm-pr-smoke-test

Conversation

@Crash0v3rrid3

@Crash0v3rrid3 Crash0v3rrid3 commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

What

Adds .github/workflows/spm-smoke-test.yml — a GitHub Actions workflow that smoke-tests the a11y-scan SwiftPM command plugin end-to-end on every PR.

On a macOS runner it:

  1. cd tests/spm && swift build — compiles the a11y-scan command plugin (via the path dependency) and the sample sources. (The repo root is a plugin-only package with no buildable target, so it is not built directly.)
  2. cd tests/spm && swift test with RUN_A11Y_SCAN=1 + credentials — this un-gates the repo's existing testA11yScanPluginRuns, which runs scripts/run-a11y-scan.sh --non-strict (a real scan: downloads the BrowserStack CLI, authenticates, scans the sample SwiftUI sources with intentional a11y issues) and asserts exit 0.

It reuses the harness the repo already ships (tests/spm/) rather than duplicating the swift package plugin … scan invocation, so there's a single source of truth for how the plugin is run.

Design notes

  • Fork-safe. The scan needs BROWSERSTACK_USERNAME / BROWSERSTACK_ACCESS_KEY, and secrets are never exposed to fork PRs. The job is gated with github.event.pull_request.head.repo.fork == false (plus workflow_dispatch for manual runs). Fork PRs skip the job instead of failing.
  • Degrades gracefully. If the secrets aren't configured yet, testA11yScanPluginRuns XCTSkips and the job still passes (build + unit test only) — so merging this doesn't turn the tree red before secrets are added.
  • --non-strict is used by the underlying test, so planted a11y issues in the sample views don't fail the run — a clean exit means the whole download → auth → scan pipeline worked.
  • actions/checkout is pinned by SHA (v4.2.2) to match Semgrep.yml / verify-selfupdate-checksums.yml (supply-chain hardening, DEVA11Y-476).

Verified locally (macOS, Swift 6.2)

  • tests/spm swift buildCompiling plugin a11y-scan + sample sources, build complete.
  • tests/spm swift test (no RUN_A11Y_SCAN) → testLibraryIdentity passes, testA11yScanPluginRuns skips, exit 0. Confirms the no-secrets path stays green.
  • The full authenticated scan path was not run locally (no creds on this machine); it exercises on CI once the secrets are set.

⚠️ Required before this actually runs the scan

Add two repo secrets (Settings → Secrets and variables → Actions):

  • BROWSERSTACK_USERNAME
  • BROWSERSTACK_ACCESS_KEY

Until then the job passes but the e2e scan self-skips.

Notes / open questions

  • Covers the SwiftPM harness (tests/spm/). The tests/xcode-app/ build-phase harness is not wired up here — happy to add an Xcode job in a follow-up if you want CI to cover that integration path too.
  • No DEVA11Y ticket linked yet — branch is chore/spm-pr-smoke-test; rename/link as needed.

🤖 Generated with Claude Code

Crash0v3rrid3 and others added 2 commits August 3, 2026 18:22
Runs an end-to-end accessibility scan on every PR: builds the plugin and
executes a real scan against the tests/spm harness (sample SwiftUI sources
with intentional a11y issues), reusing the repo's own gated integration
test (testA11yScanPluginRuns) so the invocation stays in one place.

The scan downloads the BrowserStack CLI and makes authenticated calls, so
it is gated to same-repo PRs (secrets are never exposed to fork PRs) and
manual dispatch. Without the BROWSERSTACK_USERNAME / BROWSERSTACK_ACCESS_KEY
secrets configured the e2e test XCTSkips and the job still passes.

actions/checkout pinned by SHA to match existing workflows (DEVA11Y-476).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The repo root is a plugin-only package with no buildable target, so
`swift build` there fails ("does not contain a buildable target").
Building the tests/spm harness compiles the a11y-scan command plugin via
the path dependency plus the sample sources, so use that as the build step.

Verified locally on Swift 6.2: tests/spm `swift build` compiles the plugin,
and `swift test` passes with the e2e scan test skipping when RUN_A11Y_SCAN
is unset.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant