ci: PR smoke test for a11y-scan SPM plugin (end-to-end scan) - #35
Draft
Crash0v3rrid3 wants to merge 2 commits into
Draft
ci: PR smoke test for a11y-scan SPM plugin (end-to-end scan)#35Crash0v3rrid3 wants to merge 2 commits into
Crash0v3rrid3 wants to merge 2 commits into
Conversation
Runs an end-to-end accessibility scan on every PR: builds the plugin and executes a real scan against the tests/spm harness (sample SwiftUI sources with intentional a11y issues), reusing the repo's own gated integration test (testA11yScanPluginRuns) so the invocation stays in one place. The scan downloads the BrowserStack CLI and makes authenticated calls, so it is gated to same-repo PRs (secrets are never exposed to fork PRs) and manual dispatch. Without the BROWSERSTACK_USERNAME / BROWSERSTACK_ACCESS_KEY secrets configured the e2e test XCTSkips and the job still passes. actions/checkout pinned by SHA to match existing workflows (DEVA11Y-476). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The repo root is a plugin-only package with no buildable target, so
`swift build` there fails ("does not contain a buildable target").
Building the tests/spm harness compiles the a11y-scan command plugin via
the path dependency plus the sample sources, so use that as the build step.
Verified locally on Swift 6.2: tests/spm `swift build` compiles the plugin,
and `swift test` passes with the e2e scan test skipping when RUN_A11Y_SCAN
is unset.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
.github/workflows/spm-smoke-test.yml— a GitHub Actions workflow that smoke-tests thea11y-scanSwiftPM command plugin end-to-end on every PR.On a macOS runner it:
cd tests/spm && swift build— compiles thea11y-scancommand plugin (via the path dependency) and the sample sources. (The repo root is a plugin-only package with no buildable target, so it is not built directly.)cd tests/spm && swift testwithRUN_A11Y_SCAN=1+ credentials — this un-gates the repo's existingtestA11yScanPluginRuns, which runsscripts/run-a11y-scan.sh --non-strict(a real scan: downloads the BrowserStack CLI, authenticates, scans the sample SwiftUI sources with intentional a11y issues) and asserts exit 0.It reuses the harness the repo already ships (
tests/spm/) rather than duplicating theswift package plugin … scaninvocation, so there's a single source of truth for how the plugin is run.Design notes
BROWSERSTACK_USERNAME/BROWSERSTACK_ACCESS_KEY, and secrets are never exposed to fork PRs. The job is gated withgithub.event.pull_request.head.repo.fork == false(plusworkflow_dispatchfor manual runs). Fork PRs skip the job instead of failing.testA11yScanPluginRunsXCTSkips and the job still passes (build + unit test only) — so merging this doesn't turn the tree red before secrets are added.--non-strictis used by the underlying test, so planted a11y issues in the sample views don't fail the run — a clean exit means the whole download → auth → scan pipeline worked.actions/checkoutis pinned by SHA (v4.2.2) to matchSemgrep.yml/verify-selfupdate-checksums.yml(supply-chain hardening, DEVA11Y-476).Verified locally (macOS, Swift 6.2)
tests/spmswift build→Compiling plugin a11y-scan+ sample sources, build complete.tests/spmswift test(noRUN_A11Y_SCAN) →testLibraryIdentitypasses,testA11yScanPluginRunsskips, exit 0. Confirms the no-secrets path stays green.Add two repo secrets (Settings → Secrets and variables → Actions):
BROWSERSTACK_USERNAMEBROWSERSTACK_ACCESS_KEYUntil then the job passes but the e2e scan self-skips.
Notes / open questions
tests/spm/). Thetests/xcode-app/build-phase harness is not wired up here — happy to add an Xcode job in a follow-up if you want CI to cover that integration path too.chore/spm-pr-smoke-test; rename/link as needed.🤖 Generated with Claude Code