ci(macrobenchmark): Compare the startup benchmark against its merge base (JAVA-679)

[runningcode]

📜 Description

Follow-up to #5909, which got the cold-start macrobenchmark running on a Sauce Labs real device and recovering benchmarkData.json through logcat. That reports absolute numbers for one build, which are close to unreadable on a cloud device with unlocked CPU clocks.

This measures a delta against the PR's base, under identical conditions:

• The workflow builds sentry-samples-android twice — once from refs/pull/N/merge (the PR merged into its base) and once from the base commit alone. The baseline gets applicationId suffix .baseline from a new sampleAppIdSuffix Gradle property, so both install side by side.
• Both APKs go to the same device (the baseline via Sauce otherApps), and SentryStartupBenchmark becomes a @Parameterized run that alternates between the two packages. Device, OS, thermal state and clock behaviour are common to both arms and cancel in the difference.
• Because the candidate is the merge, anything that landed on the base since the PR forked is present in both arms and cancels too — the delta is the PR's own contribution.
• The order is ABBA, not ABAB. In ABAB the candidate always follows the baseline, so drift within a pair is charged to the candidate every time. Mirroring each round makes each variant the trailing one equally often.
• scripts/parse-macrobenchmark-log.py pools iterations per build, recomputes the statistics (each step's own median over 3 iterations says nothing), and emits a base-vs-PR table with Δ, Δ% and a permutation test on the difference of medians. It posts to the job summary and to a sticky PR comment.

Runs on every PR. One run costs two Gradle builds and up to an hour of a Sauce device at concurrency 1, so the concurrency group cancels a PR's in-flight run when a new push arrives. If the aggregate spend proves too high, a paths filter or a label gate on the job puts a lid on it without touching anything else. Fork PRs get no secrets so the Sauce steps skip; pull_request_target is deliberately not used, since it would run a fork's build scripts with our Sauce credentials in scope.

Two Sauce settings are load-bearing, and are documented in the module README:

• appSettings.resigningEnabled: false — Sauce resigns the app under test on real devices but never touches otherApps. Left on, the candidate would carry an injected agent the baseline does not, and every delta would include the cost of that agent.
• espresso.otherApps — dependent apps are installed without being instrumented or modified.

Trade-offs, stated rather than hidden: this roughly doubles the Gradle build, and takes the device from 12 to 24 cold starts, which will thermally throttle it. That is acceptable only because the arms alternate, so throttling lands on both. Absolute numbers get worse; the delta gets trustworthy.

💡 Motivation and Context

The module README already prescribed interleaved A/B as the way to evaluate an SDK-init change, but left it as a manual chore ("build variant A, run, build variant B, run, repeat"). This makes CI do it.

It is worth reading alongside #5972, which removed the old benchmark workflow because "the numbers were not actionable ... the startup diff swung between 22 ms and 143 ms across unrelated changes". That workflow also ran on every PR, so the difference here is not the trigger — it is that this one alternates both builds on a single device in a single session and quantifies the result, rather than comparing across separate runs and printing a bare diff. Whether the numbers earn their cost is the thing to judge once the first runs land.

• resolves: JAVA-679

💚 How did you test it?

Verified locally where possible; the Sauce path cannot run until this is on main (see Next steps).

• Both APKs built locally and confirmed distinct: aapt2 dump badging reads the baseline as io.sentry.samples.android.baseline, and it keeps <profileable android:shell="true"> so Macrobenchmark can measure it. A plain build stays io.sentry.samples.android, and the debug variant is unchanged at io.sentry.samples.android.debug — the property doesn't leak into ordinary builds.
• compileBenchmarkKotlin and assembleBenchmark pass, and the new <queries> entries survive the manifest merge alongside androidx's own.
• Both JUnit behaviours the ABBA design depends on, checked against JUnit 4.13.2 rather than assumed: Parameterized runs parameter sets in the order @Parameters returns them, and @AfterClass fires once, not per parameter.
• The parser against synthetic device.log fixtures in five shapes: A/B, single-variant fallback, plain adb logcat format, repeated growing dumps, and a truncated final dump (errors cleanly instead of splicing two dumps together). An injected +15 ms gave p=0.000 on SentryAndroid.init but p=0.309 on timeToInitialDisplay — which is exactly why the trace-section metric exists.
• The applicationId collision guard, exercised in both directions: distinct ids pass, identical ids fail with the explanatory error.

📝 Checklist

• I added GH Issue ID & Linear ID
• I added tests to verify the changes.
• No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled.
• I updated the docs if needed.
• I updated the wizard if needed.
• Review from the native team if needed.
• No breaking change or entry added to the changelog.
• No breaking change for hybrid SDKs or communicated to hybrid SDKs.
• Public API changes reviewed by another Mobile SDK team member or implemented according to the develop docs spec.

🔮 Next steps

This PR cannot benchmark itself. The baseline is built from the base commit, and main does not yet carry the sampleAppIdSuffix property — so both builds would produce the same package. Rather than silently comparing the app against itself and reporting a delta of ~0, the staging step detects the collision and fails with an explicit error. The first real run is therefore the next labelled PR after this merges.

Then, in order:

1. A/A null test — any PR that touches no SDK or sample code. Δ should be ≈0; anything else means the harness is biased. It also gives the empirical noise floor, i.e. the smallest regression this setup can resolve. Worth recording in the module README once known.
2. Sensitivity — a scratch PR with a deliberate delay inside the SentryAndroid.init trace section, to confirm the harness catches a real regression and attributes it to the right metric.
3. Confirm the sticky comment updates in place across pushes rather than duplicating.

If resigningEnabled: false turns out not to work on our Sauce account, re-enable it and make the report state which mode it ran in, so the constant bias is visible rather than silent.

#skip-changelog

[linear-code]

JAVA-679

[sentry]

📲 Install Builds

Android

🔗 App Name	App ID	Version	Configuration
SDK Size	io.sentry.tests.size	8.53.0 (1)	release

⚙️ sentry-android Build Distribution Settings