Test-PSBuildScriptAnalysis does not fail on the PSScriptAnalyzer rule crash from #136

[tablackburn]

Follow-up to #136. The root cause analysis for that issue established that
PSScriptAnalyzer executes its script rules in parallel against a process-wide,
unsynchronised Helper singleton, so concurrent rules can observe half-updated
PowerShell session state and crash. That is an upstream defect
(PSScriptAnalyzer#1538, #1351, both open and deferred
to 2.0).

#136 covers this repository's own Analyze task. This issue covers the shipped
module, which has the same exposure but behaves differently when it hits.

Where it bites

Test-PSBuildScriptAnalysis is the single analyzer call site in the shipped
module. Both task runners reach it, so every consumer is exposed:

• PowerShellBuild/psakeFile.ps1 line 131
• PowerShellBuild/IB.tasks.ps1 line 68

Driving that function directly in a cold fresh-process loop reproduces the race,
attributed to its own line 41:

Invoke-ScriptAnalyzer: .../Public/Test-PSBuildScriptAnalysis.ps1:41
  41 | … $analysisResult = Invoke-ScriptAnalyzer @invokeScriptAnalyzerParameters …
     | Object reference not set to an instance of an object.

The problem

Measure	Result
Race fired	3 / 100 (3%)
Function threw	0 / 100

Line 41 calls Invoke-ScriptAnalyzer without -ErrorAction Stop. This
repository only ever saw a hard failure because build.ps1 sets
$ErrorActionPreference = 'Stop' for the whole process.

So the behaviour a consumer gets depends entirely on their own preference:

• Sets ErrorActionPreference = 'Stop' — same flaky red build as CI: Intermittent PSScriptAnalyzer crash in the Analyze task #136.
• Leaves it at the default — the rule crashes, the error is written to the error
  stream, and Test-PSBuildScriptAnalysis continues to its severity-threshold
  check and returns normally. The build goes green.

The second case is the reason for this issue: the function has no way to react to
a failure it never observes, so the retry added for #136 cannot simply be copied
here.

What was ruled out

The obvious worry is that a crashed rule costs a finding, letting a build pass on
an incomplete analysis. That was tested and did not happen. Over 100 cold runs
against a target with a known-good count of 5 findings:

Observation	Runs
5 findings, no race	95
5 findings, race fired	5
Fewer than 5 findings	0

Every run that raced still returned the full finding set — PSScriptAnalyzer
isolates a rule failure rather than discarding other rules' diagnostics.

A rule does genuinely fail to complete, so losing a finding remains mechanically
possible if a crash lands on a rule that had something to report on that input.
That was not observed and is not being claimed.

This lowers the severity considerably: consumers who set Stop get a flaky
build, and everyone else gets a noisy but correct result.

Possible approaches (not yet decided)

Preferred: retry on a RULE_ERROR observed via -ErrorVariable, leaving the
existing failure semantics alone. This removes the flake for the consumers who
actually suffer it, without changing what happens to anyone else. It is the only
option here that helps someone without also costing someone something.

Alternative: add -ErrorAction Stop and wrap the call in the same bounded
retry used for #136. More conventional, and it makes the two call sites
symmetrical. But it is a user-facing behaviour change — a crash that survives
every attempt becomes a hard failure where consumers previously got a pass — so
it needs a CHANGELOG.md entry and a deliberate decision.

Not recommended: document it and move on. Worth stating explicitly why this
is weaker than it first looks. "The result is still correct, only the log is
noisy" is true for consumers on the default preference, but it is not true for
consumers who set ErrorActionPreference = 'Stop' — a common thing to do in a
build script, and what this repository itself does. Those consumers get exactly
the flaky red build that #136 is about, with no mitigation, in a module whose
whole purpose is to be someone else's build. At least one has
reported hitting it across their own repositories. Doing
nothing leaves the module shipping a known flake to the people most exposed to it.

Reproducing

Self-contained; no branch or extra tooling required. Roughly 3% of analyzer runs,
so expect a few dozen iterations.

./build.ps1 -Task Build

$version  = (Import-PowerShellDataFile ./PowerShellBuild/PowerShellBuild.psd1).ModuleVersion
$target   = "./Output/PowerShellBuild/$version"
$settings = './PowerShellBuild/ScriptAnalyzerSettings.psd1'

Import-Module "$target/PowerShellBuild.psd1" -Force

foreach ($iteration in 1..400) {
    $analysisErrors = @()
    Test-PSBuildScriptAnalysis -Path $target -SeverityThreshold 'Error' -SettingsPath $settings -ErrorVariable analysisErrors *> $null

    $raced = @($analysisErrors) | Where-Object { $_.FullyQualifiedErrorId -like 'RULE_ERROR*' }
    if ($raced) {
        "Rule crashed on iteration $iteration : $($raced[0].Exception.Message)"
        'Test-PSBuildScriptAnalysis returned normally anyway - reaching this line proves it did not throw.'
        break
    }
}

Observed output:

Rule crashed on iteration 39 : Collection was modified; enumeration operation may not execute.
Test-PSBuildScriptAnalysis returned normally anyway - reaching this line proves it did not throw.

Rates were measured on Linux with PowerShell 7.6.4 and PSScriptAnalyzer 1.25.0.
The mechanism is engine-level rather than platform-specific — #136 has been seen
on both windows-latest and ubuntu-latest — but the rate was not measured on
Windows PowerShell 5.1.