Part of #2722.
The CI: Coverage workflow has failed every night since Aug 15 (last full green: run 31757710811, Aug 14), with a single incidental pass on Aug 23.
Symptom
The "Combine Coverage and Deploy" job fails at the step named "Combine coverage data":
=== Combining coverage data ===
Combined 2 files
=== Generating HTML, XML, and text reports ===
Combined 2 files
Plugin 'Cython.Coverage.Plugin' did not provide a file reporter for 'cuda/bindings/_lib/cyruntime/cyruntime.pxi'.
##[error]Process completed with exit code 1.
On other nights the same error names a different file — e.g. run 32539916399 (Aug 22) and run 32676345017 (Aug 24) both report cuda/core/_memory/_location.pxd.
The step name is misleading. coverage combine succeeds — the log prints Combined 2 files. The failure is coverage html, four commands later in the same run: block, surfacing under the step's name because the shell is bash -e. This cost real time during diagnosis.
Root cause
Cython has two file types that are not standalone modules: .pxd (declaration/header) and .pxi (textually included into another file). Neither compiles to its own translation unit, so Cython.Coverage.Plugin cannot construct a FileReporter for them. When coverage.py is handed one, the plugin returns nothing and coverage.py treats that as fatal, aborting the whole report.
This is already known — .coveragerc carries an ignore list with the comment "Omits specific definition files that causes plugin errors". The problem is that it enumerates three files by hand:
omit =
*/windll.pxd
*/_lib/windll.pxd
*/_lib/utils.pxd
Two new files of that kind have since landed and were never added:
Nothing warns you when a new .pxd/.pxi lands. Every one is a latent break.
Why it passed once (Aug 23): whether a given file reaches the report depends on whether any of its traced lines actually executed that night, and which offender trips first depends on iteration order. This is a class of failure, not a single regression — which is why the timeline looks flaky and there is no single "breaking commit".
Fix
Replace the enumeration with a pattern, in [run]:
[run]
plugins = Cython.Coverage
core = ctrace
branch = False
relative_files = True
# Cython's plugin cannot build a FileReporter for declaration/include files.
# Pattern, not an enumeration: every new .pxd/.pxi otherwise breaks `coverage html`.
omit =
*/*.pxd
*/*.pxi
It must be [run], not [report]. Verified against coverage 7.15.4:
report_core.get_analysis_to_report() calls coverage._get_file_reporters(morfs) before applying the report_omit matcher, and the PluginError is raised inside that call (control.py:1051). A [report] omit entry is evaluated too late to suppress it.
- The two sections are independent —
report_omit does not fall back to run_omit — so moving the list would silently drop the protection that currently works for windll.pxd/utils.pxd.
- The existing three entries work precisely because they are in
[run]: they keep the files out of the measured set at collection time, and both collection jobs pass --cov-config=$REPO_ROOT/.coveragerc (coverage.yml:183-195, :430-449), so the data arrives at the combining runner already filtered.
Glob behaviour confirmed against coverage.files.GlobMatcher: */*.pxd and */*.pxi match both offenders and do not match .pyx or .py files.
Also worth doing
- Split the mega-step.
coverage combine, coverage html, coverage xml and coverage report should be separate steps so the failing command is named in the UI.
- Consider
coverage html -i as belt-and-braces — but only after the omit fix. On its own it would have hidden this for months.
Severity
Not release-blocking; no product code is affected and no test signal is lost. But coverage reporting has been dark since Aug 14, so any coverage regression in that window is invisible.
Part of #2722.
The
CI: Coverageworkflow has failed every night since Aug 15 (last full green: run 31757710811, Aug 14), with a single incidental pass on Aug 23.Symptom
The "Combine Coverage and Deploy" job fails at the step named "Combine coverage data":
On other nights the same error names a different file — e.g. run 32539916399 (Aug 22) and run 32676345017 (Aug 24) both report
cuda/core/_memory/_location.pxd.Root cause
Cython has two file types that are not standalone modules:
.pxd(declaration/header) and.pxi(textually included into another file). Neither compiles to its own translation unit, soCython.Coverage.Plugincannot construct aFileReporterfor them. When coverage.py is handed one, the plugin returns nothing and coverage.py treats that as fatal, aborting the whole report.This is already known —
.coveragerccarries an ignore list with the comment "Omits specific definition files that causes plugin errors". The problem is that it enumerates three files by hand:omit = */windll.pxd */_lib/windll.pxd */_lib/utils.pxdTwo new files of that kind have since landed and were never added:
cuda/core/_memory/_location.pxd— added 2026-08-13 in cuda.core: Add copy_batch to cuda.core.utils #2593 (29acb74b6cb)cuda/bindings/_lib/cyruntime/cyruntime.pxi— present since the cybind port Move runtime's internal and cython layers to cybind #2261 (c4a5647a085);included by_internal/runtime_linux.pyx:43andruntime_windows.pyx:43, so traced lines are attributed to the.pxipathNothing warns you when a new
.pxd/.pxilands. Every one is a latent break.Why it passed once (Aug 23): whether a given file reaches the report depends on whether any of its traced lines actually executed that night, and which offender trips first depends on iteration order. This is a class of failure, not a single regression — which is why the timeline looks flaky and there is no single "breaking commit".
Fix
Replace the enumeration with a pattern, in
[run]:It must be
[run], not[report]. Verified against coverage 7.15.4:report_core.get_analysis_to_report()callscoverage._get_file_reporters(morfs)before applying thereport_omitmatcher, and thePluginErroris raised inside that call (control.py:1051). A[report] omitentry is evaluated too late to suppress it.report_omitdoes not fall back torun_omit— so moving the list would silently drop the protection that currently works forwindll.pxd/utils.pxd.[run]: they keep the files out of the measured set at collection time, and both collection jobs pass--cov-config=$REPO_ROOT/.coveragerc(coverage.yml:183-195,:430-449), so the data arrives at the combining runner already filtered.Glob behaviour confirmed against
coverage.files.GlobMatcher:*/*.pxdand*/*.pximatch both offenders and do not match.pyxor.pyfiles.Also worth doing
coverage combine,coverage html,coverage xmlandcoverage reportshould be separate steps so the failing command is named in the UI.coverage html -ias belt-and-braces — but only after the omit fix. On its own it would have hidden this for months.Severity
Not release-blocking; no product code is affected and no test signal is lost. But coverage reporting has been dark since Aug 14, so any coverage regression in that window is invisible.