Update cuda-core and cuda-pathfinder dependency versions automatically in cuda-python

[mdboom]

We currently determine the cuda-bindings version automatically to be the latest release whenever we build cuda-python. We don't currently determine the cuda-core or cuda-pathfinder versions automatically, which means we they need to be updated by hand during the release. This just automates out that one extra step. This isn't theoretically -- the cuda-core dependency is still specified as 1.0.0 here, even though 1.1.1 has been out for weeks.

This specifies the dependencies so that patch releases of the same version are considered equivalent. For cuda-pathfinder, it is looser and even minor release changes are considered equivalent. (This matches current behavior).

As an additional bugfix, this will no longer generate a cuda-python package that depends on a non-tagged git commit of any package (which we will never release, so doesn't make much sense).

(There is a larger question about whether we should be so tight with our cuda-python dependencies, which leads to other issues downstream of this, but this is just to automate a part of making a release while keeping status quo behavior.)

[github-actions]

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-2709/
https://nvidia.github.io/cuda-python/pr-preview/pr-2709/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-2709/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-2709/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

[leofang]

Would it be possible to leave paper trails on the dependency versions in some way, so that we can inspect and audit? Automating it is great but right now we lose track of things 😛

[rwgk]

Hi @mdboom, I'm still working on getting my head around this PR.

I'm posting the codex gpt-5.6-sol ultra findings here as generated, so you can run them by your agent while I look more closely manually. (Usually the findings from this model are very reliable.)

---

PR 2709 initial review

• Reviewed head: 027f16c

Severity convention: P1 findings should be fixed before merge. P2 findings should be fixed or explicitly accepted and documented.

Findings

P1 - Preserve the cuda-python distribution version outside the component loop

Location: cuda_python/setup.py:35, cuda_python/setup.py:48

version is reassigned on every loop iteration. Because cuda-pathfinder is last, the value passed to setup(version=...) is the pathfinder SCM version rather than the unprefixed cuda-python version.

The PR CI artifact confirms the result:

cuda_python-1.7.0.post12+g027f16c9d-py3-none-any.whl

Name: cuda-python
Version: 1.7.0.post12+g027f16c9d

The expected unprefixed SCM value at the same commit is 13.3.1.post248+g027f16c9d. On a release tag, the wrong distribution version would fail the release check at ci/tools/validate-release-wheels:108, which requires the wheel version to equal the requested v13.x release version. Without that validation, it could publish the metapackage under a pathfinder version.

Recommended fix: compute the cuda-python distribution version once, store each component version under a distinct name, and pass only the unprefixed value to setup(). Do not fix this merely by reordering the loop; that leaves the same bug latent.

P1 - Preserve independently derived requirements when rebuilding from an sdist

Location: cuda_python/setup.py:17, cuda_python/setup.py:35, .github/workflows/test-sdist-linux.yml:72, .github/workflows/test-sdist-windows.yml:82

The three get_version_for_module() calls work in a full Git checkout because each call can inspect a different tag namespace. That information is no longer available when a wheel is built from the generated sdist. Setuptools-scm falls back to the sdist's single top-level PKG-INFO version for every call.

I reproduced the same round trip as the Linux and Windows sdist jobs. The sdist itself contains the source-tree-derived requirements:

Version: 1.7.0.post12+g027f16c9d
Requires-Dist: cuda-bindings~=13.3.1
Requires-Dist: cuda-core~=1.1.1
Requires-Dist: cuda-pathfinder~=1.7
Requires-Dist: cuda-bindings[all]~=13.3.1; extra == "all"

The wheel built from that sdist contains:

Version: 1.7.0.post12+g027f16c9d
Requires-Dist: cuda-bindings~=1.7.0
Requires-Dist: cuda-core~=1.7.0
Requires-Dist: cuda-pathfinder~=1.7
Requires-Dist: cuda-bindings[all]~=1.7.0; extra == "all"

cuda-bindings~=1.7.0 and cuda-core~=1.7.0 make the resulting wheel unresolvable against the released package versions. The CI jobs remain green because pip wheel --no-deps checks only whether a wheel can be constructed; they do not inspect its metadata.

This is independent of the first finding. After correcting the distribution version to 13.x, the fallback would instead generate cuda-core~=13.x and cuda-pathfinder~=13.x, which is still wrong.

Recommended fix: resolve the independent component requirements while Git metadata is present, persist the resulting requirement map in the sdist, and reuse it when building without Git. Add an integration assertion that the direct wheel, sdist PKG-INFO, and wheel-from-sdist have identical Version and Requires-Dist fields.

P1 - Keep the official git archive release sources buildable

Location: cuda_python/setup.py:35, .github/workflows/release-upload.yml:59, CONTRIBUTING.md:59

The project documents git archive tarballs as buildable, and the release workflow publishes one for each release. The archival substitution records refs attached to the archived commit, but it cannot recover the nearest historical tags in all three independent namespaces.

I reproduced the release workflow with PR HEAD tagged as a temporary bare v13.4.0, created the archive with git archive, extracted it, and ran cuda_python/setup.py --version. The unprefixed lookup can use v13.4.0; the next lookup cannot find a historical cuda-core-v* tag and aborts:

ValueError: Can't parse version from tag '0.0'
(tag_regex='^cuda-core-(?P<version>v\\d+\\.\\d+\\.\\d+(?:[ab]\\d+)?)', tag_prefix='')

This regresses the explicit guarantee at CONTRIBUTING.md:61 and makes the source archive uploaded by .github/workflows/release-upload.yml unusable for building cuda-python.

Recommended fix: make the component-version manifest proposed for the sdist path available to official release archives as well. A named setuptools-scm override can be a useful manual fallback, but it does not by itself make the published archive self-contained. Add a test that creates a git archive from a synthetic bare metapackage tag and builds the metapackage from the extracted tree.

P1 - Retain alpha and beta suffixes in dependency requirements

Location: cuda_python/setup.py:21, cuda_python/setup.py:36, cuda_python/setup.py:41, cuda_python/setup.py:44

The tag regex intentionally captures aN and bN, but Version(version).base_version discards them. The code then emits ~= unconditionally.

For example:

SCM version:                       13.4.0b1
Version("13.4.0b1").base_version: 13.4.0
Generated requirement:             cuda-bindings~=13.4.0

cuda-bindings~=13.4.0 excludes cuda-bindings==13.4.0b1. Because the final 13.4.0 release may not exist yet, a cuda-python 13.4.0b1 metapackage cannot install its matching bindings. The repository has a real v13.4.0b1 tag, and the pre-PR code exact-pins prereleases as cuda-bindings==13.4.0b1.

The same loss applies if the selected core or pathfinder tag is an alpha or beta, and the cuda-bindings[all] extra is affected too.

Recommended fix: remove only SCM-generated distance/hash portions while retaining the prerelease segment. Preserve the existing exact-pin policy for prereleases unless the compatibility policy is intentionally changed, documented, and tested.

P2 - Separate the metapackage version scheme from dependency tag selection

Location: cuda_python/setup.py:23

After fixing the loop clobber, the shared helper would still change the version of every untagged cuda-python build:

Before this PR: 13.3.2.dev248+g027f16c9d
With this PR:   13.3.1.post248+g027f16c9d

The post-release scheme is useful for deriving a dependency from the last released tag without inventing a next-patch version. It is not required for the distribution's own version, and a postrelease is not semantically equivalent to a development release. The PR description calls for avoiding untagged dependency versions while otherwise preserving behavior.

Recommended fix: retain the existing setuptools-scm scheme for the cuda-python distribution version and use a separate tag-only or post-release derivation path for component requirements. If the postrelease change is intentional for the metapackage itself, document it and add an ordering test.

P2 - Add tests for the metadata contract changed by this PR

Location: .github/workflows/build-wheel.yml:423, .github/workflows/test-sdist-linux.yml:69

The PR changes every dynamic packaging field in cuda-python but adds no tests. twine check validates metadata syntax, not semantic correctness, and the sdist jobs only require construction to succeed. That is why both a 1.7.x cuda-python wheel and a wheel with cuda-bindings~=1.7.0 pass their build jobs.

At minimum, add coverage that:

1. Captures setup() arguments and verifies that the unprefixed SCM version remains the distribution version.
2. Parameterizes exact final, untagged final, alpha, beta, and intentionally stripped RC tags.
3. Places component tags on different commits and verifies all three requirements plus the all extra.
4. Compares Version and every Requires-Dist entry across a direct wheel, sdist PKG-INFO, and wheel rebuilt from that sdist.
5. Builds from a git archive containing only the bare metapackage release tag.
6. Runs ci/tools/validate-release-wheels against representative final and prerelease artifacts.

Open questions and assumptions

• I assume sdist and git archive builds remain supported. CI explicitly exercises the former, while CONTRIBUTING.md and the release workflow explicitly support the latter.
• I assume alpha and beta builds should keep the pre-PR exact-pin behavior. The regex comment says those suffixes are preserved, and v13.4.0b1 is an actual release tag.
• The manual get_version() calls do not provide dist_name, so independently named SETUPTOOLS_SCM_PRETEND_VERSION_FOR_* overrides cannot currently supply the three component values. Passing names would improve the documented no-history escape hatch, but persisted release metadata is still needed for self-contained sdists and archives.

Verification performed

• Refreshed mdboom/cuda-core-version-update and verified local HEAD, the fetched fork ref, and the live GitHub PR all point to 027f16c9d17b3112c30a01de843a98a4d9da0d7f.
• Ran git diff --check on the PR diff; it passed.
• Inspected the PR CI wheel and its METADATA.
• Built a cuda-python sdist and rebuilt a wheel from it using Python 3.12, setuptools 84.0.0, setuptools-scm 10.2.1, and packaging 26.3.
• Reproduced the official git archive workflow at a temporary v13.4.0 tag.
• Mocked independent alpha/beta component versions and captured the resulting setup() arguments.
• Confirmed there are no existing tests for cuda_python/setup.py, the generated distribution version, or Requires-Dist equality across artifact types.

No other distinct correctness findings emerged from the one-file diff.

[mdboom]

P1 - Preserve the cuda-python distribution version outside the component loop

This is a real bug. Sorry I missed that in my testing.

P1 - Preserve independently derived requirements when rebuilding from an sdist
P1 - Keep the official git archive release sources buildable

These seems like two sides of the same coin. I'll see what can be done with that.

P1 - Retain alpha and beta suffixes in dependency requirements

Seems worth fixing if we are going to do beta releases of cuda-python. I don't think we should, but it's a latent bug if we ever decide to. If this proves too problematic to get right, we can just error out for this case as an alternative.

P2 - Separate the metapackage version scheme from dependency tag selection

This only matters if we release cuda-python from a non-tagged commit which isn't currently supported on a number of levels.

P2 - Add tests for the metadata contract changed by this PR

This seems out of scope. Our CI needs better testing across the board, but we won't find a good solution while trying to fix this localized problem.

[mdboom]

Closing. There seems to be no reasonable way to solve this problem without making a bigger problem.