Skip to content

Add Python 3.13 support - #3465

Open
bochinski wants to merge 4 commits into
flyteorg:masterfrom
bochinski:feat/python-3.13-3.14-support
Open

Add Python 3.13 support#3465
bochinski wants to merge 4 commits into
flyteorg:masterfrom
bochinski:feat/python-3.13-3.14-support

Conversation

@bochinski

@bochinski bochinski commented Sep 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Tracking issue

Closes flyteorg/flyte#6993

Why are the changes needed?

flytekit declares requires-python = ">=3.10,<3.13" and cannot be installed on Python 3.13. This PR adds 3.13 to the supported and tested versions. Python 3.14 is out of scope: flyteidl declares requires-python <3.14, so flytekit cannot be installed on 3.14 regardless of this repo.

What changes were proposed in this pull request?

  • pyproject.toml: requires-python = ">=3.10,<3.14", 3.13 classifier. uv.lock regenerated (not used by CI); the lock was also behind pyproject.toml on master, so the diff additionally catches up flyteidl 1.16.4 -> 1.16.8 and the fsspec/gcsfs/s3fs bounds.
  • PythonVersion enum and default image prefixes: add 3.13 (core, sqlalchemy and openai-batch images). Without it DefaultImages.find_image_for() raises ValueError on 3.13.
  • CI: PRs run 3.13 only (a single version, as before with 3.12), nightly runs 3.10–3.13. On 3.13 pandas<2 is excluded (no wheels). The plugin matrix mirrors the 3.12 exclusions (not verified on 3.13; several of the underlying issues are closed by now) and adds duckdb (python_requires<3.13); mlflow stays excluded because its tests need tensorflow, which on 3.13 requires protobuf 5+.
  • dev-requirements.in: tensorflow on >=3.12 and torch on >=3.11, and the 3.12 skip of the extras job is removed (Support Python 3.12 tensorflow/tensorflow#62003 and Pytorch for Python 3.12 not available pytorch/pytorch#110436 are closed). protobuf<5 (and types-protobuf<5) is kept on <3.13 only: tensorflow releases with 3.13 wheels need protobuf 5+, and the reason for the pin ([Housekeeping] Add support for protobuf version 5 in flytekit flyte#5448) was fixed upstream. 3.10 and 3.11 resolve to the same versions as before. On 3.12 every job that installs this file now also gets tensorflow 2.19 and torch 2.14 (with CUDA wheels on linux), and numpy is capped to 2.1.x by tensorflow 2.19.
  • Publish py3.13-* images.
  • pytorch extras: torch.load(..., weights_only=False) for nn.Module and checkpoint objects, where the keyword exists. The default flipped to True in torch 2.6 and cannot unpickle those (this is also the current nightly 3.11 failure on master). Tensors keep torch's default. weights_only=False executes pickle code from the blob, the same trust model as FlytePickle.

How was this patch tested?

Locally on Linux x86_64 with uv and docker.

  • 3.13: make unit_test 1970 + 25 passed, make test_serialization 78411 passed, make unit_test_extras 128 passed (tensorflow 2.21 / torch 2.14 / protobuf 7).
  • 3.12: make unit_test and make unit_test_extras pass (tensorflow 2.19 / torch 2.14 / protobuf 4.25).
  • dev-requirements.in resolves for 3.10–3.13 on linux and 3.12/3.13 on windows and macos; every plugin dry-run resolves on 3.13 except mlflow and duckdb.

Check all the applicable boxes

  • I updated the documentation accordingly. (no docs mention supported versions)
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Supersedes #3401 and #3368.

🤖 Generated with Claude Code

@bochinski
bochinski marked this pull request as draft September 3, 2026 16:26
@bochinski
bochinski force-pushed the feat/python-3.13-3.14-support branch from 1310769 to 564041e Compare September 3, 2026 16:35
@bochinski bochinski changed the title Add Python 3.13 support (and unblock 3.14) Add Python 3.13 support Sep 3, 2026
@bochinski @claude
Add Python 3.13 support
605e0b1 
- pyproject.toml: requires-python ">=3.10,<3.14", add the 3.13 classifier;
  regenerate uv.lock accordingly.
- PythonVersion enum and default image prefixes: add 3.13 in flytekit
  core, the sqlalchemy plugin and the openai batch plugin. Without it,
  DefaultImages.find_image_for() raises "(3, 13) is not a valid
  PythonVersion" on 3.13.
- CI (pythonbuild.yml): run 3.13 on pull requests and in the nightly
  matrix, enable the extras (tensorflow/pytorch) tests on 3.13, exclude
  pandas<2 and plugins whose dependencies do not support 3.13 yet.
- dev-requirements.in: tensorflow>=2.21 and torch>=2.6 on 3.13; lift the
  protobuf<5 dev pin on 3.13 only (tensorflow 2.21 needs protobuf>=6, and
  the issue behind the pin, flyteorg/flyte#5448, was fixed upstream).
- Publish py3.13 images in pythonpublish.yml and build_image.yml.
- pytorch extras: torch>=2.6 defaults torch.load(weights_only=True), which
  cannot load the whole nn.Module / checkpoint objects flytekit stores.
  Pass weights_only=False where the keyword exists. This is required for
  the extras tests on 3.13 and already fails in the nightly 3.11 run.

Closes flyteorg/flyte#6993

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: bochinski <30434375+bochinski@users.noreply.github.com>
@bochinski
bochinski force-pushed the feat/python-3.13-3.14-support branch from 564041e to 605e0b1 Compare September 3, 2026 16:52
bochinski and others added 3 commits September 3, 2026 20:23
@bochinski @claude
Run the extras tests on Python 3.12
51d5fa8 
The tensorflow and pytorch transformer tests were skipped on 3.12 because
neither library had 3.12 wheels when 3.12 support was added. Both upstream
issues have been closed since: tensorflow ships 3.12 wheels from 2.16
(tensorflow/tensorflow#62003) and pytorch from 2.2
(pytorch/pytorch#110436).

- dev-requirements.in: install tensorflow on >=3.12 and torch on >=3.11
  with a single line each instead of per-version lines. With the
  protobuf<5 pin that is kept on <3.13, 3.12 resolves to tensorflow 2.19
  and 3.13 to the latest release.
- pythonbuild.yml: drop the 3.12 skip of the extras step.

Effect on the resolved dev environments: 3.10, 3.11 and 3.13 are
unchanged. On 3.12 every job that installs dev-requirements.in now also
gets tensorflow 2.19 and torch 2.14 (including the CUDA wheels torch pulls
in on linux), and numpy is capped to 2.1.x by tensorflow 2.19.

Verified locally on 3.12 with tensorflow 2.19.1 / torch 2.14.0 /
protobuf 4.25.9: unit_test_extras 123 passed, unit_test 1970 + 25 passed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: bochinski <30434375+bochinski@users.noreply.github.com>
@bochinski @claude
Keep torch's weights-only loader for tensors
e3e5c1e 
Commit 605e0b1 passed weights_only=False to torch.load for every
pytorch transformer. Plain tensors load fine under torch's default
(weights_only=True since torch 2.6), so there is no reason to give up the
restricted unpickler for them. Only whole nn.Module objects and checkpoint
dicts, whose hyperparameters can be arbitrary objects, need full
unpickling.

- native.py: rename the helper to load_torch_object and give it a
  weights_only parameter; None keeps torch's default. The base transformer
  carries a WEIGHTS_ONLY class attribute (None), the module transformer
  sets it to False. The docstring states that weights_only=False executes
  pickle code from the blob, with the same trust model as FlytePickle.
- checkpoint.py: call load_torch_object(..., weights_only=False).
- tests: cover the keyword handling (default, explicit, and torch<1.13
  without the keyword), the per-transformer setting, and a module round
  trip through the full unpickler.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: bochinski <30434375+bochinski@users.noreply.github.com>
@bochinski @claude
CI: run only Python 3.13 on pull requests, fix 3.13 comments
750bea8 
- Pull requests run a single interpreter version again (the convention
  since flyteorg#3380), now 3.13 instead of 3.12. The nightly matrix still covers
  3.10 to 3.13. This halves the pull request matrix and keeps the heavier
  3.12 dev environment (tensorflow/torch) out of pull request runs.
- Reword the comment on the 3.13 plugin exclusions: they mirror the 3.12
  list but have not been verified on 3.13, several of the underlying
  issues are closed by now.
- The tensorflow releases with 3.13 wheels require protobuf>=5, not >=6
  (2.20 needs 5.28, 2.21 needs 6.31).
- dev-requirements.in: give types-protobuf the same version split as
  protobuf, so the stubs match the runtime on 3.13.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: bochinski <30434375+bochinski@users.noreply.github.com>
@bochinski
bochinski force-pushed the feat/python-3.13-3.14-support branch from 51bac2f to 750bea8 Compare September 3, 2026 18:35
@bochinski
bochinski marked this pull request as ready for review September 3, 2026 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kumare3 kumare3 kumare3 approved these changes
@wild-endeavor wild-endeavor Awaiting requested review from wild-endeavor wild-endeavor is a code owner
@pingsutw pingsutw Awaiting requested review from pingsutw pingsutw is a code owner
@cosmicBboy cosmicBboy Awaiting requested review from cosmicBboy cosmicBboy is a code owner
@samhita-alla samhita-alla Awaiting requested review from samhita-alla samhita-alla is a code owner
@davidmirror-ops davidmirror-ops Awaiting requested review from davidmirror-ops davidmirror-ops is a code owner
@machichima machichima Awaiting requested review from machichima machichima is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add Python 3.13 support to Flytekit

2 participants

@bochinski @kumare3