Split the runtime out of the Python bindings extension - #21610
Open
shoumikhin wants to merge 9 commits into
Open
Split the runtime out of the Python bindings extension#21610shoumikhin wants to merge 9 commits into
shoumikhin wants to merge 9 commits into
Conversation
Contributor
Author
|
Stack from ghstack (oldest at bottom): |
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21610
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New Failures, 11 Pending, 20 Unclassified FailuresAs of commit 2889647 with merge base 28a7fac ( NEW FAILURES - The following jobs have failed:
UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
This PR needs a
|
shoumikhin
added a commit
that referenced
this pull request
Aug 6, 2026
`pip install executorch` ships the C++ runtime welded inside the Python
bindings extension. That one file, `_portable_lib.so`, is about 11 MB and
contains the runtime, the operator registries, the CPU kernels, the XNNPACK
delegate, the profiler and several extensions, all statically linked together.
A Python user does not notice, but it has a correctness cost. Two other
extensions in the same wheel, the training bindings and the language model
runner, each carry their own private copy of the runtime. A process that loads
more than one of them ends up with more than one backend registry and more than
one thread pool. Backends register themselves from a static initialiser into a
table that is only process-wide if the runtime is loaded once, so a second copy
means a backend can land in a table nothing else reads.
This change builds each of those pieces once, as a shared library the wheel
ships, and makes the extensions link them instead of containing them:
executorch/lib/libexecutorch.so the runtime
executorch/lib/libexecutorch_kernels_optimized.so CPU operator kernels
executorch/lib/libexecutorch_backend_xnnpack.so the XNNPACK delegate
executorch/lib/libexecutorch_threadpool.so the shared thread pool
executorch/lib/libexecutorch_etdump.so the profiler
`_portable_lib.so` keeps only the Python binding glue and drops from about 11 MB
to well under 1 MB. Every other consumer in the tree that used to embed its own
copy now resolves it from the shared library, so a process has one registry and
one thread pool no matter how many of them it loads.
A `.pte` file produces the same outputs, the same operators are registered, and
the same backends are available. The existing header set the wheel installs is
untouched, and a custom-operator build against the Python extension keeps
working: the extension is at the same path and records a dependency on each
shared library.
That contract does need one addition. A custom-operator library references
runtime symbols and used to get them from the extension itself, which contained
the runtime. Now that the extension links the runtime instead, the installed
package config puts the shipped runtime on the extension's imported target, so
those symbols resolve from where the definitions actually moved to. Without it a
custom operator still links but fails to load with an undefined runtime
symbol.
Making these libraries linkable by a standalone C++ application needs a package
config that declares them, which is a separate change on top of this one. This
change is only the split, because it touches the most widely used artifact in
the wheel and should be reviewable on its own.
The libraries are built only where a shared runtime is possible, so macOS, iOS,
Android, Windows and embedded builds keep linking static libraries exactly as
before. Those platforms either have no dynamic loader at all, or in the case of
Windows a runtime with no export annotations for a DLL, so the static path stays
and is what the second branch of each build condition preserves. Asking for the
shared build on one of them is now an error at configure time rather than a
confusing packaging failure much later.
Two parts of this apply everywhere, because they are correct everywhere rather
than only in a wheel. A backend or kernel library that registers itself from a
static initialiser is now retained on a consumer's link line, which keeps a
shared backend from being dropped under `--as-needed`. And a Python extension no
longer links the embedding form of the Python library, which otherwise leaves a
hard dependency on an absolute path to the build machine's interpreter. Both
change link lines where they apply, and both fix a real problem rather than only
enabling this one.
Test plan:
Everything below runs against an installed wheel in a new virtual environment,
from a directory holding no checkout, so the checks cannot silently inspect the
source tree instead of the package.
- The wheel ships the five libraries above, each with a versioned file name and
a soname that matches it. A library without a version, which is what a
leftover from an earlier build looks like, fails.
- `_portable_lib.so` defines none of the registry, kernel, thread pool or
delegate symbols the shipped libraries own, and records a dependency on every
one of those libraries. Its size is reported rather than asserted, since the
number differs per architecture.
- For each of those symbol groups, exactly one shipped object defines it, and it
is the library that is supposed to own it. Naming the owner matters: the
monolithic layout also had exactly one definer of each, inside the extension.
- Every shipped library resolves its dependencies, and still resolves after
being copied into a different directory with the build-tree paths stripped, so
the wheel is relocatable rather than only working where it was built. No
shipped library carries an absolute runtime search path.
- Every Python extension the wheel ships imports in a clean environment. The
list is discovered from the installed package rather than written down, so an
extension added later is covered without anyone remembering to add it.
- A custom operator still compiles against the shipped Python extension, which
is the pre-existing contract for out-of-tree operators.
- The wheel's platform tag matches what it was built for.
- A model exported to a `.pte` and run through the Python bindings matches eager
PyTorch within tolerance, and so does the same model delegated to XNNPACK. The
delegated program is checked for the delegate's own identity, because a
partitioner that claimed nothing would fall back to the CPU kernels and still
match.
- All of the above on x86_64 and on aarch64. Both are necessary rather than
redundant: linker retention behaves differently between them, and x86_64 has
kept libraries that aarch64 dropped.
ghstack-source-id: d538f38
ghstack-comment-id: 5200527760
Pull-Request: #21610
shoumikhin
added
ciflow/trunk
ciflow/binaries
ciflow/binaries/all
shoumikhin
added a commit
that referenced
this pull request
Aug 6, 2026
`pip install executorch` ships the C++ runtime welded inside the Python
bindings extension. That one file, `_portable_lib.so`, is about 11 MB and
contains the runtime, the operator registries, the CPU kernels, the XNNPACK
delegate, the profiler and several extensions, all statically linked together.
A Python user does not notice, but it has a correctness cost. Two other
extensions in the same wheel, the training bindings and the language model
runner, each carry their own private copy of the runtime. A process that loads
more than one of them ends up with more than one backend registry and more than
one thread pool. Backends register themselves from a static initialiser into a
table that is only process-wide if the runtime is loaded once, so a second copy
means a backend can land in a table nothing else reads.
This change builds each of those pieces once, as a shared library the wheel
ships, and makes the extensions link them instead of containing them:
executorch/lib/libexecutorch.so the runtime
executorch/lib/libexecutorch_kernels_optimized.so CPU operator kernels
executorch/lib/libexecutorch_backend_xnnpack.so the XNNPACK delegate
executorch/lib/libexecutorch_threadpool.so the shared thread pool
executorch/lib/libexecutorch_etdump.so the profiler
`_portable_lib.so` keeps only the Python binding glue and drops from about 11 MB
to well under 1 MB. Every other consumer in the tree that used to embed its own
copy now resolves it from the shared library, so a process has one registry and
one thread pool no matter how many of them it loads.
A `.pte` file produces the same outputs, the same operators are registered, and
the same backends are available. The existing header set the wheel installs is
untouched, and a custom-operator build against the Python extension keeps
working: the extension is at the same path and records a dependency on each
shared library.
That contract does need one addition. A custom-operator library references
runtime symbols and used to get them from the extension itself, which contained
the runtime. Now that the extension links the runtime instead, the installed
package config puts the shipped runtime on the extension's imported target, so
those symbols resolve from where the definitions actually moved to. Without it a
custom operator still links but fails to load with an undefined runtime
symbol.
Making these libraries linkable by a standalone C++ application needs a package
config that declares them, which is a separate change on top of this one. This
change is only the split, because it touches the most widely used artifact in
the wheel and should be reviewable on its own.
The libraries are built only where a shared runtime is possible, so macOS, iOS,
Android, Windows and embedded builds keep linking static libraries exactly as
before. Those platforms either have no dynamic loader at all, or in the case of
Windows a runtime with no export annotations for a DLL, so the static path stays
and is what the second branch of each build condition preserves. Asking for the
shared build on one of them is now an error at configure time rather than a
confusing packaging failure much later.
Two parts of this apply everywhere, because they are correct everywhere rather
than only in a wheel. A backend or kernel library that registers itself from a
static initialiser is now retained on a consumer's link line, which keeps a
shared backend from being dropped under `--as-needed`. And a Python extension no
longer links the embedding form of the Python library, which otherwise leaves a
hard dependency on an absolute path to the build machine's interpreter. Both
change link lines where they apply, and both fix a real problem rather than only
enabling this one.
Test plan:
Everything below runs against an installed wheel in a new virtual environment,
from a directory holding no checkout, so the checks cannot silently inspect the
source tree instead of the package.
- The wheel ships the five libraries above, each with a versioned file name and
a soname that matches it. A library without a version, which is what a
leftover from an earlier build looks like, fails.
- `_portable_lib.so` defines none of the registry, kernel, thread pool or
delegate symbols the shipped libraries own, and records a dependency on every
one of those libraries. Its size is reported rather than asserted, since the
number differs per architecture.
- For each of those symbol groups, exactly one shipped object defines it, and it
is the library that is supposed to own it. Naming the owner matters: the
monolithic layout also had exactly one definer of each, inside the extension.
- Every shipped library resolves its dependencies, and still resolves after
being copied into a different directory with the build-tree paths stripped, so
the wheel is relocatable rather than only working where it was built. No
shipped library carries an absolute runtime search path.
- Every Python extension the wheel ships imports in a clean environment. The
list is discovered from the installed package rather than written down, so an
extension added later is covered without anyone remembering to add it.
- A custom operator still compiles against the shipped Python extension, which
is the pre-existing contract for out-of-tree operators.
- The wheel's platform tag matches what it was built for.
- A model exported to a `.pte` and run through the Python bindings matches eager
PyTorch within tolerance, and so does the same model delegated to XNNPACK. The
delegated program is checked for the delegate's own identity, because a
partitioner that claimed nothing would fall back to the CPU kernels and still
match.
- All of the above on x86_64 and on aarch64. Both are necessary rather than
redundant: linker retention behaves differently between them, and x86_64 has
kept libraries that aarch64 dropped.
ghstack-source-id: f879887
ghstack-comment-id: 5200527760
Pull-Request: #21610
shoumikhin
added a commit
that referenced
this pull request
Aug 7, 2026
`pip install executorch` ships the C++ runtime welded inside the Python
bindings extension. That one file, `_portable_lib.so`, is about 11 MB and
contains the runtime, the operator registries, the CPU kernels, the XNNPACK
delegate, the profiler and several extensions, all statically linked together.
A Python user does not notice, but it has a correctness cost. Two other
extensions in the same wheel, the training bindings and the language model
runner, each carry their own private copy of the runtime. A process that loads
more than one of them ends up with more than one backend registry and more than
one thread pool. Backends register themselves from a static initialiser into a
table that is only process-wide if the runtime is loaded once, so a second copy
means a backend can land in a table nothing else reads.
This change builds each of those pieces once, as a shared library the wheel
ships, and makes the extensions link them instead of containing them:
executorch/lib/libexecutorch.so the runtime
executorch/lib/libexecutorch_kernels_optimized.so CPU operator kernels
executorch/lib/libexecutorch_backend_xnnpack.so the XNNPACK delegate
executorch/lib/libexecutorch_threadpool.so the shared thread pool
executorch/lib/libexecutorch_etdump.so the profiler
`_portable_lib.so` keeps only the Python binding glue and drops from about 11 MB
to well under 1 MB. The bytes move into `lib/` rather than disappearing, so the
wheel itself stays about the same size; what changes is that there is now one copy
instead of one per extension. Every consumer the wheel ships that used to embed its
own copy now resolves it from the shared library, so however many of these a process
loads, they share one registry and one thread pool. That is scoped to what this wheel
ships; the paragraph on bundled third-party code below explains what it does not
cover. The size is reported by the tests rather than asserted, because the number
differs per architecture and Python version.
A `.pte` file produces the same outputs, the same operators are registered, and
the same backends are available. The existing header set the wheel installs is
untouched, and a custom-operator build against the Python extension keeps
working: the extension is at the same path and records a dependency on each
shared library.
That contract does need one addition. A custom-operator library references
runtime symbols and used to get them from the extension itself, which contained
the runtime. Now that the extension links the runtime instead, the installed
package config puts the shipped runtime on the extension's imported target, so
those symbols resolve from where the definitions actually moved to. Without it a
custom operator still links but fails to load with an undefined runtime
symbol.
Making these libraries linkable by a standalone C++ application needs a package
config that declares them, which is a separate change on top of this one. This
change is only the split, because it touches the most widely used artifact in
the wheel and should be reviewable on its own.
The libraries are built only where this mechanism applies. It names them with an
ELF soname, records `$ORIGIN` runtime search paths, and uses GNU linker options to
keep a registration-only library on a link line. None of that carries to Apple,
which is served by the Swift package distribution, or to Windows, where the
runtime carries no export annotations for a DLL. So macOS, iOS, Android, Windows
and embedded builds take the second branch of each build condition and link the
static libraries, as they did before.
One behaviour does change there. `EXECUTORCH_BUILD_SHARED` previously configured
on macOS and produced something packaging could not use; it is now a configure
error naming the reason. That is a deliberate narrowing: the option was added for
Linux distribution packaging, nothing in the repository or CI enabled it
elsewhere, and failing at configure time is clearer than failing later when
packaging looks for a `.so` the build never emitted.
One piece named in the RFC is not split out here. The quantized kernels still
build into `libquantized_ops_aot_lib.so` beside the Python extension rather than
into a shared library a C++ application could link. They are rewired to resolve the
runtime from `libexecutorch.so`, so they take part in the single-registry fix; only
the C++-linkable split is deferred.
One thing this does not fix. These libraries bundle third-party code that torch
also links, and both keep it visible, so the process holds two definitions of
symbols like `pthreadpool_create` and several hundred `xnn_*` and a caller reaches
whichever the loader resolved first. That is unchanged from the monolithic layout,
where the extension bundled the same code. Hiding it wholesale does not work: with
`--exclude-libs,ALL` the optimized kernels on aarch64 can no longer resolve
`cpuinfo_initialize` from the thread pool, which they do across a library boundary.
Doing it properly needs an explicit export list, so it is left to its own change.
The tests here assert one owner among the libraries this wheel ships, which is what
this change controls.
Three parts of this apply everywhere, because they are correct everywhere rather
than only in a wheel. A backend or kernel library that registers itself from a
static initialiser is now retained on a consumer's link line, which keeps a
shared backend from being dropped under `--as-needed`. And a Python extension no
longer links the embedding form of the Python library. That matters only where the
interpreter was built with a shared libpython: there the extension records a
dependency on it, and the release build's interpreter is static, so the published
wheel does not show the problem today. Both change link lines where they apply. And the runtime search path the quantized
AOT library uses to reach the Python extension named a directory that does not
exist, `extensions/pybindings` rather than `extension/pybindings`, on both the Apple
and the ELF branch. Corrected on both, which changes the recorded path on every
platform that builds it, not only in a wheel. The dependency-closure check fails
without it.
Test plan:
Everything below runs against an installed wheel in a new virtual environment,
from a directory holding no checkout, so the checks cannot silently inspect the
source tree instead of the package.
- The wheel ships the five libraries above, each with a versioned file name and
a soname that matches it. A library without a version, which is what a
leftover from an earlier build looks like, fails.
- `_portable_lib.so` defines none of the registry, kernel, thread pool or
delegate symbols the shipped libraries own, and records a dependency on every
one of those libraries. Its size is reported rather than asserted, since the
number differs per architecture.
- For each of those symbol groups, exactly one shipped object defines it, and it
is the library that is supposed to own it. Naming the owner matters: the
monolithic layout also had exactly one definer of each, inside the extension.
- Every shipped library resolves its dependencies, and still resolves after
being copied into a different directory with the build-tree paths stripped, so
the wheel is relocatable rather than only working where it was built. No
shipped library carries an absolute runtime search path.
- Every Python extension the wheel ships imports in a clean environment. The
list is discovered from the installed package rather than written down, so an
extension added later is covered without anyone remembering to add it.
- A custom operator still compiles against the shipped Python extension, which
is the pre-existing contract for out-of-tree operators.
- The wheel's platform tag matches what it was built for.
- A model exported to a `.pte` and run through the Python bindings matches eager
PyTorch within tolerance, and so does the same model delegated to XNNPACK. The
delegated program is checked for the delegate's own identity, because a
partitioner that claimed nothing would fall back to the CPU kernels and still
match.
- All of the above on x86_64 and on aarch64. Both are necessary rather than
redundant: linker retention behaves differently between them, and x86_64 has
kept libraries that aarch64 dropped.
ghstack-source-id: 2081d24
ghstack-comment-id: 5200527760
Pull-Request: #21610
shoumikhin
added a commit
that referenced
this pull request
Aug 7, 2026
`pip install executorch` ships the C++ runtime welded inside the Python
bindings extension. That one file, `_portable_lib.so`, is about 11 MB and
contains the runtime, the operator registries, the CPU kernels, the XNNPACK
delegate, the profiler and several extensions, all statically linked together.
A Python user does not notice, but it has a correctness cost. Two other
extensions in the same wheel, the training bindings and the language model
runner, each carry their own private copy of the runtime. A process that loads
more than one of them ends up with more than one backend registry and more than
one thread pool. Backends register themselves from a static initialiser into a
table that is only process-wide if the runtime is loaded once, so a second copy
means a backend can land in a table nothing else reads.
This change builds each of those pieces once, as a shared library the wheel
ships, and makes the extensions link them instead of containing them:
executorch/lib/libexecutorch.so the runtime
executorch/lib/libexecutorch_kernels_optimized.so CPU operator kernels
executorch/lib/libexecutorch_backend_xnnpack.so the XNNPACK delegate
executorch/lib/libexecutorch_threadpool.so the shared thread pool
executorch/lib/libexecutorch_etdump.so the profiler
`_portable_lib.so` keeps only the Python binding glue and drops from about 11 MB
to well under 1 MB. The bytes move into `lib/` rather than disappearing, so the
wheel itself stays about the same size; what changes is that there is now one copy
instead of one per extension. Every consumer the wheel ships that used to embed its
own copy now resolves it from the shared library, so however many of these a process
loads, they share one registry and one thread pool. That is scoped to what this wheel
ships; the paragraph on bundled third-party code below explains what it does not
cover. The size is reported by the tests rather than asserted, because the number
differs per architecture and Python version.
A `.pte` file produces the same outputs, the same operators are registered, and
the same backends are available. The existing header set the wheel installs is
untouched, and a custom-operator build against the Python extension keeps
working: the extension is at the same path and records a dependency on each
shared library.
The package config the wheel installs also had three defects that this change
corrects, because a consumer now has a real reason to use it:
`find_package(executorch REQUIRED)` reported success even when it found nothing,
since CMake checks the case-sensitive `executorch_FOUND` and the file only set
`EXECUTORCH_FOUND`; the Python extension was declared as a static library when it
is a shared object; and the C++20 requirement was expressed with `CXX_STANDARD`,
which an imported target does not pass to anything that links it, so a consumer
following the documented pattern compiled as C++17 and failed against headers that
need C++20.
That contract does need one addition. A custom-operator library references
runtime symbols and used to get them from the extension itself, which contained
the runtime. Now that the extension links the runtime instead, the installed
package config puts the shipped runtime on the extension's imported target, so
those symbols resolve from where the definitions actually moved to. Without it a
custom operator still links but fails to load with an undefined runtime
symbol.
Making these libraries linkable by a standalone C++ application needs a package
config that declares them, which is a separate change on top of this one. This
change is only the split, because it touches the most widely used artifact in
the wheel and should be reviewable on its own.
The libraries are built only where this mechanism applies. It names them with an
ELF soname, records `$ORIGIN` runtime search paths, and uses GNU linker options to
keep a registration-only library on a link line. None of that carries to Apple,
which is served by the Swift package distribution, or to Windows, where the
runtime carries no export annotations for a DLL. So macOS, iOS, Android, Windows
and embedded builds take the second branch of each build condition and link the
static libraries, as they did before.
One behaviour does change there. `EXECUTORCH_BUILD_SHARED` previously configured
on macOS and produced something packaging could not use; it is now a configure
error naming the reason. That is a deliberate narrowing: the option was added for
Linux distribution packaging, nothing in the repository or CI enabled it
elsewhere, and failing at configure time is clearer than failing later when
packaging looks for a `.so` the build never emitted.
One piece named in the RFC is not split out here. The quantized kernels still
build into `libquantized_ops_aot_lib.so` beside the Python extension rather than
into a shared library a C++ application could link. They are rewired to resolve the
runtime from `libexecutorch.so`, so they take part in the single-registry fix; only
the C++-linkable split is deferred.
One thing this does not fix. These libraries bundle third-party code that torch
also links, and both keep it visible, so the process holds two definitions of
symbols like `pthreadpool_create` and several hundred `xnn_*` and a caller reaches
whichever the loader resolved first. That is unchanged from the monolithic layout,
where the extension bundled the same code. Hiding it wholesale does not work: with
`--exclude-libs,ALL` the optimized kernels on aarch64 can no longer resolve
`cpuinfo_initialize` from the thread pool, which they do across a library boundary.
Doing it properly needs an explicit export list, so it is left to its own change.
The tests here assert one owner among the libraries this wheel ships, which is what
this change controls.
Three parts of this apply everywhere, because they are correct everywhere rather
than only in a wheel. A backend or kernel library that registers itself from a
static initialiser is now retained on a consumer's link line, which keeps a
shared backend from being dropped under `--as-needed`. And a Python extension no
longer links the embedding form of the Python library. That matters only where the
interpreter was built with a shared libpython: there the extension records a
dependency on it, and the release build's interpreter is static, so the published
wheel does not show the problem today. Both change link lines where they apply. And the runtime search path the quantized
AOT library uses to reach the Python extension named a directory that does not
exist, `extensions/pybindings` rather than `extension/pybindings`, on both the Apple
and the ELF branch. Corrected on both, which changes the recorded path on every
platform that builds it, not only in a wheel. The dependency-closure check fails
without it.
Test plan:
Everything below runs against an installed wheel in a new virtual environment,
from a directory holding no checkout, so the checks cannot silently inspect the
source tree instead of the package.
- The wheel ships the five libraries above, each with a versioned file name and
a soname that matches it. A library without a version, which is what a
leftover from an earlier build looks like, fails.
- `_portable_lib.so` defines none of the registry, kernel, thread pool or
delegate symbols the shipped libraries own, and records a dependency on every
one of those libraries. Its size is reported rather than asserted, since the
number differs per architecture.
- For each of those symbol groups, exactly one shipped object defines it, and it
is the library that is supposed to own it. Naming the owner matters: the
monolithic layout also had exactly one definer of each, inside the extension.
- Every shipped library resolves its dependencies, and still resolves after
being copied into a different directory with the build-tree paths stripped, so
the wheel is relocatable rather than only working where it was built. No
shipped library carries an absolute runtime search path.
- Every Python extension the wheel ships imports in a clean environment. The
list is discovered from the installed package rather than written down, so an
extension added later is covered without anyone remembering to add it.
- A custom operator still compiles against the shipped Python extension, which
is the pre-existing contract for out-of-tree operators.
- The wheel's platform tag matches what it was built for.
- A model exported to a `.pte` and run through the Python bindings matches eager
PyTorch within tolerance, and so does the same model delegated to XNNPACK. The
delegated program is checked for the delegate's own identity, because a
partitioner that claimed nothing would fall back to the CPU kernels and still
match.
- All of the above on x86_64 and on aarch64. Both are necessary rather than
redundant: linker retention behaves differently between them, and x86_64 has
kept libraries that aarch64 dropped.
ghstack-source-id: 73e34be
ghstack-comment-id: 5200527760
Pull-Request: #21610
shoumikhin
added a commit
that referenced
this pull request
Aug 7, 2026
## Why
Today `pip install executorch` ships one big file that has everything welded
inside it. Every extension in the wheel that needs the runtime gets its own
private copy.
That is not just wasteful, it is incorrect. A backend registers itself into a
table when its library loads. That table is only shared across the process if
the runtime is loaded once. With three copies there are three tables, and a
backend can register into a table that nothing else ever reads.
BEFORE AFTER
_portable_lib.so ~11 MB _portable_lib.so <1 MB
runtime copy 1 just binding glue
registries |
CPU kernels |
XNNPACK _training_lib.so
threadpool |
profiler |
_llm_runner.so
_training_lib.so |
runtime copy 2 |
... v
lib/libexecutorch.so (one copy)
_llm_runner.so lib/libexecutorch_kernels_optimized.so
runtime copy 3 lib/libexecutorch_backend_xnnpack.so
... lib/libexecutorch_threadpool.so
lib/libexecutorch_etdump.so
3 registries, 3 thread pools 1 registry, 1 thread pool
## What this change does
Builds each piece once as its own shared library, and makes the extensions link
those instead of containing them.
| library | what it is |
| ------------------------------------ | ---------------------- |
| `libexecutorch.so` | the runtime |
| `libexecutorch_kernels_optimized.so` | CPU operator kernels |
| `libexecutorch_backend_xnnpack.so` | the XNNPACK delegate |
| `libexecutorch_threadpool.so` | the shared thread pool |
| `libexecutorch_etdump.so` | the profiler |
They land in `executorch/lib/` inside the installed package.
## What to expect
For a Python user, nothing changes. Same imports, same results.
| |before|after|
|---|---|---|
|`_portable_lib.so` size|~11 MB|under 1 MB|
|wheel size|—|about the same|
|copies of the runtime in a process|one per extension|one|
|backend registries|one per extension|one|
|thread pools|one per extension|one|
|model outputs|—|identical|
The wheel does not get smaller. The bytes move out of the extension and into
`lib/`, so the total is similar. What changes is that there is now one copy
instead of one per extension.
Every consumer the wheel ships now resolves the runtime from the shared library,
so however many of these a process loads, they share one registry and one thread
pool.
## Scope
Linux only, because the mechanism is Linux-only: ELF sonames, `$ORIGIN` runtime
search paths, and GNU linker options to keep a registration-only library on a
link line. macOS, iOS, Android, Windows and embedded builds take the existing
branch and link static libraries exactly as before.
One behaviour does change there. `EXECUTORCH_BUILD_SHARED` used to configure on
macOS and produce something packaging could not use. It is now a configure error
that says why. The option was added for Linux distribution packaging and nothing
in the repository or CI enabled it elsewhere, so failing early is clearer than
failing later when packaging looks for a `.so` the build never made.
Making these libraries linkable by a standalone C++ application needs a package
config that declares them. That is a separate change on top of this one. This
change is only the split, because it touches the most widely used artifact in the
wheel and should be reviewable on its own.
## Two things this does not fix
**Quantized kernels are not split out yet.** They still build into
`libquantized_ops_aot_lib.so` beside the Python extension rather than into a
library a C++ application could link. They are rewired to resolve the runtime
from `libexecutorch.so`, so they do take part in the single-registry fix; only
the C++-linkable split is deferred.
**Bundled third-party code still collides with torch.** These libraries bundle
code that torch also links, and both keep it visible, so a process holds two
definitions of symbols like `pthreadpool_create` and several hundred `xnn_*`
ones, and a caller reaches whichever the loader found first. That is unchanged
from before, where the extension bundled the same code. Hiding it wholesale does
not work: with `--exclude-libs,ALL` the optimized kernels on aarch64 can no
longer resolve `cpuinfo_initialize` from the thread pool, which they do across a
library boundary. Doing it properly needs an explicit export list, so it is left
to its own change. The tests here assert one owner among the libraries this wheel
ships, which is what this change controls.
## Fixes to the installed CMake package config
A consumer now has a real reason to use this file, and it had three bugs:
- `find_package(executorch REQUIRED)` reported success even when it found
nothing, because CMake checks the case-sensitive `executorch_FOUND` and the
file only set `EXECUTORCH_FOUND`.
- The Python extension was declared as a static library when it is a shared
object.
- The C++20 requirement used `CXX_STANDARD`, which an imported target does not
pass to anything linking it, so a consumer following the documented pattern
compiled as C++17 and failed against headers needing C++20.
It also needs one addition. A custom-operator library references runtime symbols
and used to get them from the extension, which contained the runtime. Now that
the extension links the runtime instead, the config puts the shipped runtime on
the extension's imported target so those symbols resolve from where the
definitions moved. Without it a custom operator links and then fails to load with
an undefined runtime symbol.
## Changes that apply on every platform
Three fixes here are correct everywhere, not only in a wheel:
- A backend or kernel library that registers itself from a static initialiser is
now retained on a consumer's link line, so a shared backend is not dropped
under `--as-needed`.
- A Python extension no longer links the embedding form of the Python library.
This matters only where the interpreter was built with a shared libpython; the
release interpreter is static, so the published wheel does not show it today.
- The runtime search path the quantized AOT library uses to reach the Python
extension named a directory that does not exist, `extensions/pybindings`
instead of `extension/pybindings`, on both the Apple and the ELF branch.
Corrected on both. The dependency-closure check fails without it.
## Fixes from review of an earlier revision
**A retention helper was silently doing nothing on static targets.** `PRIVATE`
link options are dropped on a static library, while `PRIVATE` link libraries
still propagate as `$<LINK_ONLY:...>`. So a consumer got the shared runtime with
no `--no-as-needed` around it, which is the exact thing the helper exists to
prevent. Reachable, because the operators libraries are static unless a caller
asks for shared. It is now a fatal error on a static target, and the operators
library only calls it when shared; a static one is extracted whole into its
consumer, which is what retains the runtime.
**The checks ran in an order that let the weakest hide the strongest.** The
platform tag check sat in the middle, and when it failed it took the
custom-operator, runtime-path and numeric-parity checks with it in every wheel
job. Those now run first and the tag check runs last, because a wrong answer
there means the wheel is labelled wrong, not that the split is broken.
**The tag comparison's own test re-implemented the comparison instead of calling
it**, so it could not have caught the bug it was written for. Both the wheel
check and a unit test now call one function, and that unit test moved to
`.ci/scripts/tests/` where it runs without building a wheel. It immediately found
that the architecture list matched `macosx_11_0_arm64`, so `arm64` is gone from
it; Linux uses `aarch64`.
**Seven places looked for a tool with a helper that also checks beside the
interpreter, then invoked the bare name**, which the shell resolves through
`PATH` only. A tool found by the fallback would pass the check and fail to run.
Also declares `py-cpuinfo`, which `backends/qualcomm/__init__.py` raises
`ImportError` without, so that extension could not be imported from a clean
install of this wheel.
Test plan:
Everything below runs against an installed wheel in a new virtual environment,
from a directory holding no checkout, so the checks cannot silently inspect the
source tree instead of the package.
- The wheel ships the five libraries above, each with a soname matching its file
name. A name packaging does not produce, which is what a leftover from an
earlier build looks like, fails. That happened: a wheel once picked up three
libraries from a different revision and passed every symbol check.
- `_portable_lib.so` defines none of the registry, kernel, thread pool or
delegate symbols the shipped libraries own, and records a dependency on every
one of them. Its size is reported rather than asserted, since the number
differs per architecture and Python version.
- For each symbol group, exactly one shipped object defines it, and it is the
library supposed to own it. Naming the owner matters: the old layout also had
exactly one definer of each, inside the extension.
- Every shipped library resolves its dependencies, and still does after being
copied to a different directory with build-tree paths stripped, so the wheel is
relocatable rather than only working where it was built. No shipped library
carries an absolute runtime search path.
- Every Python extension the wheel ships imports in a clean environment. The list
is discovered from the installed package rather than written down, so an
extension added later is covered without anyone remembering.
- A custom operator still compiles against the shipped Python extension, which is
the pre-existing contract for out-of-tree operators.
- The wheel's platform tag names the architecture it was built for.
- A model exported to a `.pte` and run through the Python bindings matches eager
PyTorch within tolerance, and so does the same model delegated to XNNPACK. The
delegated program is checked for the delegate's own identity, because a
partitioner that claimed nothing would fall back to CPU kernels and still match.
- All of the above on x86_64 and aarch64. Both are necessary rather than
redundant: linker retention differs between them, and x86_64 has kept libraries
that aarch64 dropped.
ghstack-source-id: 4474691
ghstack-comment-id: 5200527760
Pull-Request: #21610
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Today
pip install executorchships one big file that has everything weldedinside it. Every extension in the wheel that needs the runtime gets its own
private copy.
That is not just wasteful, it is incorrect. A backend registers itself into a
table when its library loads. That table is only shared across the process if
the runtime is loaded once. With three copies there are three tables, and a
backend can register into a table that nothing else ever reads.
What this change does
Builds each piece once as its own shared library, and makes the extensions link
those instead of containing them.
libexecutorch.solibexecutorch_kernels_optimized.solibexecutorch_backend_xnnpack.solibexecutorch_threadpool.solibexecutorch_etdump.soThey land in
executorch/lib/inside the installed package.What to expect
For a Python user, nothing changes. Same imports, same results.
_portable_lib.sosizeThe wheel does not get smaller. The bytes move out of the extension and into
lib/, so the total is similar. What changes is that there is now one copyinstead of one per extension.
Every consumer the wheel ships now resolves the runtime from the shared library,
so however many of these a process loads, they share one registry and one thread
pool.
Scope
Linux only, because the mechanism is Linux-only: ELF sonames,
$ORIGINruntimesearch paths, and GNU linker options to keep a registration-only library on a
link line. macOS, iOS, Android, Windows and embedded builds take the existing
branch and link static libraries exactly as before.
One behaviour does change there.
EXECUTORCH_BUILD_SHAREDused to configure onmacOS and produce something packaging could not use. It is now a configure error
that says why. The option was added for Linux distribution packaging and nothing
in the repository or CI enabled it elsewhere, so failing early is clearer than
failing later when packaging looks for a
.sothe build never made.Making these libraries linkable by a standalone C++ application needs a package
config that declares them. That is a separate change on top of this one. This
change is only the split, because it touches the most widely used artifact in the
wheel and should be reviewable on its own.
Two things this does not fix
Quantized kernels are not split out yet. They still build into
libquantized_ops_aot_lib.sobeside the Python extension rather than into alibrary a C++ application could link. They are rewired to resolve the runtime
from
libexecutorch.so, so they do take part in the single-registry fix; onlythe C++-linkable split is deferred.
Bundled third-party code still collides with torch. These libraries bundle
code that torch also links, and both keep it visible, so a process holds two
definitions of symbols like
pthreadpool_createand several hundredxnn_*ones, and a caller reaches whichever the loader found first. That is unchanged
from before, where the extension bundled the same code. Hiding it wholesale does
not work: with
--exclude-libs,ALLthe optimized kernels on aarch64 can nolonger resolve
cpuinfo_initializefrom the thread pool, which they do across alibrary boundary. Doing it properly needs an explicit export list, so it is left
to its own change. The tests here assert one owner among the libraries this wheel
ships, which is what this change controls.
Fixes to the installed CMake package config
A consumer now has a real reason to use this file, and it had three bugs:
find_package(executorch REQUIRED)reported success even when it foundnothing, because CMake checks the case-sensitive
executorch_FOUNDand thefile only set
EXECUTORCH_FOUND.object.
CXX_STANDARD, which an imported target does notpass to anything linking it, so a consumer following the documented pattern
compiled as C++17 and failed against headers needing C++20.
It also needs one addition. A custom-operator library references runtime symbols
and used to get them from the extension, which contained the runtime. Now that
the extension links the runtime instead, the config puts the shipped runtime on
the extension's imported target so those symbols resolve from where the
definitions moved. Without it a custom operator links and then fails to load with
an undefined runtime symbol.
Changes that apply on every platform
Three fixes here are correct everywhere, not only in a wheel:
now retained on a consumer's link line, so a shared backend is not dropped
under
--as-needed.This matters only where the interpreter was built with a shared libpython; the
release interpreter is static, so the published wheel does not show it today.
extension named a directory that does not exist,
extensions/pybindingsinstead of
extension/pybindings, on both the Apple and the ELF branch.Corrected on both. The dependency-closure check fails without it.
Fixes from review of an earlier revision
A retention helper was silently doing nothing on static targets.
PRIVATElink options are dropped on a static library, while
PRIVATElink librariesstill propagate as
$<LINK_ONLY:...>. So a consumer got the shared runtime withno
--no-as-neededaround it, which is the exact thing the helper exists toprevent. Reachable, because the operators libraries are static unless a caller
asks for shared. It is now a fatal error on a static target, and the operators
library only calls it when shared; a static one is extracted whole into its
consumer, which is what retains the runtime.
The checks ran in an order that let the weakest hide the strongest. The
platform tag check sat in the middle, and when it failed it took the
custom-operator, runtime-path and numeric-parity checks with it in every wheel
job. Those now run first and the tag check runs last, because a wrong answer
there means the wheel is labelled wrong, not that the split is broken.
The tag comparison's own test re-implemented the comparison instead of calling
it, so it could not have caught the bug it was written for. Both the wheel
check and a unit test now call one function, and that unit test moved to
.ci/scripts/tests/where it runs without building a wheel. It immediately foundthat the architecture list matched
macosx_11_0_arm64, soarm64is gone fromit; Linux uses
aarch64.Seven places looked for a tool with a helper that also checks beside the
interpreter, then invoked the bare name, which the shell resolves through
PATHonly. A tool found by the fallback would pass the check and fail to run.Also declares
py-cpuinfo, whichbackends/qualcomm/__init__.pyraisesImportErrorwithout, so that extension could not be imported from a cleaninstall of this wheel.
Test plan:
Everything below runs against an installed wheel in a new virtual environment,
from a directory holding no checkout, so the checks cannot silently inspect the
source tree instead of the package.
name. A name packaging does not produce, which is what a leftover from an
earlier build looks like, fails. That happened: a wheel once picked up three
libraries from a different revision and passed every symbol check.
_portable_lib.sodefines none of the registry, kernel, thread pool ordelegate symbols the shipped libraries own, and records a dependency on every
one of them. Its size is reported rather than asserted, since the number
differs per architecture and Python version.
library supposed to own it. Naming the owner matters: the old layout also had
exactly one definer of each, inside the extension.
copied to a different directory with build-tree paths stripped, so the wheel is
relocatable rather than only working where it was built. No shipped library
carries an absolute runtime search path.
is discovered from the installed package rather than written down, so an
extension added later is covered without anyone remembering.
the pre-existing contract for out-of-tree operators.
.pteand run through the Python bindings matches eagerPyTorch within tolerance, and so does the same model delegated to XNNPACK. The
delegated program is checked for the delegate's own identity, because a
partitioner that claimed nothing would fall back to CPU kernels and still match.
redundant: linker retention differs between them, and x86_64 has kept libraries
that aarch64 dropped.