[do not land] CUDA wheel built with the device-activation copy fix - #21701
Closed
shoumikhin wants to merge 6 commits into
Closed
[do not land] CUDA wheel built with the device-activation copy fix#21701shoumikhin wants to merge 6 commits into
shoumikhin wants to merge 6 commits into
Conversation
## The problem `pip install executorch` gives you the Python half of ExecuTorch and nothing a C++ program can link. Everything is fused into one large Python extension file, so a C++ developer has to clone the repository, sync submodules, and build from source. There is a correctness problem underneath the packaging one. Because the runtime is fused into the extension, anything else that needs it gets its own private copy, and two copies mean two registries. A backend registered in one is invisible to the other. ## The change Build the runtime and the pieces around it as separate shared libraries, and make the Python extension link them instead of embedding them. A shared library is a file a program loads at run time, so several programs can use one copy instead of each carrying its own. ``` executorch/ lib/libexecutorch.so the runtime lib/libexecutorch_kernels_optimized.so CPU operator kernels lib/libexecutorch_backend_xnnpack.so the XNNPACK delegate lib/libexecutorch_threadpool.so one thread pool per process lib/libexecutorch_etdump.so the profiler _portable_lib.so now 0.8 MB, links the above ``` The extension drops from about 11 MB to under a megabyte, because it no longer contains what it now links. Linux only. macOS and Windows keep the fused extension, so their wheels are unchanged. ## Test plan Built the wheel from source, installed it into a clean environment, and checked: - exactly one library defines each component, and it is the library that should own it. Counting owners alone would also pass on the old fused layout, which has exactly one too. - the Python extension defines none of them and resolves all of them from outside. - every shipped library loads with no unresolved dependency, and none of them searches a directory from the machine that built the wheel. - a C++ application built outside the wheel links the runtime, runs a model, and matches eager PyTorch, including a delegated model. It still runs after being copied away from the wheel. Ran on Linux x86_64 and aarch64, including a Jetson device. Not fixed here: these libraries bundle third-party code that torch also links, and both keep it visible, so a process holds two definitions of symbols like `pthreadpool_create`. A caller reaches whichever the loader found first. Fixing it means hiding or dropping the bundled copies, which is a larger change. ghstack-source-id: 7393c1a ghstack-comment-id: 5200527760 Pull-Request: #21610
## The problem
The previous change split the runtime, kernels, delegate, thread pool and profiler into separate
shared libraries, and the wheel ships them. But nothing outside Python can use them, because the
installed CMake package names none of them. A C++ application would have to hard-code paths into
the wheel's private layout.
The headers have the same gap. The wheel installs only the subset a custom-operator build needs,
which leaves out `extension/module`, the entry point the documentation tells C++ callers to use. So
the wheel ships the libraries to run a model and no way to call them.
## The change
Name each shipped library as a CMake component, so `find_package` locates them, and ship the
headers a caller needs. A component is just a name a consumer can ask for, and CMake reports a
missing one while configuring rather than at link time.
```cmake
find_package(executorch 1.5 REQUIRED COMPONENTS kernels_optimized)
target_link_libraries(my_app PRIVATE executorch::runtime
executorch::kernels_optimized)
```
| component | library it resolves to |
| --- | --- |
| `executorch::runtime` | `libexecutorch.so` |
| `executorch::kernels_optimized` | `libexecutorch_kernels_optimized.so` |
| `executorch::backend_xnnpack` | `libexecutorch_backend_xnnpack.so` |
| `executorch::threadpool` | `libexecutorch_threadpool.so` |
| `executorch::etdump` | `libexecutorch_etdump.so` |
Each component records where the wheel keeps its libraries, so an application built against it
finds them without the caller setting a library search path.
CMake 3.28 or newer gets these targets. Older versions still get the long-standing
`EXECUTORCH_LIBRARIES` variable, carrying the runtime and the CPU kernels by path, because they write
the `$ORIGIN` marker (the "look next to me" token in a library search path) incorrectly:
```
3.24.3, 3.27.9 Makefiles double the dollar sign, Ninja drops the name
3.28.4, 3.31.8 both write the token correctly
```
## Test plan
Built the wheel, installed it into a clean environment, and built a C++ application against the
installed wheel alone:
- the application links the runtime, runs a model, and matches eager PyTorch, and still runs after
being copied away from the wheel.
- asking for a component the wheel does not ship fails while configuring, naming the component.
- a version request is honoured, including ranges.
- every shipped header can be included, and the ones a caller would call also link. A header that
compiles but has no implementation in any shipped library would otherwise ship unnoticed.
- on real CMake 3.24 and 3.27, an application configures, builds and runs through
`EXECUTORCH_LIBRARIES`. The kernels need scoped retention there, because a registration-only
library exports nothing the application references and the linker drops it, which showed up as
"Missing operator" at run time rather than as a link error.
- `find_package` succeeds when the interpreter on PATH is not the one the wheel was built for. The
extension's own file name carries its suffix, so asking a different interpreter for it reported a
complete install as not found.
Ran on Linux x86_64 and aarch64, and on macOS.
ghstack-source-id: ae195a8
ghstack-comment-id: 5215967468
Pull-Request: #21639
## The problem
A quantized model uses smaller numbers than a normal one, so it runs faster and takes less memory.
Running one needs the quantized operator kernels.
Those kernels are compiled into the Python extension, so a C++ application cannot get them. It
links the runtime, loads a quantized model, and the model fails at run time with a missing
operator, which looks like a model problem rather than a packaging one.
## The change
Build the quantized kernels as their own shared library and name it as a CMake component, the same
way the previous change did for the other kernels.
```cmake
find_package(executorch REQUIRED COMPONENTS kernels_quantized)
target_link_libraries(my_app PRIVATE executorch::runtime
executorch::kernels_quantized)
```
The wheel now ships `lib/libexecutorch_kernels_quantized.so`.
Note that the wheel also ships a second copy of these kernels, in the library torch loads when you
export a model. Export happens in Python and never loads the runtime libraries, so nothing loads
both, and the two are built for different callers.
## Test plan
Built the wheel, installed it into a clean environment, and:
- exported a quantized model and ran it from Python, matching eager PyTorch exactly.
- built a C++ application that links `executorch::kernels_quantized`, ran the same program, and got
the same output as Python, byte for byte.
- confirmed the Python extension does not depend on the run-time copy. Loading both copies in one
process would abort, because the runtime treats a repeated operator registration as fatal.
Ran on Linux x86_64 and aarch64.
ghstack-source-id: 66d3be0
ghstack-comment-id: 5217087046
Pull-Request: #21642
## The problem
The CUDA delegate runs a model on an NVIDIA GPU. It is built into the Python extension, so only
Python can use it. A C++ application has no way to link it, and nothing else can reuse it either.
There is a sharing problem too. A program may use more than one GPU backend at once, and they need
to agree on which CUDA stream (the queue the GPU runs work on) the caller chose. If each backend
carries its own copy of that state, work queued through one is invisible to the other.
## The change
Ship the CUDA delegate and a small stream helper as their own shared libraries, and name both as
CMake components. The stream helper is shared so a process has exactly one copy of the caller's
stream choice, which is what lets two backends agree on it.
```cmake
find_package(executorch REQUIRED COMPONENTS backend_cuda)
target_link_libraries(my_app PRIVATE executorch::runtime
executorch::backend_cuda)
```
A CUDA wheel does not bundle the CUDA runtime. It declares it as a dependency, the way the PyTorch
CUDA wheels do, so one copy is shared with torch rather than shipping a second one:
```
executorch/lib/libexecutorch_backend_cuda.so the delegate
executorch/lib/libexecutorch_extension_cuda.so the stream helper
executorch/backends/cuda/libaoti_cuda_shims.so the GPU device code
```
Each library records a relative path to where pip installs the CUDA runtime, so it resolves without
the caller setting a library search path and without depending on a toolkit being installed.
The stream helper's header includes `cuda_runtime.h`, which the wheel does not publish. A caller of
that header is writing GPU code anyway, so it supplies a CUDA toolkit's include directory; the
package says so while configuring.
## Test plan
Built a CUDA wheel, installed it into a clean environment, and:
- ran a GPU model from Python, matching eager PyTorch exactly.
- built a C++ application against the installed wheel alone and ran the same model, matching the
same reference.
- confirmed one library defines the stream state and the GPU shims, not several. Extracting them
into every consumer put three copies in one wheel, and a stream selected through one was invisible
to the others.
- confirmed no shipped library records a CUDA toolkit path from the build machine, and every library
that links the CUDA runtime has a relative path to it.
- confirmed a CPU wheel ships none of the CUDA libraries and no CUDA-only header.
Ran end to end on H100, A100 and Jetson Thor, covering compute capabilities 9.0, 8.0 and 11.0.
Known gap, not introduced here: the Python `Runtime.load_program` path allocates activation memory
on the host, so a program exported to keep activations on the GPU fails there. The supported Python
loader and the C++ path both work. This is upstream in the Python bindings, which this change does
not touch.
ghstack-source-id: 87b2335
ghstack-comment-id: 5219161655
Pull-Request: #21645
## The problem The previous change makes the wheel able to carry the CUDA delegate, but nothing builds one. There is no CUDA row in any workflow, so a GPU user still has to build from source. ## The change Add the workflows that build and publish CUDA wheels for Linux x86_64 and aarch64, and a smoke test that runs a real model on a GPU before release. ``` executorch-1.5.0-cp312-cp312-manylinux_2_28_x86_64.whl +cu130 ``` A release publishes CUDA 12.6, 13.0 and 13.2, for Python 3.10 through 3.13. A pull request builds a single row instead of all twelve, because a full matrix costs hours for little extra signal. Which GPU architectures each row compiles for is chosen per row rather than detected on the builder. Detecting it would produce a wheel carrying device code for whatever machine happened to build it, which installs fine and then fails at the first GPU call. Two guards keep a release honest: - if the shared matrix generator stops offering a combination this policy advertises, the step fails instead of quietly publishing fewer wheels. A missing job is otherwise a green check for a wheel that was never built. - if a row reaches the architecture list with no CUDA version, the build refuses rather than falling back to the builder's GPU. Windows CUDA is deliberately absent. The separate shared libraries this wheel exists to ship are Linux only today, so a Windows CUDA wheel would carry a delegate a C++ application still could not link. ## Test plan - built the full release matrix, twelve wheels, and confirmed each one's contents match the row it claims: the CUDA libraries present, the CUDA runtime declared, and device code for every GPU architecture the row advertises. - ran a GPU model end to end from a CI-built wheel on H100, A100 and Jetson Thor, matching eager PyTorch exactly on each, and inspected an Orin Nano wheel. - ran the matrix filter over thousands of generated inputs, including incomplete and malformed ones, and confirmed it refuses rather than silently publishing a partial release. - confirmed a CPU row still produces a CPU wheel on a builder that happens to have a CUDA toolkit installed. ghstack-source-id: aced89e ghstack-comment-id: 5220374521 Pull-Request: #21668
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21701
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New Failure, 282 PendingAs of commit cb64bdd with merge base 730b77a ( NEW FAILURE - The following job has failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
## The problem ExecuTorch can export a model two ways for a GPU. The default copies your input from main memory (the host) to the GPU for you. The other way skips those copies, so you hand over memory that is already on the GPU and the output stays there. That second way saves a round trip, which matters for a model in a loop. The second way crashes. It crashes from Python and from C++, immediately, before your data is touched: ``` [cuda_backend.cpp:429] Created new CUDA stream 0x10ae920 for method Segmentation fault (core dumped) ``` The reason is one line. When the runtime puts your input into the buffer the memory plan reserved, it always used a plain host copy: ```cpp std::memcpy(t_dst.mutable_data_ptr(), t_src.const_data_ptr(), t_src.nbytes()); ``` If that destination is GPU memory, a host `memcpy` into it is undefined and the process dies. The memory itself was allocated correctly on the GPU; only the copy assumed the host. ## The change Route the copy through the device that owns the memory instead of assuming the host. ExecuTorch already has an interface for this, and a backend registers an implementation when it is linked in, so the core runtime still names no GPU code and still builds for bare metal: ``` core (no GPU code) CUDA backend copy_between_devices() -> get_device_allocator(CUDA) ──────── register_device_allocator(...) -> allocator->copy_host_to_device() ──► cudaMemcpy ``` A host to host copy keeps using `memcpy` and does not need any device registered. One primitive was missing. The interface had host to device and device to host, but not device to device, which is exactly what this feature needs: your input is already on the GPU and the planned buffer is too. It is added with a default that returns `NotSupported`, so an allocator that cannot do it declines instead of being forced to implement it, and the CUDA one implements it. ## Test plan Added `device_copy_test.cpp`, which needs no GPU: the repository already has a mock allocator that copies with `memcpy` and counts calls, so the routing is observable. - a host to host copy still works and does not touch the allocator - host to device, device to host, and device to device each reach the matching call - zero bytes is accepted, which a tensor with an empty dimension relies on Before this change no test executed a program with device activations at all. The closest one loads the method and tolerates failure without running it, so the crash had no coverage. Also built the runtime and ran the existing runtime and extension tests on Linux x86_64. ## Known gaps, not fixed here Two problems on the same feature remain, and each deserves its own change: - The Python bindings build the source tensor without reading its device, so a GPU input is labelled as host memory. A caller passing GPU tensors from Python is still not correctly described to the runtime. - For a program whose output stays on the GPU, the Python return path reads that memory as if it were host memory. ghstack-source-id: 3d35054 ghstack-comment-id: 5233812152 Pull-Request: #21699
shoumikhin
force-pushed
the
gh/shoumikhin/98/head
branch
from
August 9, 2026 22:40
c550cae to
cb64bdd
Compare
Contributor
Author
|
Closing this. It existed only to give continuous integration a CUDA wheel row to build the change in #21699 against, and that change is being withdrawn. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft, for continuous integration coverage only. Do not land.
Combines the device-activation copy fix with the CUDA wheel rows so a wheel is built that contains the fix. Without this, the fix has no wheel to verify against: it targets main, which does not yet have CUDA wheel rows, so no wheel job runs on it.
The fix and the wheel work are reviewed separately in their own pull requests.