Repository navigation
Sync with Microsoft ONNX Runtime - 08102026 - #1336
Merged
Merged
Conversation
) ## Summary Follow-up to microsoft#32528 (the initial WebGPU `SparseAttentionIndexer` implementation). This PR reduces the QSA recomputation called out in its follow-up list. The sparse-indexer prerequisite microsoft#32656 has merged into `main`; this PR now targets `main` directly. The optimization changes only the WebGPU QSA selection path. - Cache visible token positions and block scores per query in bounded workgroup memory. Reuse query/key RMS normalization within each block score. - Reuse cached positions on visibility overflow, scanning only the uncached tail; retain exact visibility and score recomputation fallbacks outside the cache bounds, preserving score-descending/index-ascending TopK semantics and non-prefix masks. - Bound both caches plus the count to 15,364 bytes, below WebGPU's guaranteed 16 KiB workgroup-storage limit. - Add FP32/FP16 non-prefix-mask parity tests and document the optimized/fallback paths. ## Validation - The original Qwen integration run passed 18 focused `SparseAttentionIndexerShapeInferenceTest` cases. On current `main`, a Linux Release WebGPU provider/provider-test build and all 23 focused QSA/CSA and packed-indexer cases passed on NVIDIA H200/Vulkan, including both bounded-cache fallback thresholds. Focused formatting and `git diff --check` passed. - End-to-end NVIDIA H200 Vulkan testing used a non-paged, no-MTP Qwen3.8 raw INT4 RTN/INT8-last-matmul Generator export, with 16-token prefill chunks: | Context | Before | After | | --- | --- | --- | | 16-token prefill | 43.27 s | 1.51 s | | Next decode step after 16 tokens | 42.63 s | 0.70 s | | 1K prefill | Did not complete within 181 s of prefill | 68.33 s (14.99 tok/s) | | 1K decode | No valid rate | 1.29 tok/s over eight tokens | | 2K prefill | No prior completed result | 145.37 s (14.09 tok/s) | | 2K decode | About 30.8 s/step at the old cache boundary | 1.03 tok/s over eight tokens | The eight generated pieces from the four-token smoke prompt were unchanged, as was the first token ID from the 16-token probe. Model-load time (~58–60 s) is excluded from prefill measurements. The H200 static WebGPU build now executes the WebGPU-specific parity tests; all 23 focused QSA/CSA and packed-indexer cases passed on current `main`. The EngramGate and GenAI builder/runtime fixes in the working tree are not part of this PR. The performance table above was measured on the earlier Qwen3.8 integration branch; GPU regression tests, not full-model timing, were rerun on current `main`. ## Current main update (October 6) Kunal's microsoft#32656 merged, and this PR is now based on `main` with a three-file QSA-only review diff. Merged current `main` again after that restack. The WebGPU provider-test build and all 23 focused H200 QSA/CSA plus packed-indexer tests passed. The October 6 Windows WebGPU job exposed a long-running visibility-overflow boundary test that lost the D3D12 device. Commit 9499988 avoids redundant prefix rescans; all 22 focused WebGPU QSA/CSA tests pass on H200/Vulkan, with that boundary test taking 241 ms. Android minimal-size CI passed on the preceding head; CI for the fix is pending. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This pull request adds improved validation for indices in the CUDA implementation of the `GatherND` operator and introduces new CUDA-specific tests to ensure invalid indices are properly detected and reported. The main goal is to catch out-of-bounds indices on the host before launching CUDA kernels, improving error handling and robustness. Validation improvements: * Added host-side validation in `GatherNDBase::PrepareCompute` to check that all indices are within valid bounds, both for CPU and CUDA tensors, preventing invalid memory accesses during CUDA execution. * Introduced use of `cudaMemcpyAsync` and `cudaStreamSynchronize` to copy indices from device to host for validation when running on CUDA. Testing enhancements: * Added CUDA-specific tests in `gather_nd_op_test.cc` to verify that invalid indices are properly detected and result in expected error messages for both standard and contrib ops. Code maintenance: * Included `<algorithm>` header for use of `std::copy_n` in `gather_nd.cc`. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
### Description - Mark the ONNX, optional FP16, and ONNX-ML CPU kernel factory tables `constexpr`. - Add a `consteval` invariant that requires the disabled `BuildKernelCreateInfo<void>` entry first and rejects null factory pointers. - Place that helper in the shared header `onnxruntime/core/providers/kernel_registration_validation.h` (namespace `onnxruntime::registration_internal`, `op_kernel.h` include guarded by `SHARED_PROVIDER`) so other execution providers can adopt the same check. Applying it to the remaining provider tables is tracked in microsoft#31348. - Preserve reduced-operator behavior: disabled registrations may legitimately repeat the non-null disabled entry. - Add independent compile-time tests for valid, missing-sentinel, null-factory, and null-sentinel tables, plus a runtime check that the real CPU registry initializes non-empty. ### Motivation and Context CPU kernel registration is a fixed build-time descriptor set, but malformed tables were only discoverable while constructing the runtime registry. These checks make sentinel placement and factory-pointer validity compile-time requirements across the normal, reduced-ops, optional FP16, and optional ONNX-ML table variants. The existing compact runtime registration loops are intentionally unchanged. A templated shared loop increased the provider library by 5,006 bytes, and an inline span version increased it by 33,970 bytes; keeping the loops produced the smallest measured result. ### Measurement Windows RelWithDebInfo, MSVC 19.51, touched-file rebuild of `cpu_execution_provider.cc`: | Artifact | Before | After | Delta | | --- | ---: | ---: | ---: | | `cpu_execution_provider.obj` | 2,033,704 B | 2,033,836 B | +132 B | | `onnxruntime_providers.lib` | 678,492,398 B | 678,492,530 B | +132 B | The single baseline rebuild was 8.55 s; final post-change samples were 6.78 s, 6.54 s, and 6.44 s. Timing is noisy, so this PR does not claim a compile-time improvement. ### Testing - `cmake --build build/Windows/RelWithDebInfo --config RelWithDebInfo --target onnxruntime_test_all --parallel 8` - `onnxruntime_test_all.exe --gtest_filter=CpuKernelRegistrationTest.*:KernelRegistryTests.*` from the test output directory: 9 tests from 2 suites passed. - Re-verified after the header move: `onnxruntime_test_all` builds and `CpuKernelRegistrationTest.RegistryIsNotEmpty` passes. - `lintrunner -a` on all changed files: no lint issues. --------- Co-authored-by: Gopalakrishnan Nallasamy <gopalakrishnan.nallasamy@microsoft.com>
This pull request updates the build pipeline to improve Python package installation and proxy handling, and ensures SSL certificate trust for Python requests. The most important changes are: **Build Pipeline Improvements:** * The pipeline now installs both `requests` and `truststore` Python packages to improve SSL certificate handling in Python scripts. * Added steps to inspect and sanitize the `NO_PROXY` environment variable, ensuring it does not contain invalid entries before running the Python GitHub Action wrapper. **Python Script Enhancements:** * Updated `run_gh_action.py` to import and inject `truststore` into the Python SSL context, ensuring trusted certificate handling for HTTP requests.
### Description - Recognize non-PCI DRM devices using the NVIDIA `nvgpu`/`nvidia` driver. - Surface them as integrated NVIDIA GPUs without PCI identity metadata. - Continue excluding AMD XCP and other non-NVIDIA platform devices. - Add focused Jetson-style sysfs discovery coverage. ### Motivation and Context Jetson GPUs use the platform/host1x bus and lack PCI vendor attributes, causing Linux device discovery to skip them. Consequently, the CUDA plugin EP is absent from `GetEpDevices()` despite loading successfully. <!-- START COPILOT CODING AGENT SUFFIX --> - Fixes microsoft#32541 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com>
…ft#33066) ## Description Preserve tensor element types across C API and ONNX boundaries without changing public enum values. Add explicit conversions where the ABI-stable `ONNXTensorElementDataType` meets TensorProto-numbered internal lookups, kernel constraints, and serialized metadata. ## Summary of Changes ### Type Representations These representations must not be treated as interchangeable: | Representation | Meaning | |---|---| | `ONNXTensorElementDataType` | Public C API enum with ABI-stable numeric values. | | `ONNX_NAMESPACE::TensorProto_DataType` | ONNX serialized element-type enum. | | `PrimitiveDataTypeBase::GetDataType()` | Returns an internal primitive's TensorProto-numbered element tag, not a C API enum. | | `MLDataType` | `const DataTypeImpl*`, an internal type descriptor. `DataTypeImpl::TypeFromProto(proto)` returns this pointer from an ONNX `TypeProto`; it does not return an enum. | The existing numeric differences remain unchanged: | Type | C API value (unchanged) | TensorProto value (unchanged) | |---|---:|---:| | `FLOAT8E8M0` | 26 | 24 | | `UINT2` | 24 | 25 | | `INT2` | 25 | 26 | ### Boundary Conversions - Add public, header-only `ToTensorProtoElementType` and `ToOrtTensorElementDataType` helpers without an ONNX protobuf dependency, and reuse them instead of direct enum casts. - Convert C API tensor creation, custom-op schemas and shape inference, standalone operator constraints, plugin EP type lookups, and Vitis AI custom-op constraints to TensorProto numbering before internal lookups or metadata writes. - Convert TensorProto-numbered tags from `PrimitiveDataTypeBase::GetDataType()` to C API values before plugin adapter calls. Leave `TypeFromProto` inputs and internal primitive tags in ONNX numbering. - Convert NVIDIA TensorRT RTX type checks at the ONNX-to-C-API boundary, and preserve type information in model-editor and graph serialization paths. ### Packed Tensors and Regression Coverage - Retain actual tensor byte-size handling in the internal testing EP so packed tensors are handled correctly. - Retain byte-sized and packed-type support in graph serialization, with error propagation for unsupported element sizes. - Cover all 27 enum conversions, invalid inputs, and the legacy C API numeric assignments. Add explicit checks that `TypeFromProto` and primitive type descriptors retain TensorProto numbering, and that both plugin adapter conversions return the correct type handles. - Retain regressions for dense/sparse tensor creation, custom-op registration and shape inference, standalone operators, plugin EP type lookup, model-editor metadata, and graph serialization. - Keep Go's copied C API enum ABI-stable and add compile-time guards for its three divergent values. ## Motivation and Context Directly passing C API `UINT2` (24) to a TensorProto-based lookup selects `FLOAT8E8M0`, while directly casting an internal `FLOAT8E8M0` tag (24) to the C API enum selects `UINT2`. Explicit conversions at the boundary fix both directions while leaving internal ONNX numbering and the public ABI intact. ## Compatibility No public enum values or ONNX serialization values change, and no C API version bump is introduced. Existing applications and plugins do not need to rebuild to accommodate enum renumbering because this PR does not renumber the enum. The public and Go copied headers document that the C API enum is not numerically identical to `TensorProto::DataType`. ## Testing - Compile-time checks and `TypeInfoTests.TensorElementTypeConversions` verify both conversion directions and round trips for all 27 enum entries, invalid inputs, and legacy public ABI assignments. - `TypeInfoTests.InternalTensorTypesUseTensorProtoNumbers` checks `TypeFromProto`, internal primitive tags, and the distinction from C API values for the three divergent types. - Run `onnxruntime_test_all --gtest_filter="TypeInfoTests.*:GraphTest.ModelEditorPreservesTensorElementTypes"` from the build output directory. - Run `onnxruntime_shared_lib_test --gtest_filter="CApiTest.*OrtElementTypes"` for dense/sparse tensor creation, custom-op registration and shape inference, and standalone operator constraints. - Run `onnxruntime_ep_graph_test --gtest_filter="EpGraphTest.ByteOrderConversion*:EpGraphTest.SerializeToProto_PreservesTensorElementTypes"` for serialization and byte-order regressions. - Run `onnxruntime_autoep_test --gtest_filter="OrtEpLibrary.GetTensorDataTypeUsesOrtElementTypes"` for plugin EP type lookup. - Run `ORT_LIB_PATH=<build-output-directory> go test ./onnxruntime -run '^(TestCreateTensor.*|TestGetVersion|TestNewTensorFromBytes|TestTensorDataTypeMismatch)$' -count=1` from `go/`. ## Checklist - [x] Regression tests added and updated. - [x] Public enum values and ABI compatibility preserved. - [x] Public-header type-numbering documentation updated. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com>
## Summary Fix two portability issues encountered by Windows clang builds that disable C++ exceptions: - Open MoE expert state files through the standard `std::filesystem::path` stream constructor instead of relying on MSVC's wide-string overload. Add an inference-session regression for ASCII and UTF-8/Unicode filenames. - Backport ONNX portability fixes into both the CMake and vcpkg patch mirrors: const-reference error helpers, tensor dimension errors routed through the no-exceptions-aware helper, and traits-based resource sentinels that avoid a non-constant `INVALID_HANDLE_VALUE` template argument. The ONNX changes are selected backports of already merged fixes: - onnx/onnx#8126 (const-reference error helpers) - onnx/onnx#8220 (tensor error routing) - onnx/onnx#8279 (resource sentinel portability) Keep the existing ONNX patches and the pinned ONNX 1.22.0 version unchanged. No opset or public API changes. ## Validation - Based on freshly fetched upstream main `b5cb51748b5b9efb3de90b31670c91b6829c4dba`. - Windows x64 CPU build of `onnxruntime_test_all`; all 30 selected native-path and neighboring MoE/KernelPilot tests executed and passed. - Full patch checked with both Git and GNU patch against a pristine ONNX 1.22.0 archive whose SHA1 matches the dependency pin; both mirrors are byte-identical. - All seven patched ONNX sources match the dependency sources used by the successful CPU build. - Windows clang-cl C++20 syntax checks with `-fno-exceptions` passed for the patched tensor/resource headers, native-path stream constructor, and `assertions.cc`. The probe rejects exception-enabled compilation. - Scoped lintrunner, whitespace, and editor diagnostics passed. Linux/macOS runtime tests and a full compiler-no-exceptions ORT build were not run for this latest-main branch.
…s/nextjs-default (microsoft#33138) Bumps [sharp](https://github.com/lovell/sharp) from 0.35.4 to 0.35.5. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/lovell/sharp/releases">sharp's releases</a>.</em></p> <blockquote> <h2>v0.35.5</h2> <p><a href="https://github.com/lovell/sharp-libvips/releases/tag/v1.3.4">https://github.com/lovell/sharp-libvips/releases/tag/v1.3.4</a></p> <ul> <li> <p>Add upper bounds check on length of <code>linear</code> and GIF <code>delay</code> arrays.</p> </li> <li> <p>Improve error handing when WebAssembly fallback also fails. <a href="https://redirect.github.com/lovell/sharp/pull/4593">#4593</a> <a href="https://github.com/lazerg"><code>@lazerg</code></a></p> </li> <li> <p>TypeScript: Allow multi-frame options for JXL output. <a href="https://redirect.github.com/lovell/sharp/pull/4602">#4602</a> <a href="https://github.com/ramin-010"><code>@ramin-010</code></a></p> </li> <li> <p>TypeScript: Remove non-existent named export. <a href="https://redirect.github.com/lovell/sharp/issues/4604">#4604</a></p> </li> <li> <p>Increase accepted dimensions when extending an image. <a href="https://redirect.github.com/lovell/sharp/issues/4605">#4605</a></p> </li> <li> <p>Improve gain map support for <code>extract</code> and <code>rotate</code> operations. <a href="https://redirect.github.com/lovell/sharp/issues/4606">#4606</a></p> </li> <li> <p>Tests: Ensure composite tests pass on big endian platforms. <a href="https://redirect.github.com/lovell/sharp/issues/4609">#4609</a></p> </li> </ul> <h2>v0.35.5-rc.1</h2> <p><a href="https://github.com/lovell/sharp-libvips/releases/tag/v1.3.4-rc.1">https://github.com/lovell/sharp-libvips/releases/tag/v1.3.4-rc.1</a></p> <ul> <li> <p>Add upper bounds check on length of <code>linear</code> and GIF <code>delay</code> arrays.</p> </li> <li> <p>Improve error handing when WebAssembly fallback also fails. <a href="https://redirect.github.com/lovell/sharp/pull/4593">#4593</a> <a href="https://github.com/lazerg"><code>@lazerg</code></a></p> </li> <li> <p>TypeScript: Allow multi-frame options for JXL output. <a href="https://redirect.github.com/lovell/sharp/pull/4602">#4602</a> <a href="https://github.com/ramin-010"><code>@ramin-010</code></a></p> </li> <li> <p>TypeScript: Remove non-existent named export. <a href="https://redirect.github.com/lovell/sharp/issues/4604">#4604</a></p> </li> <li> <p>Increase accepted dimensions when extending an image. <a href="https://redirect.github.com/lovell/sharp/issues/4605">#4605</a></p> </li> <li> <p>Improve gain map support for <code>extract</code> operation. <a href="https://redirect.github.com/lovell/sharp/issues/4606">#4606</a></p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/lovell/sharp/commit/51a990faa26ade5586a4934ac9673c98d8893326"><code>51a990f</code></a> Release v0.35.5</li> <li><a href="https://github.com/lovell/sharp/commit/96de105d9d36ab04c76c2b78b97255171630d236"><code>96de105</code></a> Upgrade to sharp-libvips v1.3.4</li> <li><a href="https://github.com/lovell/sharp/commit/3a61390260e0aa017046af9ad107abd62f3c98c0"><code>3a61390</code></a> CI: Configure Dependabot with all package.json locations</li> <li><a href="https://github.com/lovell/sharp/commit/4940c500a63a9193b1ad5d3804a2d241ca36995b"><code>4940c50</code></a> Improve gain map support for rotate/flip/flop ops</li> <li><a href="https://github.com/lovell/sharp/commit/20654aa60eaa85a37369ef69a7ca6506b65f8c85"><code>20654aa</code></a> Prerelease v0.35.5-rc.1</li> <li><a href="https://github.com/lovell/sharp/commit/ef4f9348a97053ceddacd23dcb6e39cce5a1fc7d"><code>ef4f934</code></a> CI: Upgrade to Ubuntu 26.04</li> <li><a href="https://github.com/lovell/sharp/commit/358df954c5f58487392e1a9af743e112e8998d67"><code>358df95</code></a> Upgrade to libvips v8.18.7</li> <li><a href="https://github.com/lovell/sharp/commit/49f4903d34eddcd68b79872c511c198f124885a3"><code>49f4903</code></a> Improve gain map support for rotate-then-extract <a href="https://redirect.github.com/lovell/sharp/issues/4606">#4606</a></li> <li><a href="https://github.com/lovell/sharp/commit/0e2e55eb3a44da5d602adedd6b51a03e652e3b20"><code>0e2e55e</code></a> Silence a couple of compiler/static analysis warnings</li> <li><a href="https://github.com/lovell/sharp/commit/cef3b8c467c21f7e84bd51d5e53f2d115669f8d8"><code>cef3b8c</code></a> Improve gain map support for extract operation <a href="https://redirect.github.com/lovell/sharp/issues/4606">#4606</a></li> <li>Additional commits viewable in <a href="https://github.com/lovell/sharp/compare/v0.35.4...v0.35.5">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/microsoft/onnxruntime/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
### Description Optimize CPU `BatchNormalization` for inputs with a singleton spatial size, such as `[N, C]`, `[N, C, 1]`, and `[N, C, 1, 1]`. The existing implementation processes these inputs as `N * C` spatial blocks of length one. This change maps each sample as one contiguous channel vector, allowing Eigen to vectorize computation across channels. Training mean and variance are accumulated using the same layout. The existing implementation remains unchanged for inputs with larger spatial dimensions. The change introduces no additional buffers, tensor transposes, public API changes, or threading changes. For a float32 `[2000, 64]` input on an AMD EPYC 9654: | Mode | Before | After | Speedup | |---|---:|---:|---:| | Inference | 0.3334 ms | 0.0297 ms | 11.23× | | Training | 0.8225 ms | 0.0625 ms | 13.15× | ### Motivation and Context Addresses the `BatchNormalization` bottleneck reported in microsoft#25599. For NC-shaped inputs exported from PyTorch, the current CPU kernel performs scalar-sized operations for each `(N, C)` pair even though channels are contiguous in memory. The per-block overhead makes `BatchNormalization` significantly slower than the equivalent vectorized operation. Legacy exports mentioned in the issue may also contain a slow `ReduceMean` over `[N, C, 1]`. That is a separate CPU reduction issue and is outside the scope of this PR. ### Validation - All 20 `BatchNormTest.*` tests pass. - Added 72 CPU test cases covering: - float and double - inference and training - opsets 9, 14, and 15 - rank-one and NC inputs - singleton spatial dimensions - single batch/channel inputs - SIMD tails and constant channels - Verified output, running statistics, and opset-9 saved statistics. - All 86 captured benchmark output arrays were bitwise identical between the baseline and patched builds on the tested platform. - Benchmarked 40 configurations across spatial and contiguous layouts, float/double, inference/training, and one/four threads. No measured regression exceeded both 20% and 1 microsecond. - `lintrunner` and `git diff --check` pass.
…ization ranges (microsoft#33104) ### Description Promote both range endpoints to float64 before subtracting them in `compute_scale_zp`. The scale is still returned in the original floating-point dtype, and the zero point keeps the quantized dtype. Add coverage for FP16 and FP32 ranges with signed and unsigned 8-bit and 16-bit quantization, including symmetric ranges. Add QDQ MatMul tests that quantize large finite weights and run CPU inference with per-tensor and per-channel weight quantization. The FP16 inference cases first check that the current CPU build can initialize the original FP16 MatMul. They skip only when that kernel is unavailable; the FP32 inference cases and dtype-level regression tests remain enabled. ### Motivation and Context The current subtraction runs in the input dtype before its result is converted to float64. For example, FP16 endpoints of -40000 and 40000 are both finite, but their difference overflows FP16. This produces an infinite scale even though dividing the range by the quantized range would produce a representable scale. FP32 inputs can encounter the same problem. Doing the subtraction in float64 also changes rounding for some finite, non-overflowing ranges. This change therefore preserves the output dtypes, but does not promise identical scale or zero-point values for every previously finite input. Validation on macOS arm64, Python 3.12.3, NumPy 2.5.3: - All 20 new regression subtests fail against the original function and pass with the change. - The complete `test_quant_util.py` and `test_op_matmul.py` modules report 15 passed, 8 skipped, and 35 subtests passed. These totals include the new regressions. The skips cover missing SciPy, existing FP16 calibration failures, and unsupported FP16 QLinearMatMul cases; the four new QDQ inference cases run successfully. - Repository-pinned `lintrunner` and `git diff --check` pass. After adding the CPU kernel check, both modules pass again with the same counts on this arm64 machine. A separate injected-error probe confirms that a missing FP16 MatMul skips only its two subcases while both FP32 cases still run, and an unrelated missing-operator error is not skipped. This probe does not replace a native x86/x64 test run. The tests import the checkout's Python quantization code and use the ONNX Runtime 1.30.0 CPU wheel for native execution. This validation does not include a native build from the current checkout, GPU execution, or the full repository test suite.
…uced (microsoft#33091) ### Description **What happens** `FuseContiguousReshapes` folds a chain of `Reshape` / `Squeeze` / `Unsqueeze` nodes into a single `Reshape`: - The replacement node consumes only the **data input** of the first node; its shape comes from a **new constant initializer** derived from the last node's inferred output shape. - `FinalizeNodeFusion` moves the first node's input edges onto the replacement **by tensor name** and asserts every moved name is an input of the replacement. - A shape input **produced by another node** (e.g. a `Where`) has no matching input on the replacement, so the assert fires in `GetIndexFromName` and **session initialization fails**. Initializer-backed shape inputs carry no edge, which is why only dynamic shapes hit this. **The fix** (in `onnxruntime/core/optimizer/reshape_fusion.cc`): 1. Before the fusion, remove the edge feeding the first node's shape input when a node produces it. 2. After the fusion, delete that producer when the fusion removed its last consumer and it does not feed a graph output. This mirrors the handling `SimplifiedLayerNormFusion` uses for inputs the replacement does not consume (microsoft#29196). **Regression tests** (in `onnxruntime/test/optimizer/graph_transform_test.cc`): - `ReshapeFusionContiguousReshapesWithNodeProducedShape` — transformer level: the chain collapses to one `Reshape` with constant shape `{6}` and the `Where` producer is deleted. - `ReshapeFusionContiguousReshapesWithNodeProducedShapeExecution` — full `InferenceSession` run: initialization succeeds and `Run()` returns the input reshaped to `[6]`. - Both reproduce the reported error without the fix. ### Motivation and Context **Fixes microsoft#32837** A model whose first of two chained `Reshape`s takes a runtime-computed shape fails session creation under `ORT_ENABLE_BASIC` and above: `Where(cond, target_shape, target_shape) → Reshape(x, shape1) → Reshape(r1, shape2) → y` > `GetIndexFromName ... Attempting to get index by a name which does not exist: shape1 for node: reshape_dynamic_new_reshape` - The only workaround is disabling graph optimizations; `ORT_DISABLE_ALL` runs fine, so the model itself is valid. - The fusion is semantically correct for these graphs — the target shape is fully determined by the last node's inferred output shape — so the fix **repairs the edge handling and keeps the optimization** rather than bailing out. **Verified locally** (macOS arm64, Release, CPU EP): - `GraphTransformationTests.ReshapeFusion*`: 24/24 pass, including both new tests - `onnxruntime_test_all`: 2267 passed / 39 skipped / 0 failed - `lintrunner`: clean
…soft#33089) ### Description - Add an explicit float16 zero `value` attribute when converting a `ConstantOfShape` node that omits the optional attribute. - Add regression coverage for the `Shape → ConstantOfShape → Add` pattern. ```python helper.make_node( "ConstantOfShape", ["shape"], ["constant"], value=helper.make_tensor("value", TensorProto.FLOAT16, [1], [0]), ) ``` ### Motivation and Context `ConstantOfShape` defaults to a float32 zero when `value` is absent. Retyping its output to float16 without updating this default produces an invalid ONNX model. This ensures dynamically exported models remain loadable after float16 conversion. <!-- START COPILOT CODING AGENT SUFFIX --> - Fixes microsoft#33087 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
## Summary - Include string object storage and payload bytes in embedded initializer size limits. - Use overflow-safe cumulative accounting while preserving normal and exact-boundary payloads. ## Validation - Warnings-as-errors build completed successfully. - Focused string initializer boundary tests passed. Co-authored-by: danielsongmicrosoft <danielsongmicrosoft@users.noreply.github.com>
## Summary - Reject negative, overflowing, and mismatched ORT-format input argument counts during load. - Preserve valid variadic input count normalization. ## Validation - Warnings-as-errors build completed successfully. - Focused invalid and valid ORT-format input-count tests passed. Co-authored-by: danielsongmicrosoft <danielsongmicrosoft@users.noreply.github.com>
### Description - `CONTRIBUTING.md`: the docs folder link lacked `tree/main` and 404s - `docs/onnxruntime_extensions.md`: `custom_text_ops.md` was merged into `custom_ops.md` in onnxruntime-extensions - `docs/ReleaseNotesWorkflow.md` and `docs/design/webgpu_paged_attention.md`: the skills live under `.github/skills`, not `.agents/skills` (4 links) - `docs/python/README.rst`: release notes link `v1.24.0` and `v1.13.0`, which were never published; `v1.24.1` and `v1.13.1` are the first releases of those series - `docs/Roadmap.md`: `InferenceHighLevelDesign.md` does not exist anywhere in the repo, so the reference is kept as plain text - two Python notebooks: ONNX model zoo paths moved under `validated/` ### Motivation and Context All eleven links return 404; each new target was confirmed to exist via the GitHub API. Docs only.
## Description CUDA 13.4 headers trigger MSVC warnings C4100 and C4193 while compiling the CUDA execution provider plugin, and NVCC-generated objects can trigger LNK4099 because their compiler PDB is not retained. Warning-as-error builds therefore fail on toolkit diagnostics outside ONNX Runtime's control. This change suppresses only those known diagnostics for the Windows CUDA plugin while preserving warnings-as-errors for ONNX Runtime code. ## Summary of Changes ### CUDA plugin build configuration - Suppress MSVC C4100 from `cuda_fp4.hpp` and C4193 from CCCL headers for CUDA 13 and newer. - Ignore LNK4099 when linking the CUDA plugin because NVCC does not retain the referenced host compiler PDB. - Keep all suppressions scoped to MSVC CUDA plugin builds. ## Testing - `lintrunner -a` - Regenerated the existing Visual Studio build with the `ZERO_CHECK` target. - Verified the generated CUDA plugin project contains `/wd4100`, `/wd4193`, and `/IGNORE:4099` in the expected compile and link configurations. ## Checklist - [x] No breaking changes - [x] No documentation changes required - [x] Build configuration regenerated successfully
…crosoft#33001) ## Description Factors the **sequence-length-independent** parts of the CUDA `GroupQueryAttention` backend-selection logic out of `GroupQueryAttention::ComputeInternal` into a new header-only module, `group_query_attention_eligibility.h`. It provides four pure "seq-free" route-eligibility predicates: | Helper | Route | |---|---| | `IsGQAXqaEligibleSeqFree<U>` | XQA (Extreme Query Attention) decode kernel | | `IsGQACudnnSdpaCoreEligibleSeqFree<T, U>` | cuDNN SDPA | | `IsGQAFlashEligibleSeqFree<T>` | FlashAttention | | `IsGQAMemoryEfficientEligibleSeqFree<T>` | CUTLASS memory-efficient attention | A **seq-free** predicate depends only on partition-time facts (geometry, device capability, quantization, feature flags) and *not* on the per-step sequence length or phase (prompt vs decode). The runtime keeps the sequence-length-dependent gates and dispatch priorities at the call sites: - **XQA** — phase gates (`is_first_prompt` / `sequence_length == 1` / `kv_sequence_length`) and the per-node shared-memory capability probe. - **cuDNN** — `cudnn_sdpa::is_stable()` and the seq-dependent `is_supported(...)` probe. - **Flash / MEA** — the `!use_xqa` / `!use_cudnn_sdpa` / `!use_flash_attention` dispatch priority. ## No functional change The eligibility booleans are computed from the same expressions in the same short-circuit-equivalent order. Every moved term is a side-effect-free read, so the selected backend is identical for every input. The GQA operator tests exercise all four routes and validate this (a wrong predicate would dispatch the wrong kernel and fail the tests). ## Why header-only All four helpers are templated on the Q/KV element types (`T`/`U`), so the module is header-only rather than the usual `.h`/`.cc` split. ## Motivation This is a small NFC step toward **policy-based, capacity-aware CPU offloading** (see the design doc in microsoft#32944). The shared seq-free core is what the Level-1 workspace estimator will later reuse, so runtime route selection and workspace estimation cannot drift. ## Relationship to sibling PRs - microsoft#32944 — design doc (policy-based workspace estimation). - microsoft#32945 — consolidates GQA **scratch sizing** into recipe helpers. This PR is the sibling that consolidates **route eligibility**. Both are independent NFC refactors; whichever merges second will need a trivial merge in the XQA/Flash region of `group_query_attention.cc`. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7fa98d07-aec3-41b7-92b8-b0ec317a95bf Copilot-Session: 37ad7066-fcd3-45df-891e-44bee9e1cf04
… shared workspace recipes (microsoft#32945) ## Description Consolidates the GroupQueryAttention (CUDA) runtime scratch **sizing** for the memory-efficient (MEA), unfused fallback, XQA, and Flash paths into the shared workspace recipe builders that already back the Level-1 workspace estimator. `ComputeInternal` previously sized these buffers with inline math that duplicated the estimator''s recipe logic, so the allocation and the estimate could drift. This routes the runtime allocations through `GetGQAMemoryEfficientWorkspaceRecipe`, `GetGQAUnfusedWorkspaceRecipe`, `GetGQAXqaWorkspaceRecipe`, and `GetGQAFlashWorkspaceRecipe`, giving a single source of truth. ### No functional change (byte-identical) The recipe byte counts are identical to the previous inline sizing: - **MEA** expanded-K/V = `sizeof(T) * batch * num_heads * seqlen_present_kv_cache * head_size`; FP32 accumulator = `sizeof(float) * batch * sequence_length * num_heads * head_size`, gated exactly as `MemoryEfficientAttentionParams::need_workspace(head_size, /*is_float=*/false)` (i.e. `head_size > 128`). - **Unfused** reproduces the 256-byte-aligned Q and Y regions plus the two `GetUnfusedAttentionWorkspaceSize`-equivalent QK/softmax regions (same 256 alignment → total bytes and all sub-buffer offsets unchanged). - **XQA** reproduces the internal scratch (`GetXQAScratchSize`) plus the optional 256-aligned rotary-Q/K and dynamic head-sink regions, with the same pointer wiring. The prepacked head sink (`xqa_head_sink_`) stays out of scratch. - **Flash** reproduces `get_num_splits_and_buffer_sizes` (including the fast-decode accumulator correction) and preserves the `parameters.num_splits` side effect and the two accumulator `cudaMemsetAsync` guards. - GQA is only instantiated for `MLFloat16` / `BFloat16` (`qkv_element_size == 2`); the MEA/unfused fallbacks are only reached with an unquantized KV cache, satisfying the recipe validators. ### Scope - One file: `onnxruntime/contrib_ops/cuda/bert/group_query_attention.cc`. - All four backend scratch sizings now source from recipes. - The **preparation** allocations (windowed staging, compaction, seq-lens vectors, and the `GQABufferRequirements` QKV preprocess buffer) are intentionally **deferred to a follow-up PR**. Unlike the backend scratch, those allocations are interleaved with route selection and mutate `seqlen_present_kv_cache`, so a byte-identical migration needs a dedicated runtime-equivalence test (mirroring the XQA/Flash equivalence tests) first. - Part of the workspace-estimation series; see the design doc PR microsoft#32944. ### Validation Relying on CI''s CUDA build + GQA/workspace tests (a local CUDA build was disk-constrained). Because sizes and offsets are byte-identical, existing `group_query_attention_workspace*_test` and GQA op tests pass unchanged. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7fa98d07-aec3-41b7-92b8-b0ec317a95bf Copilot-Session: 37ad7066-fcd3-45df-891e-44bee9e1cf04
…tor (reader-only) (microsoft#33014) ## What Adds the session option `ep.cuda.gqa_workspace_max_total_sequence_length` and threads its value into the CUDA GroupQueryAttention (GQA) Level-1 workspace estimator. `total_sequence_length` (accumulated past + current KV tokens) is a **runtime scalar input** whose value cannot be recovered from graph shapes. For the non-windowed KV cache, the backend workspace scales linearly with it, so the L1 estimator currently returns `std::nullopt` for that case (it cannot bound the value). This knob lets the user hand the estimator that **KV-length envelope** directly, for capacity-aware partitioning (microsoft#31962). ## No functional change (reader-only) This is a **pure NFC reader**: - The value is parsed (`WorkspaceEstimatorConfig` → `GQAWorkspaceEstimateConfig::max_total_sequence_length`) and **stored**, but **not consumed**. - `BuildBounds` still returns `std::nullopt` for the non-windowed case, so **the estimate output is unchanged** for all inputs. - A follow-up PR (PR 4) consumes the field to produce the non-windowed bound. The plumbing mirrors the existing `ep.cuda.fpa_intb_*` knobs (config key → `WorkspaceEstimatorConfig` → CUDA EP parse → estimator param). ## Test Extends the existing `RealAccountantTest.FactoryRetainsNarrowWorkspaceEstimatorConfig` framework test to validate the reader end-to-end (config string → `WorkspaceEstimatorConfig` field) in CPU CI. ## Context Part of the policy-based workspace-estimation PR chain (design doc: `docs/annotated_partitioning/policy_based_workspace_estimation.md`). This is **PR 3 (envelope plumbing)** in the roadmap: - PR 1 — GQA scratch sizing consolidation (microsoft#32945) - PR 2 — route-eligibility extraction (microsoft#33001) - **PR 3 — envelope plumbing (this PR)** - PR 4 — route-aware, envelope-bounded estimate (consumes this field) --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7fa98d07-aec3-41b7-92b8-b0ec317a95bf Copilot-Session: 37ad7066-fcd3-45df-891e-44bee9e1cf04
### Description <!-- Describe your changes. --> Raise the Android minimal native binary size threshold from 1,605,632 bytes to 1,607,680 bytes, the next 2 KiB boundary above the current measured size of 1,607,126 bytes. This leaves 554 bytes of headroom. Validation: - `git diff --check` — passed - Android minimal CI on `main` measured 1,607,126 bytes against the current 1,605,632-byte threshold — failed by 1,494 bytes ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> microsoft#32867 added string-payload accounting to the common embedded-initializer validation path. That code is part of model loading and remains linked into the no-operator Android minimal build, increasing its measured section size from 1,605,030 to 1,607,126 bytes. microsoft#32867 passed its original Android size check against an older `main`, but intervening changes had consumed the available headroom by the time it merged. As a result, the current `main` branch and unrelated pull requests now fail `AndroidBinarySizeCheckJob_MinimalBaseline` with the same 1,607,126-byte artifact. This change updates the baseline to match the intentional model-validation code now present on `main`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…on (microsoft#33179) ## Summary Restore the function-local `is_int8` cache-type predicate in CUDA GroupQueryAttention, next to `is_fp8`. This fixes the main-branch Windows CUDA Plugin CI failure: https://github.com/microsoft/onnxruntime/actions/runs/37663718452/job/112937456604 ```text group_query_attention.cc(595): error C2065: 'is_int8': undeclared identifier ``` ## Why the prerequisite PRs passed CI This is an integration regression between microsoft#33001 and microsoft#32945, not a plugin-only feature-guard issue. - microsoft#32945's tested head `fcbdcf7722` had the local predicate. Its Windows and Linux plugin CI runs passed on October 6: https://github.com/microsoft/onnxruntime/actions/runs/37501448666 https://github.com/microsoft/onnxruntime/actions/runs/37501449256 - microsoft#33001 subsequently merged and removed the predicate while extracting XQA eligibility into a helper. - microsoft#32945 then merged and introduced another use of the predicate in workspace-recipe bit-width normalization. The resulting combined main-branch file had the reference but no declaration. The one-line restoration preserves the intended INT8/FP8 normalization and does not change dispatch, allocation formulas, or lifetimes. ## Validation - Reproduced the exact C2065 failure from the pre-fix main source with the actual plugin adapter force-includes and compile definitions. - Compiled the corrected GQA translation unit with MSVC `/W4 /WX` in the plugin configuration (`BUILD_CUDA_EP_AS_PLUGIN`, `ORT_USE_EP_API_ADAPTERS=1`, `ORT_API_MANUAL_INIT`; adapters included before `cuda_kernel_adapter.h`). - Compiled the same corrected source in the standard shared-provider configuration. - `clang-format --dry-run --Werror` and `git diff --check` passed. These are fresh targeted translation-unit builds, not a full plugin DLL build or GPU inference run. The PR's CI covers the full build. Copilot-Session: 7fa98d07-aec3-41b7-92b8-b0ec317a95bf
…3030) **What** Updates the broken technical reference documentation link in CONTRIBUTING.md. The previous link: https://github.com/microsoft/onnxruntime/docs returns a 404. It has been updated to point to the correct docs/ directory: https://github.com/microsoft/onnxruntime/tree/main/docs **Context** The docs/ directory contains the technical reference documentation for developers working on ONNX Runtime source code. Fix Updated the broken link in CONTRIBUTING.md No source code or functionality changed Fixes microsoft#33018
ai-fw-intg
requested review from
Jaswanth51,
ankitm3k,
jatinwadhwa921 and
vthaniel
October 7, 2026 20:38
hdharpure9922
self-requested a review
October 8, 2026 04:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated daily backmerge from ORT main to ovep-develop. No conflicts detected. Do NOT squash or rebase - use merge commit only.