Sync with Microsoft ONNX Runtime - 18102026 by ai-fw-intg · Pull Request #1346 · intel/onnxruntime

ai-fw-intg · 2026-10-17T20:39:02Z

Automated daily backmerge from ORT main to ovep-develop. No conflicts detected. Do NOT squash or rebase - use merge commit only.

### Description  Make `NchwcOptimizerTests.ConvWithZeroInputChannelsIsNotTransformed` a graph-only regression instead of running the zero-channel convolution through `NchwcOptimizerTester`. - Build and resolve the same zero-input-channel Conv with and without bias. - Assign the Conv to the CPU EP and apply `NchwcTransformer` directly, following the adjacent rank-validation regression's pattern. - Assert that the transformer reports no modification, the ordinary Conv remains, and no NCHWc Conv, ReorderInput, or ReorderOutput nodes are inserted. - Explicitly skip platforms without NCHWc support. No production optimizer or kernel changes. The zero-output-channel regression and other numerical optimizer tests are unchanged. Validation: whitespace (`git diff --check`) and changed-range clang-format checks passed. No local build or C++ test run, as requested. ### Motivation and Context  Follow-up to microsoft#33346. The new regression currently executes the degenerate Conv at optimization levels 2 and 3 and compares the outputs. macOS arm64 CI reports NaN-versus-zero mismatches at `nchwc_optimizer_test.cc:235`, despite this test's intended contract being that NCHWc conversion is skipped. The same assertion is confirmed independently on upstream `main` and multiple PRs: - [main at 9cdf0f1, macOS WebGPU arm64 Release](https://github.com/microsoft/onnxruntime/actions/runs/38675468798/job/116227944699) - [microsoft#33016, macOS WebGPU arm64 Debug](https://github.com/microsoft/onnxruntime/actions/runs/38599291872/job/116132732574) - [microsoft#33347, macOS WebGPU arm64 Debug](https://github.com/microsoft/onnxruntime/actions/runs/38707806696/job/116332127965) Check the optimizer's graph contract directly without making this regression depend on zero-channel Conv execution. This is a test-only CI fix, not a correction to CPU Conv's zero-channel numerical behavior. Co-authored-by: Chi Lo <lochi@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b95559c4-5cbe-4787-8dd7-ffa45a2157f9

…oute-aware L1 estimate) (microsoft#33016) ## Summary Adds opt-in, route-aware Level-1 workspace estimation for non-windowed CUDA GroupQueryAttention using the declared total-KV envelope from `ep.cuda.gqa_workspace_max_total_sequence_length`. The present-cache capacity bound is **`max(allocated past-cache capacity, declared total-KV envelope)`**. Static caches may be larger than the active sequence: GQA permits `total_sequence_length <= past_sequence_length` and retains that allocated capacity. Bounding only the envelope would underestimate MEA expansion and partial-alias preservation. The existing graph-free aggregate sizes reachable routes and admitted prompt/decode phase corners; this PR does not introduce a second sizing engine. ## Envelope validation and caller contract - Explicit values are parsed once during session initialization and must be decimal integers in **`[0, INT32_MAX]`**, matching the runtime ABI. Malformed, negative, overflowing, and greater-than-`INT32_MAX` values fail initialization. - Zero remains the unspecified value. With the envelope unset or zero, non-windowed estimation is unavailable; windowed estimation continues to use its cache-shape bound. - The envelope must cover the query-length bound, and the resulting cache-capacity bound must fit the runtime int32 ABI. - The query/max-shape and total-KV declarations are estimation inputs, **not runtime-enforced admission limits or a no-OOM guarantee**. The proof applies only within the caller-declared workload envelope; runtime enforcement remains follow-up work. ## Workspace accounting and proof policy - Capacity-sensitive preparation and MEA expansion cover the full static-cache extent, even when it exceeds the active total-KV envelope. - Partial aliasing copies the full past tensor. Its preservation scratch coexists with the chosen route and is added above the per-route maximum, bounded by the full cache-capacity bound. - Preparation recipes already account for sequence-length vectors; there is no new missing-vector gap. - Reachable cuDNN without a workspace oracle, unsupported attention-bias cases, and otherwise unprovable route domains conservatively make the estimator unavailable. - For capacity-aware CUDA placement, every GQA candidate must have a successful analytical estimate. If proof is unavailable, CUDA assignment for that node is declined. - The generic accountant fallback is an ad hoc safety margin, not a proven GQA bound. Profile data is empirical and likewise cannot substitute for the missing analytical proof. Unrelated operators retain their existing generic fallback behavior. ## Regression coverage - Accepted boundary value `INT32_MAX`; rejected `INT32_MAX + 1`, `INT64_MAX`, decimal overflow, malformed text, and negative values. - Unset-envelope behavior and monotonicity when the envelope increases. - Static past capacity **1024** with total-KV envelope **512**, checking full preservation and MEA expansion against complete concrete runtime recipes. - An envelope below the query-length bound still declines, even when the static cache is larger. - Static capacity outside the runtime int32 ABI declines. - Parsed-envelope plumbing and unchanged windowed estimates. - Additive full-past preservation above the per-route maximum. - Capacity-aware CUDA placement declines GQA when the analytical estimator is unavailable, including when profile data exists; unrelated operators retain generic fallback. ## Dependencies Targets `main`, which now includes microsoft#33118's shared preparation-sizing implementation. The duplicated sizing commits were dropped during the post-merge rebase; this PR contains only its envelope-estimation, hardening, test, and documentation changes. Part of the policy-based workspace-estimation series described in microsoft#32944. Committed per-`(node, phase)` route policy and runtime workload-envelope enforcement remain follow-up work. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Chi Lo <lochi@microsoft.com> Copilot-Session: 7fa98d07-aec3-41b7-92b8-b0ec317a95bf Copilot-Session: c132130a-b0c7-422c-9ff7-75826b6ea3fc Copilot-Session: b95559c4-5cbe-4787-8dd7-ffa45a2157f9

…rosoft#33241)" (microsoft#33391) ### Description Fully reverts microsoft#33241 (commit `67c61ae72012fe588b9683a970622a5d713d093f`), restoring the previous Python factory selection, CMake test condition, Windows WebGPU workflow, plugin package smoke test, and documentation. The patch is the complete inverse of the original commit across all five files. ### Motivation and Context The original change was motivated by building a host wheel with `--use_webgpu shared_lib --build_wheel` for GenAI WebGPU benchmarks. The intended setup is a plain ORT host built without `--use_webgpu`, together with a separately built WebGPU plugin EP. GenAI uses native plugin registration and device-selection APIs, so those benchmarks do not require the ORT Python factory changes. This restores the plugin package test to its original scope and removes the provider-option assertion that fails with the supported ORT 1.24.4 host. It also removes the Python provider-name fallback added by microsoft#33241; explicit plugin device selection remains available. Supersedes microsoft#33389. ### Validation - Verified the staged patch matches the original commit's complete inverse using `git patch-id --stable`. - `lintrunner` and `git diff --check` passed for all changed files. Workflow YAML and all 17 explicit PowerShell steps passed syntax parsing. - Restored package smoke test passed on Windows with plain ORT 1.31.0 and plugin 0.5. - With plain ORT 1.24.4 and plugin `0.6.0.dev20261014085412`, the packaged DXC 1.8.2502.8 still encounters the separate local `DXC compile failed: E_FAIL` failure. The full smoke test passed after temporarily substituting DXC 1.9.0.1 in the isolated test environment; the original DLL was restored and its SHA-256 verified. No compiler or packaging changes are included. No native rebuild or Linux/macOS execution was performed; these smoke tests do not establish full packaging-pipeline success.

…t#32716) (microsoft#32775) **Problem** `ConvActivationFusion` in the core library writes the fused `activation` attribute, while the table of activations the WebGPU Conv kernel can fuse lives in the EP. A core library newer than the WebGPU EP binary (an `onnxruntime` wheel next to an older `onnxruntime-ep-webgpu` plug-in) can therefore fuse an activation the EP does not implement, and session creation failed with a bare `GetFusedActivationAttr(info, activation_).IsOK() was false` that named neither the activation nor the cause. **Solution** - Throws the `Status` from `GetFusedActivationAttr` instead of `ORT_ENFORCE`-ing it, so the error names the activation - Says where the attribute comes from and how to work around a version skew (`ORT_ENABLE_BASIC` keeps the activation as its own node) - Drops the duplicate parse in `FusedConv`; the `Conv<_, is_fused = true>` base already does it **Testing** - `Conv_WebGPU.UnknownFusedActivationIsReportedByName` builds a `FusedConv` with an unknown activation and checks the error names it; `FusedActivationParamCountMismatchIsReported` does the same for a wrong `activation_params` count. Both pass on Metal; the first also on Vulkan/lavapipe. Existing `FusedConvTest.*` cases still pass on WebGPU - Split out of microsoft#32717 at reviewer request Refs microsoft#32716

…on (microsoft#33094) ### Description Adds offline NVFP4 weight-layout support, native GEMV optimizations, and default-on fused block-scaled FP8 execution to CUDA QMoE. **Offline NVFP4 weights** Reuse the existing `weights_prepacked` attribute with quantization-specific semantics: | Value | CUDA NVFP4 storage | |---|---| | Omitted / `-1` | Legacy N-packed raw storage | | `0` | Explicit legacy N-packed raw storage | | `1` | Offline K-packed row-major storage | Mode `1` retains logical `[E,K,N/2]` tensor dimensions while storing physical `[E,N,K/2]` bytes. Integer quantization semantics remain unchanged; mode `2` is unsupported. - Preserve logical tensor dimensions, checkpoint weight codes, and scale values while sharing the packed payload across GEMV, native GEMM, and dense fallback. - Use contiguous vector loads and warp reductions, with support for up to 80 expanded expert rows when other GEMV eligibility requirements hold. - Optimize short reductions, including Qwen FC2 shapes. - Add experimental two-row FC1 weight/scale reuse for repeated expert groups, controlled by `ORT_NVFP4_FC1_EXPERT_REUSE`. - Add layout, dispatch, fallback, and CUDA-graph regression coverage, plus standalone NVFP4 parity and timing probes. FC1 reuse is disabled by default because its performance depends on expert overlap and can regress sparse expert distributions. Existing integer-quantization semantics and legacy NVFP4 defaults remain unchanged. **Block-scaled FP8 execution** - On CUDA SM80-or-newer GPUs, supported SiLU/SwiGLU block-FP8 workloads use fused weight-only GEMV/GEMM by default, decoding and scaling weight tiles on chip without full FP16/BF16 weight buffers. - Set `ORT_ENABLE_FP8_FUSED=0` before session creation to select the compact dense-dequant fallback; unsupported activations or routing sizes retain that fallback. - Update the CUDA QMoE guide to match the implemented dispatch and fallback contract. ### Motivation + Context Large NVFP4 MoE models can incur additional memory and initialization costs from runtime weight repacking and expanded scale buffers. Accepting an offline K-packed payload reduces that duplication without export-time dequantization or requantization. Multi-token verification with top-10 routing can produce 70–80 expanded expert rows. Extending GEMV eligibility avoids fallback solely because the previous 64-row limit was exceeded. Short-reduction tiling and optional FC1 reuse target additional execution costs. Packed-QSA optimizations are isolated in [microsoft#33328](microsoft#33328). ### Review Fixes and Validation - Merge current `main`, preserving both CUDA fused-FP8 and WebGPU block-FP8 documentation. - Dense NVFP4 fallback now selects retained raw FC1/FC2 block-scale inputs rather than native TMA-swizzled buffers. This avoids an additional raw-scale allocation for offline weights. - Add FP16/BF16 regression cases for nine top-10 decode tokens (90 expanded rows), with prepacking enabled and disabled. - All four new cases pass on the existing installed A100 CUDA artifact; dispatch diagnostics confirm the dense-dequant route. This is baseline test validation, not a fresh build of the fix. - A standalone compiled probe using the edited FC1/FC2 selection expressions passes 16 raw/swizzled-buffer checks. Python lint, syntax, editor diagnostics, and PR-diff whitespace checks pass. - A fresh ORT build and SM120/SM121 runtime validation were not performed; only SM80 hardware is available locally. --------- Co-authored-by: Akshay Sonawane <asonawane@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Sayan Shaw <52221015+sayanshaw24@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

…aring (microsoft#33224) ### Description Introduce `PackedSparseAttentionIndexerMerge` and extend `PackedSparseAttentionIndexer` to support index sharing, configurable output capacity, and explicit state-sharing policies. - Add the merge operator for combining base and additional sparse-attention indices, with CUDA and WebGPU support and dedicated documentation. - Extend input/output slot definitions, operator schemas, and kernel registration for shared indices and merged indices, counts, and status, including in-place merged-tensor operations. - Update the CUDA indexer to handle `indexshare_mode`, validate interface consistency, and manage merged outputs. - Add `max_output_entries` to control output capacity consistently across output shapes and QSA/CSA kernel parameters. - Add `indexshare_state_policy` to control CSA state advancement and sharing, with validation and error handling for incompatible interface usage. ### Motivation and Context Advanced sparse-attention workflows need to reuse selected indices and combine them with additional selections rather than generate each index set independently. These changes provide explicit index-sharing and merging interfaces to support those workflows. Configurable output capacity lets callers bound selection outputs, while an explicit CSA state policy distinguishes state advancement from sharing. Together, these controls make index reuse more flexible and state handling more predictable in large-scale transformer models. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ft#32796) ### Description Prefer the 16x16x16 FP16 subgroup-matrix configuration with subgroup size 64 for MatMulNBits on AMD D3D12 adapters that report support for it. Retain the existing wave32 fallback and preserve other adapters' configuration preferences. Adapt the workgroup tile shape and K blocking to the workload. Small M uses a single-wave 64x64 tile when N is exactly 64, avoiding an entirely padded column wave; other small-M widths use a two-wave 64x128 tile. Larger M uses two or four waves sharing dequantized weights. Reduce K blocking when necessary to divide K exactly, and include the tile dimensions in the program cache key. Add self-contained WebGPU 4-bit and 8-bit correctness tests covering the single-, two- and four-wave tile policies, K-block reductions and partial prepack/output tiles, with and without bias and zero points. The tests explicitly select WebGPU and disable CPU fallback. ### Motivation and Context The existing wave32 tile policy leaves performance available on AMD D3D12. Wave64 and workload-dependent tiling improve most measured prefill shapes by sharing dequantized weights across row waves and reducing barriers with larger K blocks where beneficial. This uses the configuration selection already landed in microsoft#32784. ### Performance (measured 2026-10-13) Phi-4 Mini, native ORT WebGPU/D3D12: PR implementation at `36f6c04eb2` compared with main's wave32 implementation at `9036f076a2`. | Metric | Result | |---|---:| | Median per-shape prefill gain across 11 prompt lengths, 32–2048 | +9.0% | | 512-token prefill gain | +11.0% | | 512-token throughput, wave64 / wave32 | 6,974 / 6,283 tokens/s | | 32-token prefill difference | -1.3% | | Median per-shape decode difference | +0.23% (effectively unchanged) | Three interleaved process runs per variant and shape, each with five warm-ups, ten timed repetitions and eight generated tokens; graph capture enabled. Each shape's throughput is the median of the three process averages. Prefill spread stayed below 1% except at 256 tokens (1.56% for main, 2.18% for the PR). The 32-token slowdown occurred in all three paired rounds. Both variants use identical model, original GenAI benchmark executable, DXC/DXIL and Agility SDK files. Shader dumps verify the intended wave32/wave64 kernels, zero DP4A prefill fallback, and active graph capture (524 diagnostic replays each). The comparison measures the complete change against main, including shader changes, rather than only a subgroup-size toggle. These are application prefill throughput measurements, not isolated GPU timestamps. `ORT_DISABLE_TELEMETRY=1` was set equally for both variants. Timings exclude diagnostic runs. Windows 11 build 26300, Balanced power plan, RDP session, high-performance adapter preference. The N=64 single-wave specialization is correctness-validated; its latency has not been benchmarked separately. ### Validation Windows x64 Release; AMD Radeon RX 9070 XT; driver `32.0.31007.2064` (`26.10.07.02` package); D3D12; Agility SDK `1.721.3-preview`; Dawn `v20260916.214021`. - Revalidated on 2026-10-16 at `cb5090c9d3` after merging main `f33c1744e1`. Release builds of `onnxruntime`, `onnxruntime_provider_test`, and `onnxruntime_test_all` passed. - The eight targeted GPU tests below passed again, with zero skips. All 80 tile combinations executed wave64 without fallback; generated wave64 shaders match the previously validated version, and all 73 shader keys passed consistency checks. - Three NCHWc regressions passed, including `ConvWithZeroInputChannelsIsNotTransformed`, the zero-output-channel case, and the unexpected-rank case. The merge includes the upstream graph-only regression fix from microsoft#33398; macOS ARM64 confirmation is pending CI. - Eight targeted tests passed with zero skips: `WebGpuMatMulNBitsTest.*`, `SubgroupMatrixConfigTest.*`, `MatMulNBits.Float16_4b_Accuracy4`, `MatMulNBits.Float16_8b_AccuracyLevel4`, `MathOpTest.MatMulSubgroupMatrix`, and `MoETest.QMoETest_WebGPU_WideTileExpertZeroPointsAndFC3`. - Tile tests cover ten shapes for 4-bit and 8-bit inputs, with and without bias and zero points (80 combinations). `N=64` checks single-wave execution, `N=128` validates both column waves, and `N=192` checks partial-N output. - Four-wave `K=192` and `K=96` cases exercise BK reduction from 128 to 64 and 32. `(M,N,K)=(257,64,128)` and `(33,128,32)` cover two-wave BK=128 and wide BK=32 with partial 16-row prepack blocks; `(33,64,32)` checks reduced BK in the single-wave tile. - All 80 combinations executed wave64 subgroup-matrix kernels with zero DP4A fallback. Shader dumps confirm 64-thread single-wave workgroups, 128-thread two-wave workgroups, 256-thread four-wave workgroups, and the expected tile dimensions. For the single-wave BK=64 tile, declared workgroup storage is 9,728 bytes versus 19,456 bytes for the wide tile. - Shader-key consistency passed across 73 distinct keys. - Both A/B shared ORT DLLs were rebuilt. The main baseline also passed four selector and FP16 quantized MatMul correctness tests. - All 66 benchmark process measurements completed successfully; shader dumps and diagnostic logs confirmed the intended kernels and active graph capture. - Lintrunner and `git diff --check` passed. Intel runtime/performance validation is not included in these results.

…ang (microsoft#33408) This change fixes the loader to ensure that it can build with no_exception. ### Description This pull request refactors exception handling in the `D3D12FileBufferLoader` class to use the `ORT_TRY`, `ORT_CATCH`, and `ORT_HANDLE_EXCEPTION` macros for improved consistency and error reporting. It also introduces overloads for the `Create` and `Load` methods that provide default arguments for easier usage. Additionally, minor code cleanups and variable initializations are included. **Exception handling improvements:** * Replaced manual `try`/`catch` blocks with `ORT_TRY`, `ORT_CATCH`, and `ORT_HANDLE_EXCEPTION` macros in `LoadInternal`, `Create`, and `Load` methods to standardize error handling and improve diagnostics. [[1]](diffhunk://#diff-f37e2cabb2669401f283bffae8747139a71f1e260960151e591da38ea33557deL312-R323) [[2]](diffhunk://#diff-f37e2cabb2669401f283bffae8747139a71f1e260960151e591da38ea33557deR1115-R1179) **API usability enhancements:** * Added overloads for `Create` and `Load` methods in `D3D12FileBufferLoader` to allow usage with default configuration and cancellation token, simplifying client code. [[1]](diffhunk://#diff-8978fdd7a77e204adbc45d1e28e98c529bfc09604ab29d65e50690c11306236aR58-R64) [[2]](diffhunk://#diff-8978fdd7a77e204adbc45d1e28e98c529bfc09604ab29d65e50690c11306236aR75-R81) [[3]](diffhunk://#diff-f37e2cabb2669401f283bffae8747139a71f1e260960151e591da38ea33557deR1115-R1179) **Code cleanup and initialization:** * Moved the declaration of `BufferCollection loaded_batch` to ensure proper initialization order and removed redundant declarations. [[1]](diffhunk://#diff-f37e2cabb2669401f283bffae8747139a71f1e260960151e591da38ea33557deR334) [[2]](diffhunk://#diff-f37e2cabb2669401f283bffae8747139a71f1e260960151e591da38ea33557deL355) * Simplified future handling in `LoadInternal` by replacing `get` with `wait`, as the result is not used.

…osoft#33328) ### Description Adds targeted optimizations to CUDA `PackedSparseAttentionIndexer`: - Bound hierarchical scoring work using device-resident causal and state lengths. - Reuse key vectors across heads and verification queries in eligible single-request calls. - Compact candidate lists before subsequent hierarchical merging. - Add regression coverage for ragged batches, empty requests, and multi-query verification across FP32, FP16, and BF16. - Extend the existing device-source probe with QSA parity checks and score/merge timing modes. The changes retain query-specific visibility and stable selection ordering without changing the operator schema or state ABI. ### Motivation + Context Packed QSA can reserve substantially more state capacity than a query can causally access. Scoring unused capacity and repeatedly processing full candidate lists creates unnecessary work. Multi-token verification also lets multiple queries share key loads. Device-side bounds, query-paired reuse, and candidate compaction reduce redundant scoring and merging without host metadata readback. This PR is stacked on [microsoft#33094](microsoft#33094). NVFP4 GEMV changes and their probes belong to that lower PR and are not part of this PR’s diff. --------- Co-authored-by: Akshay Sonawane <asonawane@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Sayan Shaw <52221015+sayanshaw24@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

…files (microsoft#33311) ## Summary When a node is absent from a configured memory-profile CSV, the resource accountant currently returns zero cost even if the execution provider supplies an analytical `Level1MemoryEstimate`. This can bypass a proven workspace charge during capacity-aware placement. - Fall through to existing static initializer/output accounting when an omitted node has a Level-1 estimate. - Charge its Level-1 runtime workspace and persistent prepack memory using the existing non-profile path. - Continue tracking initialization scratch as a session-wide diagnostic peak outside the additive budget. - Preserve the historical zero-cost behavior for profile-missing nodes that have no Level-1 estimate. This is framework behavior and applies to current and future operator-specific estimators, rather than being GQA-specific. ## Validation - Added focused coverage for the preserved no-estimator behavior and the workspace, prepack, initialization-scratch, commit, and source-accounting behavior with an estimator. - `clang-format --dry-run --Werror` passed for both changed files. - `git diff --check` passed. - Local build/test execution was skipped by request; CI will execute the regression. Co-authored-by: Chi Lo <lochi@microsoft.com> Copilot-Session: c132130a-b0c7-422c-9ff7-75826b6ea3fc

### Description  Add experimental C API usage documentation. Minor updates to existing docs. ### Motivation and Context  Documentation. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

### Description  Create the `QLinearSoftmax` output before returning early for an empty input, matching the existing CPU `Softmax` behavior. The nonempty computation is unchanged. Add CPU regression tests for int8/uint8, opsets 12/13, and zero-sized dimensions, with and without shape metadata and with constant or runtime quantization parameters. Tested on Windows x64 Release: - The two new tests fail before the fix and pass afterward. - All 18 lookup-table operator tests pass. - A broader Softmax run passes 52 tests; 7 CUDA/WebGPU tests are skipped in this CPU-only build. ### Motivation and Context  Fixes microsoft#33262. For an empty input such as `[0, 2, 37]`, the kernel returned success without creating an output tensor. A downstream `DequantizeLinear` node then failed with `Missing Input`. Creating the output first preserves its shape and type while still skipping the arithmetic when there are no elements. Co-authored-by: David Luong <luongdavid@microsoft.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3f6d4277-d757-4418-9b0e-91c7c1fab8f6

### Description This patch updates Dawn to use its latest release to enable subgroup matrix on Intel GPUs.

…ft#33117) ### Description Zero initialize the OrtEpFactory base in the CUDA, QNN, MIGraphX, VitisAI, TensorRT RTX, and OpenVINO factories. Previously, unassigned callback pointers were left uninitialized and could pass ORT's null checks, leading to calls through invalid pointers. Set ort_version_supported in CudaEpFactory as well. Document in OrtEp, OrtEpFactory, OrtKernelImpl, and OrtEpProfilerImpl that some callbacks are optional and unimplemented callbacks must be set to NULL.

**Description:** **Summary** Add com.microsoft::GatedDeltaNetStateReplay v1 to replay captured GatedDeltaNet transitions directly on WebGPU. This provides the ORT-side building block for partial-accept state reconstruction without downloading recurrent state to the CPU. The operator supports one request and one descriptor, with FP32 state and capsule backing tensors. CPU metadata supplies element offsets, head geometry, original capture capacity, and the number of leading transitions to keep. **Implementation** - Use one invocation per state element and 128 threads per workgroup to apply the captured recurrence: ``` hk = floor(h * Hk / Hv) s = source[h,v,i] for t = 0 .. kept_count-1: s *= decay[t,h] s += key[t,hk,i] * delta[t,h,v] destination[h,v,i] = s ``` - Use existing ProgramBase , ComputeContext , and BufferView machinery. Logical offsets do not require storage-binding alignment, and binding-limit validation follows actual buffer ownership and segmentation. - Require output prebinding to the inactive destination allocation. Preserve source, capsule, and neighboring destination elements; reject writable destination aliasing with read-only inputs. - Call the existing FlushAndWaitChecked() helper after dispatch so successful execution returns only after checked GPU completion. Keep enqueue and completion separate for future multi-dispatch integration. - Register the schema and kernel, regenerate contrib documentation, and document the original incoming-state and capture-capacity invariants. **Validation** Coverage includes all kept prefixes, nonzero offsets, Qwen’s unaligned capsule-row layout, segmented/shared backing, poisoned unused tails, destination sentinels, and invalid metadata. Integration coverage replays an actual WebGPU producer capsule against independent prefix runs. A separate test checks queue completion immediately after Session::Run , before blocking readback. Native Release validation on Vulkan/lavapipe: 73 tests passed, one expected skip. The completion test also passed ten consecutive repetitions. Numerical comparisons use explicit FP32 tolerances. **Scope** No GenAI changes or new public C API/ABI. Convolution replay, batching, graph capture, and GenAI helper-session integration are out of scope. Hardware-GPU performance and plugin/browser execution are not established by this validation. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…microsoft#33078) ### Description Stacked on microsoft#29588. Builds on the opt-in fpA_intB `MatMulNBits` tactic cache: tactics are now loaded when a session is created and saved when it is closed. This PR also adds a cache mode setting and an opt-in model sidecar location. ### Key Changes **Load on create** - Kernel construction loads matching tactics before profiling (as before). If the cache file changed since this process last read or wrote it (last-write time and size), the file is read again first, so a new session sees tactics written by other sessions or processes. - After a flush, rows merged in from other writers are kept in memory, so a later unchanged-file check does not hide them. **Save on close** - All tuned tactics, from the construction-time sweep and from lazy profiling, are only staged in memory. They are written once, when the session closes (`~CUDAExecutionProvider` / plugin `~CudaEp`). Before this change, the file was rewritten once per unique shape during session creation. - Staging skips buckets whose tactic matches the cached row and overwrites rows that differ (e.g. after a stale row was dropped and re-profiled). - A last-chance flush at provider library unload saves sessions that are never released. It does not log, because the default logger may already be gone. The cache registry is leaked on purpose, so EP teardown that runs after static destruction never touches a destroyed object. **New settings** (session config first, then env var) | Session config | Env var | Values | |---|---|---| | `ep.cuda.gemm_tactic_cache_mode` | `ORT_CUDA_GEMM_TACTIC_CACHE_MODE` | `load_save` (default), `load` (read-only, e.g. a pre-tuned cache shipped with an app), `save` (ignore existing rows and re-tune). Invalid values fail session creation. | | `ep.cuda.gemm_tactic_cache_to_model` | `ORT_CUDA_GEMM_TACTIC_CACHE_TO_MODEL` | `1` stores the cache as `<model_path>.matmulnbits_fpa_intb.tsv`. Ignored for models loaded from bytes. | Location resolution: session prefix → session dir → model sidecar → env prefix → env dir. **Model path plumbing** - The CUDA EP records the model path in `GetCapability`: `GraphViewer::ModelPath()` for the built-in EP and `Graph_GetModelPath` for the plugin EP. The plugin stores it in the kernel adapter runtime config. Kernels read it through `CudaKernel::GetSessionModelPath()`. **Tooling** - `fpa_intb_tune.py --retune` uses `save` mode to profile every bucket again. ### Design notes - Session close writes every dirty cache in the process, not only the closing session's caches. The write is dirty-guarded and merges with the file, so writing another live session's rows early is harmless. Restricting it would need per-session tracking of resolved cache paths. - Persistence stays opt-in. No location is used unless a dir, prefix, or sidecar is enabled. ### Testing - `gemm_tactic_cache_test.cc` (19 tests, all passing in a standalone gtest link): adds coverage for mode and flag parsing, `SameTactic`, sidecar path resolution, `ReloadIfChanged` picking up other writers, and flush adopting other writers' rows. - Compiled `matmul_nbits.cc`, `fpA_intB_gemm_profiler.cc`, `gemm_profiler.cc`, `gemm_tactic_cache.cc`, `cuda_execution_provider.cc`, and a generic CUDA kernel (compact fpA_intB, CUDA 13.0). Syntax-checked plugin `cuda_ep.cc` and `matmul_nbits.cc` with `USE_FPA_INTB_GEMM=1`. - Not yet run: an end-to-end GPU check (create session → close → second session reuses the cache with no profiling; `load` mode leaves the file unchanged; sidecar file appears next to the model). Suggested command: `python -m onnxruntime.tools.fpa_intb_tune --model <model> --output-prefix <prefix>`, run twice. The second run should report the same tuned shapes, and the file timestamp should not change in `load` mode. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Extend the existing WebGPU FP16 MatMulGemv fast path from M=1 to 1 <= M <= 8, preserving its existing shape and structural constraints. Each row uses independent workgroups with the existing FP32 accumulation and reduction. The shader derives row and column-group indices from the flattened workgroup index and applies row offsets to input/output accesses. ## Motivation Small-M projections occur during speculative decoding. Previously, moving from one row to two rows selected generic MatMul, introducing a substantial performance cliff for skinny FP16 projections such as Qwen's recurrent gates: [M, 5120] x [5120, 48]. Isolated A100 / WebGPU / Vulkan measurements using Dawn GPU timestamps show: | M | Generic MatMul | Extended GEMV | Speedup | |---|---:|---:|---:| | 2 | 166 us | 30 us | 5.53x | | 4 | 167 us | 30 us | 5.57x | | 8 | 170 us | 33 us | 5.15x | These are median kernel timings from one warmup and three measured runs, not end-to-end generation results. ## Scope The existing eligibility conditions remain unchanged except for M: - FP16 inputs and output; single batch after existing batch folding. - 16 <= N <= 64, with N divisible by 4. - 2048 <= K <= 8192. - No bias or fused activation. Existing optimized-path precedence is preserved. No device-specific heuristic, MatMulNBits selection change, profiling infrastructure, or benchmark-only option is included. Regression coverage includes every M from 1 through 8, K/N boundaries, non-aligned K, folded rows, and fallback shapes outside the eligibility bounds. ## Validation of the original cleaned implementation - Rebuilt the production WebGPU runtime, Python binding, and provider tests. - 19 existing/new native tests passed; four existing unsupported cases skipped. - 25 GPU-only routing/reference checks passed: eligible M=2–8 cases selected GEMV, while excluded shapes retained their previous path. - M=1 matrix and vector outputs were bit-identical to the original runtime. - Formatting and git diff --check passed. GEMV retains the existing FP32 reduction. Newly eligible shapes can differ numerically from generic FP16 MatMul; bitwise equivalence is not claimed for M=2–8. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

…oft#33291) ### Description Native inference can finish successfully, write its profiling output, and then hang indefinitely while Windows terminates the process. Skip 1DS uploader teardown in `OneDsTelemetry::~OneDsTelemetry` when Windows DLL shutdown is in progress, before acquiring telemetry locks. The Windows helper resolves `RtlDllShutdownInProgress` from the already loaded `ntdll.dll`. This process-wide check also works for telemetry compiled into an EP plugin DLL. Normal object destruction and explicit `FreeLibrary` retain the existing cleanup path; non-Windows behavior is unchanged. Pending telemetry is not flushed during process termination. ### Motivation and Context Two captured native benchmark hangs had only the terminating thread remaining, with this shutdown stack: ```text ExitProcess -> LdrShutdownProcess -> WindowsEnv static destructor -> OneDsTelemetry::~OneDsTelemetry -> LogManagerImpl::FlushAndTeardown -> HttpClientManager::cancelAllRequests / cancelAllRequestsAsync -> HttpClient_WinHttp::CancelAllRequests -> condition-variable wait ``` Windows terminates the other threads before calling DLL process-detach destructors. The uploader can therefore wait forever for WinHTTP callbacks that can no longer run; a telemetry lock may also be held by a terminated thread. Checking shutdown state before taking locks avoids both waits. Resolved ORT/SDK frames were verified against identical machine code in a build with symbols. ### Regression Coverage Three committed Windows CTest cases load a fixture DLL linked to the production `onnxruntime_common` telemetry implementation. A test peer injects a mock log manager with network telemetry disabled; no model, GPU, or symbol-address patching is required. - `onnxruntime_telemetry_process_exit_test`: calls `ExitProcess` with a pending mock callback and verifies that the real DLL-owned telemetry destructor skips uploader cleanup. - `onnxruntime_telemetry_normal_destruction_test`: destroys the object before DLL unload and verifies one `Flush`, one `FlushAndTeardown`, and cleared shared state. - `onnxruntime_telemetry_explicit_unload_test`: verifies the same cleanup during `FreeLibrary`. The parent requires both a successful child exit and an inherited event signaled only after teardown assertions pass. Each child has a 10-second timeout, with a 15-second CTest limit. The mock drain itself is bounded to two seconds so a regressed shutdown cannot strand a child inside DLL teardown. Run from the CMake build directory after building `onnxruntime_telemetry_shutdown_test`: ```text ctest -C Release -R "^onnxruntime_telemetry_(process_exit|normal_destruction|explicit_unload)_test$" --output-on-failure ``` ### Validation - [x] Windows x64/MSVC Release: configured a separate build and compiled the committed test target and its production dependencies; all three lifecycle tests passed. - [x] Local mutation checks: removing the shutdown guard failed only the process-exit case; returning unconditionally from the destructor failed both normal destruction and explicit unload. Restoring the fix passed all three cases again. - [x] Windows/MSVC runtime validation: recompiled the changed translation units and relinked isolated built-in WebGPU ORT, plugin ORT host, and WebGPU plugin binaries using compatible cached build objects. - [x] Original local lifecycle harness with network telemetry suppressed and a mock pending uploader injected into the actual ORT DLL: baseline entered SDK draining and timed out; fixed runtime exited normally in **20/20** trials. Explicit DLL unload with an inactive uploader passed **3/3** trials for both baseline and fixed runtimes. - [x] Native GenAI benchmark: **built-in/plugin WebGPU × R32 FP16 LoRA on/off**, 1,024 prompt tokens and 32 output tokens, batch 1, one warmup and two measured runs. All four configurations exited with code 0; generated text matched across providers for each adapter state. An additional built-in/LoRA-on run matched the original runtime's output and exited normally. - [x] Changed-file `lintrunner -a` and `git diff --check` passed. The committed lifecycle tests use a mock uploader; they do not cover real pending-upload unload stress. The benchmark executable, GenAI DLL, model graph, configuration, and adapter were unchanged during runtime validation.

chilo-ms and others added 20 commits October 15, 2026 13:42

[WebGPU] Upgrade Dawn to the latest version (microsoft#33390)

0767b73

### Description This patch updates Dawn to use its latest release to enable subgroup matrix on Intel GPUs.

Merge remote-tracking branch 'origin/master' into sync_msft_18102026

5c8bb04

ai-fw-intg requested review from Jaswanth51, ankitm3k, jatinwadhwa921 and vthaniel October 17, 2026 20:39

hdharpure9922 closed this Oct 22, 2026

hdharpure9922 deleted the sync_msft_18102026 branch October 22, 2026 04:45

Navigation Menu

Repository navigation

Sync with Microsoft ONNX Runtime - 18102026 - #1346

Sync with Microsoft ONNX Runtime - 18102026#1346
ai-fw-intg wants to merge 20 commits into
ovep-developfrom
sync_msft_18102026

ai-fw-intg commented Oct 17, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

Conversation

ai-fw-intg commented Oct 17, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants