Sync with Microsoft ONNX Runtime - 31102026

.github/workflows/windows-web-ci-workflow.yml

@@ -158,7 +158,6 @@ jobs:
 
       # WebGPU EP tests
       - name: Run ort-web tests - WebGPU EP
-        continue-on-error: true
         shell: cmd
         run: |
           mkdir ${{ runner.temp }}\web\test\07

.github/workflows/windows_webgpu.yml

@@ -161,7 +161,6 @@ jobs:
         run: python tools\ValidateNativeDelegateAttributes.py
         shell: cmd
         working-directory: ${{ github.workspace }}\csharp
-        continue-on-error: true
 
   webgpu_test_x64_RelWithDebInfo:
     name: webgpu_test_x64_RelWithDebInfo (${{ matrix.vcpkg_option }})

cmake/external/cuda_configuration.cmake

@@ -121,16 +121,19 @@ macro(setup_cuda_architectures)
       message(WARNING "Detecting native CUDA compute capability - fail")
       message(WARNING "CUDA compute capability detection failed, compiling for all optimized architectures")
       unset(CMAKE_CUDA_ARCHITECTURES)
+      unset(CMAKE_CUDA_ARCHITECTURES CACHE)
     else()
       message(STATUS "Detecting native CUDA compute capability - done")
       set(CMAKE_CUDA_ARCHITECTURES "${CUDA_ARCH_OUTPUT}")
     endif()
   elseif(CMAKE_CUDA_ARCHITECTURES STREQUAL "all")
     unset(CMAKE_CUDA_ARCHITECTURES)
+    unset(CMAKE_CUDA_ARCHITECTURES CACHE)
     message(STATUS "Setting CMAKE_CUDA_ARCHITECTURES to all enables a list of architectures OnnxRuntime optimized for, "
                    "not all architectures CUDA compiler supports.")
   elseif(CMAKE_CUDA_ARCHITECTURES STREQUAL "all-major")
     unset(CMAKE_CUDA_ARCHITECTURES)
+    unset(CMAKE_CUDA_ARCHITECTURES CACHE)
     message(
       STATUS "Setting CMAKE_CUDA_ARCHITECTURES to all-major enables a list of architectures OnnxRuntime optimized for, "
              "not all major architectures CUDA compiler supports.")

cmake/onnxruntime_providers_cuda_plugin.cmake

@@ -486,6 +486,12 @@ endif()
 message(STATUS "CUDA Plugin EP: cuDNN include: ${CUDA_PLUGIN_CUDNN_INCLUDE_DIR}")
 message(STATUS "CUDA Plugin EP: cuDNN runtime library: ${CUDA_PLUGIN_CUDNN_LIBRARY}")
 
+# With cuDNN in /usr drop CUDA_PLUGIN_CUDNN_INCLUDE_DIR; we do not want
+# "-isystem /usr/include" ahead of libstdc++, breaking #include_next mechanism.
+if(CUDA_PLUGIN_CUDNN_INCLUDE_DIR IN_LIST CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES)
+  set(CUDA_PLUGIN_CUDNN_INCLUDE_DIR "")
+endif()
+
 # Include directories — only public ORT headers + CUDA toolkit + cuDNN + internal headers for adapter
 target_include_directories(onnxruntime_providers_cuda_plugin PRIVATE
     ${REPO_ROOT}/include

docs/ContribOperators.md

@@ -133,6 +133,7 @@ Do not modify directly.*
   * <a href="#com.microsoft.SparsePagedAttention">com.microsoft.SparsePagedAttention</a>
   * <a href="#com.microsoft.SparseToDenseMatMul">com.microsoft.SparseToDenseMatMul</a>
   * <a href="#com.microsoft.SpeculativePrefixCommit">com.microsoft.SpeculativePrefixCommit</a>
+  * <a href="#com.microsoft.SwiGLU">com.microsoft.SwiGLU</a>
   * <a href="#com.microsoft.Tokenizer">com.microsoft.Tokenizer</a>
   * <a href="#com.microsoft.TorchEmbedding">com.microsoft.TorchEmbedding</a>
   * <a href="#com.microsoft.TransposeMatMul">com.microsoft.TransposeMatMul</a>
@@ -8360,6 +8361,62 @@ This version of the operator has been available since version 1 of the 'com.micr
 </dl>
 
 
+### <a name="com.microsoft.SwiGLU"></a><a name="com.microsoft.swiglu">**com.microsoft.SwiGLU**</a>
+
+  Clamped SwiGLU, the gated activation of a feed-forward or MoE expert block:
+    G = clamp(gate, max=limit)
+    L = clamp(up, min=-limit, max=limit)
+    output = G * Sigmoid(activation_alpha * G) * (L + activation_beta)
+  with the arithmetic done in float regardless of T, widened to double for the product when `activation_alpha * G` is below -87 so that a negative tail is not lost to float exp underflow. A `limit` of zero or less disables both clamps.
+
+  The result is rounded to T once, after the multiply by (L + activation_beta). With `round_activation` set, G * Sigmoid(activation_alpha * G) is first rounded to T, as a QuickGelu node followed by a Mul in T would, which keeps a fused QuickGelu + Mul bit-identical to the graph it replaces.
+
+  This uses the same activation_alpha / activation_beta formula and positive-limit clamping rule as MoE and QMoE. The no-clamp sentinel differs: here any `limit` of zero or less disables clamping, while MoE/QMoE use infinity when `swiglu_limit` is omitted and clamp finite values, including an explicit zero.
+  With `up` supplied and the default `limit` and `activation_beta`, it equals the ONNX SwiGLU operator (opset 28) with `alpha` = `activation_alpha`.
+
+  `up` is optional. When it is omitted, `gate` carries both halves of one `[.., 2 * inter]` projection -- gate first, then up -- and the split is done internally, so a fused sibling GEMM needs no `Split` node.
+
+#### Version
+
+This version of the operator has been available since version 1 of the 'com.microsoft' operator set.
+
+#### Attributes
+
+<dl>
+<dt><tt>activation_alpha</tt> : float</dt>
+<dd>Multiplier inside the sigmoid.</dd>
+<dt><tt>activation_beta</tt> : float</dt>
+<dd>Value added to the linear half. A zero value leaves it unchanged, including a negative zero.</dd>
+<dt><tt>limit</tt> : float</dt>
+<dd>Clamp applied to both halves; zero or less disables it.</dd>
+<dt><tt>round_activation</tt> : int</dt>
+<dd>If non-zero, round G * Sigmoid(activation_alpha * G) to T before multiplying by (L + activation_beta). Zero rounds only the final result.</dd>
+</dl>
+
+#### Inputs (1 - 2)
+
+<dl>
+<dt><tt>gate</tt> : T</dt>
+<dd>Gate half of the projection, or both halves concatenated.</dd>
+<dt><tt>up</tt> (optional) : T</dt>
+<dd>Up half of the projection, same shape as gate.</dd>
+</dl>
+
+#### Outputs
+
+<dl>
+<dt><tt>output</tt> : T</dt>
+<dd>Activated tensor, shaped like one half of the projection.</dd>
+</dl>
+
+#### Type Constraints
+
+<dl>
+<dt><tt>T</tt> : tensor(float16), tensor(bfloat16), tensor(float)</dt>
+<dd>Constrain the activation type to float tensors.</dd>
+</dl>
+
+
 ### <a name="com.microsoft.Tokenizer"></a><a name="com.microsoft.tokenizer">**com.microsoft.Tokenizer**</a>
 
   Tokenizer divides each string in X into a vector of strings along the last axis. Allowed input shapes are [C] and [N, C].

docs/OperatorKernels.md

@@ -703,6 +703,9 @@ The **OpSet Version** column uses the following notation:
 |||14|**T** = tensor(double), tensor(float), tensor(float16)<br/> **U** = tensor(double), tensor(float), tensor(float16)|
 |||[9, 13]|**T** = tensor(double), tensor(float), tensor(float16)|
 |||[7, 8]|**T** = tensor(double), tensor(float), tensor(float16)|
+|BitShift|*in* X:**T**<br> *in* Y:**T**<br> *out* Z:**T**|28+|**T** = tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
+|||[11, 27]|**T** = tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
+|BitwiseXor|*in* A:**T**<br> *in* B:**T**<br> *out* C:**T**|18+|**T** = tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
 |Cast|*in* input:**T1**<br> *out* output:**T2**|28+|**T1** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(float4e2m1), tensor(float8e4m3fn), tensor(float8e5m2), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)<br/> **T2** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(float4e2m1), tensor(float8e4m3fn), tensor(float8e5m2), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
 |||[25, 27]|**T1** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(float4e2m1), tensor(float8e4m3fn), tensor(float8e5m2), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)<br/> **T2** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(float4e2m1), tensor(float8e4m3fn), tensor(float8e5m2), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
 |||[23, 24]|**T1** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(float4e2m1), tensor(float8e4m3fn), tensor(float8e5m2), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)<br/> **T2** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(float4e2m1), tensor(float8e4m3fn), tensor(float8e5m2), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
@@ -1179,6 +1182,7 @@ The **OpSet Version** column uses the following notation:
 |SparseAttentionIndexer|*in* query:**T**<br> *in* key:**T**<br> *in* query_norm_weight:**T**<br> *in* key_norm_weight:**T**<br> *in* cos_cache:**T**<br> *in* sin_cache:**T**<br> *in* mask:**TB**<br> *in* past_key:**T**<br> *in* gate:**T**<br> *in* position_bias:**T**<br> *in* head_weights:**T**<br> *in* position_ids:**I**<br> *in* past_sequence_length:**M**<br> *in* past_proj_buffer:**T**<br> *out* selected_indices:**M**<br> *out* present_key:**T**<br> *out* present_proj_buffer:**T**|1+|**I** = tensor(int64)<br/> **M** = tensor(int32)<br/> **T** = tensor(bfloat16), tensor(float), tensor(float16)<br/> **TB** = tensor(int64)|
 |SparsePagedAttention|*in* query:**T**<br> *in* key:**T**<br> *in* value:**T**<br> *in* key_cache:**T_CACHE**<br> *in* value_cache:**T_CACHE**<br> *in* cumulative_sequence_length:**S**<br> *in* past_seqlens:**S**<br> *in* block_table:**S**<br> *in* slot_mapping:**S**<br> *in* selected_indices:**S**<br> *in* selected_counts:**S**<br> *in* auxiliary_key:**T**<br> *in* auxiliary_value:**T**<br> *in* auxiliary_lengths:**S**<br> *in* cos_cache:**T**<br> *in* sin_cache:**T**<br> *in* head_sink:**T**<br> *in* q_norm_weight:**T**<br> *in* k_norm_weight:**T**<br> *in* k_scale:**T_KV_SCALE**<br> *in* v_scale:**T_KV_SCALE**<br> *in* attention_metadata:**S**<br> *out* output:**T**<br> *out* key_cache_out:**T_CACHE**<br> *out* value_cache_out:**T_CACHE**|1+|**S** = tensor(int32)<br/> **T** = tensor(bfloat16), tensor(float16)<br/> **T_CACHE** = tensor(bfloat16), tensor(float16), tensor(int8)<br/> **T_KV_SCALE** = tensor(float)|
 |SpeculativePrefixCommit|*in* next_tokens:**I64**<br> *in* input_ids:**I64**<br> *in* status:**I32**<br> *in* ctl:**I64**<br> *in* ctx_commit:**I64**<br> *in* prefixes:**I64**<br> *in* ptrs:**I64**<br> *out* head_c:**I32**<br> *out* updated_ctl:**I64**<br> *out* updated_ctx_commit:**I64**|1+|**I32** = tensor(int32)<br/> **I64** = tensor(int64)|
+|SwiGLU|*in* gate:**T**<br> *in* up:**T**<br> *out* output:**T**|1+|**T** = tensor(bfloat16), tensor(float), tensor(float16)|
 |TransposeMatMul|*in* A:**T**<br> *in* B:**T**<br> *out* Y:**T**|1+|**T** = tensor(bfloat16), tensor(double), tensor(float), tensor(float16)|
 |Trilu|*in* X:**T**<br> *in* k:**tensor(int64)**<br> *out* Y:**T**|1+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
 |UnfoldTensor|*in* input:**T**<br> *out* output:**T**|1+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|

docs/Pattern_Rewriter.md

@@ -1245,6 +1245,75 @@ before entering the transformer or after an error. Callers must not concurrently
 graph. Options and compiled rules are immutable; match state and statistics are
 per call.
 
+### Restricting rewriting to a selected region
+
+The scoped overload accepts an explicit set of live node IDs:
+
+```cpp
+InlinedHashSet<NodeIndex> region(selected_nodes.begin(), selected_nodes.end());
+ORT_RETURN_IF_ERROR(rewriter.Run(graph, statistics, region));
+ORT_RETURN_IF_ERROR(graph.Resolve());
+```
+
+Only roots in the region are visited. Every node consumed by a match must also
+belong to the region; a match crossing its boundary is rejected before either
+rule callback runs. Inputs may come from outside the region, and returned values
+may feed outside consumers. Their `NodeArg` identities are preserved, although
+boundary edges are rewired to new producers. Existing deletion-boundary and EP
+checks still apply. Region-scope and deletion-boundary rejections both report
+`FailureStage::Boundary`; the diagnostic reason distinguishes them.
+
+Each successful rewrite removes its source IDs from `region` and adds all of its
+replacement IDs. New nodes participate in the same ordered, iterative matching;
+unmatched region nodes remain in the returned set. This lets a capability
+callback reconcile its node list without rescanning unrelated nodes. The
+overload neither selects an EP nor performs kernel lookup, placement, or resource
+accounting. Callers must still validate those contracts for the resulting region.
+
+An empty region does no work. Invalid input IDs return an error before matching
+or mutation. Match-only mode leaves the set unchanged. Failures do not roll back
+earlier rewrites: the set reflects completed replacements, not a transaction.
+An ordinary staging failure leaves that match's source IDs in the set and does
+not add its staged replacement IDs.
+As with unscoped `Run`, resolve the graph explicitly before later consumers.
+
+### Function expansion and capability application
+
+The rewriter does not decide when ORT expands functions or assigns nodes to an
+EP. A pattern can match a function-call node, but this does not inspect or
+rewrite its model-local function definition.
+
+Capability discovery and rewrite application are separate operations. In
+ordinary ONNX initialization, AOT function inlining can query `GetCapability`
+before normal partitioning. Reporting a capability can preserve a function
+call for later partitioning; it is not permission to run its optimization
+callback during that query. Discovery must be read-only and repeatable.
+Normal partitioning checks availability before invoking the selected
+capability's optimization callback and placing its updated node set.
+
+`PatternCapabilityTests.PreservesFunctionUntilAcceptedRewrite` and
+`PatternCapabilityTests.ExpandsUnclaimedFunctionAndRediscoversBody` exercise
+these paths with a test-only CPU EP and existing CPU kernels:
+
+| Function-call candidate | Expected integration behavior |
+|---|---|
+| Recognized with an executable rewrite | Discovery leaves the call intact. After acceptance, scoped rewriting replaces the call and updates the parent capability before placement. |
+| Not recognized | Existing function expansion exposes the body. A later capability query discovers its Transpose chain and returns a fresh region for rewriting. |
+
+Both tests run with AOT inlining enabled and disabled. With AOT disabled, the
+unclaimed call expands after a partitioning pass, and partitioning repeats.
+No borrowed matches or capabilities from the function-call graph are retained
+for matching the expanded body. Assertions cover the original call's removal,
+callback and rewrite counts, final node types and CPU assignment, output names,
+and numerical equality with ordinary CPU execution.
+
+This is a fixed-function integration probe, not general function-equivalence
+checking or a production registration interface. Recognizing only a function
+name is insufficient for arbitrary model-local definitions. The probe does not
+change expansion policy or cross-EP priority, compare whole-function and
+expanded-body placement alternatives, or establish ORT-format export,
+resource-accounting, or general N-to-M placement contracts.
+
 ### Resolved metadata and mutation boundaries
 
 Call `Graph::Resolve` after constructing a source graph and before `Run` or
@@ -1290,9 +1359,10 @@ remain unsupported; identity elimination is a separate transformation.
 
 `Options::max_iterations` is a positive upper bound, defaulting to 10.
 Each iteration processes a fixed candidate batch, initially all live nodes in
-topological order. Replacements and affected existing nodes are queued for the
-next batch, never appended to the current batch. An existing later candidate
-can still match an earlier replacement in the same batch. Processing stops
+topological order. For scoped `Run`, the initial batch contains only region nodes.
+Replacements and affected existing nodes are queued for the next batch, never
+appended to the current batch. An existing later candidate can still match an
+earlier replacement in the same batch. Processing stops
 when a batch performs no rewrites, there are no pending candidates, or the bound
 is reached. Match-only mode processes exactly one batch without mutation.