Skip to content

MIGraphX catalog EP requests API 27 but ORT 1.24 only supports API 24 #1283

Description

@KayMKM

Summary

MIGraphX GPU evaluation is completely broken because the current catalog-provided amdgpu-ep.dll requests ONNX Runtime provider API 27, while winml-cli's supported ONNX Runtime 1.24.x runtime only exposes provider API 24.

The failure reproduces before model loading, affects multiple models, and occurs with both the last-known-good winml-cli commit and the current release/v0.3.0 commit. This indicates an external MIGraphX catalog package regression rather than a Python code regression in release 0.3.0.

Error

winml sys --list-ep reports:

MIGraphXExecutionProvider  [incompatible]
  [failed] Catalog   (catalog default)
            Version: 7.2.2608.30
            Path:    ...\MicrosoftCorporationII.WinML.AMD.GPU.EP.1.8_1.8.60.0_x64__8wekyb3d8bbwe\ExecutionProvider\amdgpu-ep.dll
            Error: The requested API version [27] is not available, only API versions [1, 24] are supported in this build. Current ORT Version is: 1.24.5

Without verbose diagnostics, winml perf only prints the generic warning below and exits with code 1:

[W:onnxruntime:Default, onnxruntime_pybind_module.cc:44 onnxruntime::python::CreateOrtEnv] Init provider bridge failed.

The E2E harness consequently reports [FAIL (UNKNOWN)].

Regression Evidence

Last successful pipeline run:

  • Run: 20260802.2 / build 153738978
  • Source: b470c218e6dffdcb10b34b67a50035de79224aa7
  • Agent: DESKTOP-36UR64G
  • facebook/convnext-tiny-224, image-classification, fp16, MIGraphX GPU: passed in 61.6 seconds

Current failing pipeline run:

  • Run: 20260804.2 / build 153962701
  • Source: 4df4a043353167e38f17c83255af3fb8e31e9b19
  • Same agent: DESKTOP-36UR64G
  • The same ConvNeXT MIGraphX task failed twice, in 108.5 seconds and 19.3 seconds
  • openai/clip-vit-base-patch32 also fails on MIGraphX, while VitisAI and TensorRT-RTX tasks pass

Both successful and failing jobs installed the same runtime dependency:

onnxruntime-windowsml==1.24.6.202605042033

The two source commits after the last successful run only change the package version/build tooling and local-path telemetry handling; neither changes runtime dependencies or MIGraphX registration.

On the same self-hosted agent, the currently installed package is:

AppX package:  MicrosoftCorporationII.WinML.AMD.GPU.EP.1.8 1.8.60.0
EP DLL:        amdgpu-ep.dll
File version:  7.2.2608.30
Last write:    2026-08-04 18:01:58 local time

The DLL write time falls strictly between the successful and failing runs.

As an A/B check, both the last successful source commit (b470c218) and current release source commit (4df4a043) now report the same MIGraphX incompatibility when run against the current machine environment.

Steps to Reproduce

On the affected AMD agent:

uv run winml sys --list-device --list-ep

For the model-level failure with the underlying native diagnostic visible:

$env:WINMLCLI_SHOW_ALL_WARNINGS = "1"
uv run winml -vv perf `
  -m facebook/convnext-tiny-224 `
  --task image-classification `
  --device gpu `
  --ep migraphx `
  --precision fp16 `
  --iterations 1 `
  --warmup 0

The command exits with code 1 after reporting the API 27/24 mismatch.

Expected Behavior

The catalog-provided MIGraphX EP should register against the supported ONNX Runtime 1.24.x runtime and execute GPU workloads.

If the EP is incompatible, winml perf and the E2E harness should surface a clear environment/provider compatibility error instead of [FAIL (UNKNOWN)].

Proposed Resolution

  1. EP package fix (required): roll back or republish amdgpu-ep.dll built against ONNX Runtime provider API 24. Updating winml-cli to an API 27 runtime is not a suitable release hotfix because the project intentionally constrains onnxruntime-windowsml to >=1.24.5,<1.25 and the rest of the Windows ML stack is aligned to 1.24.x.
  2. Diagnostic improvement: preflight EP compatibility before model work and preserve the structured registration error in winml perf output.
  3. E2E classification: classify an incompatible registered EP as ENVIRONMENT rather than UNKNOWN.

Related Issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    GPUGPU specificP1High — major feature broken or significant UX impactbugSomething isn't workingdependenciesPull requests that update a dependency fileneed triageNeeds triage

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions