feat(core): resolve model modalities from API metadata - #8529
Open
DragonnZhang wants to merge 12 commits into
Open
feat(core): resolve model modalities from API metadata#8529DragonnZhang wants to merge 12 commits into
DragonnZhang wants to merge 12 commits into
Conversation
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
Collaborator
Author
E2E test report
Environment: macOS, Node.js 22, no sandbox. Windows and Linux runtime behavior were not manually tested. |
Collaborator
🩺 serve daemon A/BBuilt the PR base vs this PR head ✅ No response changes against the PR base across 4 scenario(s). — Qwen Code · serve A/B |
DragonnZhang
marked this pull request as ready for review
August 5, 2026 07:15
DragonnZhang
requested review from
LaZzyMan,
doudouOUC,
tanzhenxin,
wenshao and
yiliang114
as code owners
August 5, 2026 07:15
DragonnZhang
enabled auto-merge
August 5, 2026 07:15
Collaborator
Collaborator
|
Thanks for the PR! Template looks good ✓ — all sections present, bilingual body, and a detailed reviewer test plan. Problem: observed, not theoretical. There is a concrete before/after: on 0.20.1 a JPEG attached to Direction: aligned. #8558 explicitly sketches the staged approach this PR takes — modalities first via a models.dev snapshot, with limits/reasoning/pricing as later steps — and even names this PR as the demonstration of that approach. The issue also requires the cold-start guarantees implemented here (bundled snapshot, non-blocking startup, disk cache + background refresh). Claude Code's CHANGELOG has no direct modality-catalog reference, but the area is clearly relevant and the linked roadmap issue is the stronger signal. Size: core paths are touched ( Approach: scope feels right for the first step. It deliberately reads modalities only (no limits, pricing, reasoning flags, or model-list migration — boundaries stated in the design doc), and the precedence chain (explicit config → exact catalog match → name heuristic → text-only) matches what #8558 asks for. The largest complexity chunk is provider-identity resolution (regional Alibaba endpoints, gateway model-family borrowing, OpenRouter variant suffixes, Idealab aliases) — that's more machinery than a naive "exact provider/model lookup", but each case maps to a real configuration surface in this repo's presets, so it earns its place. One genuine question for the maintainer: is landing all of the alias machinery together with the preset-template modality removals the preferred slice, or would you rather have the exact-match core first? Not a blocker — the tests cover every one of those paths. Risk: no elevated risk signals — none of the changed files match the repo's revert-correlated high-risk paths. The PR does touch a sensitive area (model configuration and Moving on to code review. 🔍 中文说明感谢贡献！ 模板完整 ✓ —— 各部分齐全、中英双语、审阅者测试计划详细。 问题：已观测到，非理论性问题。 有具体的 before/after：0.20.1 中附加到 方向：对齐。 #8558 明确规划了本 PR 采用的分阶段方案——先经 models.dev 快照解决模态，limits/reasoning/价格作为后续步骤——并点名本 PR 作为该方案的示范。issue 还要求了这里已实现的冷启动保障（内置快照、启动不阻塞、磁盘缓存 + 后台刷新）。Claude Code 的 CHANGELOG 没有直接相关的目录化模态条目，但该领域明显相关，关联的路线图 issue 是更强的信号。 规模： 触及核心路径（ 方案：作为第一步范围合理。 刻意只读取模态（不含 limits、价格、reasoning 标志和远端模型列表迁移——设计文档已声明边界），优先级链（显式配置 → 精确目录命中 → 名称启发式 → 纯文本）与 #8558 要求一致。复杂度最高的一块是 provider 身份解析（Alibaba 区域 endpoint、网关模型家族借用、OpenRouter 变体后缀、Idealab 别名）——比朴素的"精确 provider/model 查找"机制更多，但每个 case 都对应本仓库 preset 中真实存在的配置面，因此是值得的。留给维护者的一个真诚问题：alias 机制与 preset 模板模态移除一起落地，是否是希望的切分方式，还是更愿意先合入精确匹配核心？不作为阻断项——这些路径都有测试覆盖。 风险： 无升级风险信号——改动文件均未命中本仓库与 revert 相关的高风险路径。但 PR 触及敏感区域（模型配置与 进入代码审查 🔍 — Qwen Code · qwen3.8-max Reviewed at |
Collaborator
Code reviewI read the full diff against the base and verified every integration point (imports, the async-ification of the workspace-status provider, JSON import support in the root tsconfig, the call sites in
Three non-blocking nits:
The new runtime flow, for orientation: sequenceDiagram
participant P1 as CLI config load
participant P2 as Catalog loader
participant P3 as Disk cache
participant P4 as models.dev
participant P5 as ModelsConfig and registry
P1->>P2: load catalog
P2->>P3: read cache
P3-->>P2: valid cache used immediately, even when stale
P2-->>P1: catalog resolved without waiting for network
P1->>P5: config captures the catalog reference
P2-)P4: background refresh, also when cache is missing (built-in snapshot covers it)
P4-->>P2: new catalog, strictly validated
P2->>P3: atomic cache write
Note over P2: refreshed data applies to configs created later - existing sessions keep theirs
Files changed (26)
Test evidenceThis is an unattended CI run — per the gate rules no PR code was built or executed here. The evidence below is quoted from the PR's own CI on the reviewed commit:
All three The author reports 286 passing focused tests plus a manual attachment E2E against a local OpenAI-compatible endpoint on macOS — that is the author's claim, not independently re-run here (and Windows/Linux are untested per the PR's own table). Sandboxed verification would settle the remaining behavioural gap: 中文说明代码审查通读了完整 diff 并对照 base 验证了所有集成点（imports、workspace-status provider 的异步化、根 tsconfig 的 JSON import 支持、serve 调用点）。未发现关键阻断问题。 实现在网络元数据源需要谨慎的地方都很到位：加载器卫生（发布/缓存前严格结构校验、流式 8 MB 上限、10 秒超时、原子写缓存、unref 的每小时刷新定时器、所有失败路径非致命）；冷启动保障由结构保证（从磁盘缓存或内置快照解析，网络刷新 fire-and-forget，且有永不完成的冷缓存请求测试钉住）；会话稳定性（每个 ModelsConfig 构造时捕获 catalog 引用，后台刷新只影响之后创建的配置）；优先级真实强制（显式值永远优先，WeakMap 来源追踪 + hasExplicitModalities；MiniMax-M3 强制归一化改为尊重用户值——有意为之且有测试）；遵守 #8558 硬性要求（只读模态，不向请求体/请求头/路由注入任何目录数据）。 三个非阻断小问题：modelConfigResolver.ts 的 hunk 顺手删了一条已有注释并格式化了一个本就合规的 if（项目约定保留已有注释）；Vitest 环境检测三元条件在两个文件里重复；loadModelMetadataCatalog 按缓存路径维护状态，首个调用方的 proxy 选项对后续调用方生效（当前两个调用点来源一致，暂无害，新增调用点时需注意）。 测试证据无人值守 CI 运行——按门禁规则未构建或执行任何 PR 代码。以上证据引自该 commit 的 PR 自身 CI：ubuntu 单元套件、Serve A/B（4 场景无响应变化）、Desktop Shell 均通过；macOS/Windows 单元与 CLI 集成按工作流设计跳过；web-shell E2E Smoke 审查时仍在进行。三个 pull_request 事件工作流全部绿色完成。作者自述 286 个聚焦测试通过并在 macOS 上做了本地 endpoint 附件 E2E——这是作者声明，未在此独立复跑；Windows/Linux 按其自述未测试。 沙箱验证可以补上剩余的行为缺口： — Qwen Code · qwen3.8-max Reviewed at |
Collaborator
|
Confidence: 3/5 — clean review on the merits, but this is a 500+-production-line feature in core, and policy puts that decision in a maintainer's hands rather than an auto-approve. Stepping back: this PR is exactly the first step #8558 carved out — modalities only, staged before limits/reasoning/pricing — and the issue even names this PR as the demonstration of the approach. My independent baseline for this problem would have been smaller (exact provider/model lookup only), but the extra machinery here — regional Alibaba endpoints, gateway family borrowing, OpenRouter variant suffixes, Idealab aliases — maps one-to-one onto configuration surfaces that already exist in this repo's presets, and every one of those paths has a test. So the scope reads as earned, not padded. The parts I weighed hardest:
What I'm not doing: approving. Not because I found doubt in the code — I didn't — but because a ⏸️ Deferring to @wenshao @tanzhenxin @yiliang114 @LaZzyMan @doudouOUC (core CODEOWNERS) — needs a human call on: (1) approving this first modality-only step of #8558 as-is or after the sandboxed verification above, and (2) whether snapshot regeneration should be wired into release automation as a follow-up. Review found no blocking issues; the three nits in my Stage 2 comment are non-blocking. 中文说明置信度：3/5 —— 就代码本身而言是一次干净的审查，但这是核心模块 500+ 生产行的 feature，按策略这类决定应由维护者做出，而不是自动批准。 退一步看：这个 PR 正是 #8558 规划的第一步——只做模态，先于 limits/reasoning/价格分阶段推进——issue 甚至点名本 PR 作为该方案的示范。我对这个问题的独立基线会更小（只做精确 provider/model 查找），但这里额外的机制——Alibaba 区域 endpoint、网关模型家族借用、OpenRouter 变体后缀、Idealab 别名——每一条都对应本仓库 preset 中已存在的配置面，且都有测试覆盖。所以范围是挣来的，不是堆砌的。 我最慎重权衡的部分：信任网络加载的元数据是否安全（发布/缓存前严格校验、响应上限与超时、原子写、失败回退到最后可用数据；只读模态，目录数据不进入请求体/请求头/路由——这是 #8558 的硬性边界）；是否会回归（显式配置在原先优先的地方依然优先；唯一有意的行为变化是 MiniMax-M3 改为尊重用户持久化值而非强制归一化，有测试且与 PR 意图一致；Serve A/B 对 base 零响应漂移）；长期维护性（内置快照会随发布逐渐过时，生成脚本已有但尚无自动化——这是唯一建议跟进的事项，不构成阻断）；无法验证的部分（wire 层"图片以 image_url 发送"依赖作者的 macOS E2E 声明，Windows/Linux 未测试——审查评论中已点名 我不做批准——不是因为代码有疑点（没有），而是因为触及核心、约 797 生产行的 ⏸️ 转交 @wenshao @tanzhenxin @yiliang114 @LaZzyMan @doudouOUC（core CODEOWNERS）——需要人工决定：(1) 是否按现状批准 #8558 的这个"仅模态"第一步，或先做上述沙箱验证；(2) 快照再生成是否应接入发布自动化作为后续事项。审查未发现阻断问题；Stage 2 评论中的三个小问题均为非阻断。 — Qwen Code · qwen3.8-max Reviewed at |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
This PR resolves missing model input modalities from models.dev for configured models and runtime model switches. It ships a compact modality-only snapshot, uses a valid disk cache immediately, and refreshes remote metadata in the background without making cold startup wait for the network.
Successful refreshes become available to configurations created later in the same long-running process, while existing sessions keep stable capabilities. Explicit user configuration remains authoritative, catalog metadata is used after an exact provider/model match or a narrow documented provider-family alias, and existing model-name heuristics remain the final fallback.
Why it's needed
Model modality support is currently inferred mainly from model names. OpenAI-compatible providers can expose models whose identifiers are unknown to Qwen Code, and their model-list APIs usually do not include normalized input-modality metadata, so valid image attachments can be replaced with an unsupported-image text placeholder.
Using normalized metadata recognizes new models without adding a model-name rule for every provider release. The built-in snapshot and stale-while-revalidate cache also keep first use, offline use, and long-running daemon sessions deterministic when models.dev is slow or unavailable.
Reviewer Test Plan
How to verify
/authprovider-install flow. Confirm both paths apply catalog modalities when no explicit value is present.bailian/DeepSeek, Kimi, MiniMax, and Qwen model families through their official-provider counterparts while unknown families remain unmatched./authtemplate, then refresh auth and switch between registry models. Confirm the explicit value remains unchanged. Confirm newly generated/authentries omit modalities and use catalog metadata instead.Evidence (Before & After)
Before: Qwen Code 0.20.1 treated a JPEG attached to
thinkingmachines/inklingas unsupported and sent only the text placeholder because the model identifier was unknown to the built-in heuristic.After: the source build sent the same JPEG as an
image_url. The focused metadata, registry, configuration, resolver, and workspace-status suite reports 286 passing tests, including a cold-cache request that never resolves, stale refresh success and failure, later same-process loads, manual configuration, runtime provider installation, explicit overrides, and unknown-provider isolation. Repository build and typecheck also complete successfully.Tested on
Environment (optional)
Local Node.js 22 TypeScript runtime without a sandbox, using injected catalog requests and temporary caches for deterministic failure and refresh scenarios, plus a local OpenAI-compatible endpoint for attachment verification.
Risk & Scope
Linked Issues
Related to #8558.
中文说明
这个 PR 做了什么
这个 PR 为已配置模型和运行时模型切换从 models.dev 补全缺失的输入模态。它内置一份仅包含模态信息的精简快照，立即使用有效磁盘缓存，并在后台刷新远端元数据，不让冷启动等待网络。
刷新成功后，同一长生命周期进程里之后创建的配置会使用新数据，而已有会话保持稳定能力。用户显式配置始终具有最高优先级；目录元数据只在精确 provider/model 命中或已登记的窄范围模型家族别名命中后使用；现有基于模型名称的启发式规则继续作为最终兜底。
为什么需要
当前模型模态支持主要通过模型名称推断。OpenAI 兼容提供商可能暴露 Qwen Code 未知标识符的模型，而其模型列表 API 通常不包含统一的输入模态元数据，因此有效图片附件可能被替换为“不支持图片”的文本占位符。
使用统一元数据可以识别新模型，无需为每次 provider 发布都添加一条模型名称规则。内置快照和 stale-while-revalidate 缓存也能在 models.dev 缓慢或不可用时，让首次使用、离线使用和长生命周期 daemon 会话保持确定性。
审阅者测试计划
如何验证
/authprovider 安装流程配置同一个目录模型。确认没有显式值时，两条路径都会应用目录模态。bailian/下的 DeepSeek、Kimi、MiniMax、Qwen 模型家族使用对应官方 provider 的同型号元数据，未知家族保持不匹配。/auth已持久化的模态，再刷新 auth 并切换 registry 模型，确认显式值保持不变；同时确认新版/auth不再写入默认模态，而是交给 catalog 解析。证据（修改前与修改后）
修改前：Qwen Code 0.20.1 会将附加到
thinkingmachines/inkling的 JPEG 视为不支持；由于内置启发式规则不识别该模型标识符，因此只发送文本占位符。修改后：源码构建会将同一张 JPEG 作为
image_url发送。聚焦于元数据、注册表、配置解析和 workspace status 的测试套件共 286 个测试通过，其中包括永不完成请求的冷缓存场景、陈旧缓存刷新成功与失败、同进程后续加载、手动配置、运行时 provider 安装、显式覆盖和未知 provider 隔离。仓库构建和类型检查也都成功完成。测试平台
环境（可选）
本地 Node.js 22 TypeScript 运行时，未启用沙箱；通过注入目录请求和临时缓存稳定复现失败与刷新场景，并通过本地 OpenAI 兼容 endpoint 验证附件行为。
风险与范围
关联 Issue
关联 #8558。