feat(gpt-oss): combine xgrammar structural_tag grammar with Harmony parser

[windreamer]

Motivation

GPT-OSS Harmony response parsing relies on openai_harmony for token stream parsing, but lacks grammar-level constraints — tool_choice=required cannot enforce tool calling, and response_format can only be injected into the prompt (soft constraint). This PR combines xgrammar structural_tag as a grammar constraint layer with the existing Harmony parser to achieve hard guarantees.

Modification

Protocol (lmdeploy/serve/openai/protocol.py)

Add structural_tag to ResponseFormat.type and a structural_tag: dict[str, Any] | None field.

Parser (lmdeploy/serve/parsers/_openai_harmony.py)

• Replace _convert_response_format_to_harmony() with _maybe_inject_tool_grammar() in __init__.
• With tools + tool_choice != 'none': build a Harmony-compatible structural_tag via xgrammar.get_model_structural_tag("harmony", ...), set it on response_format without clearing (grammar reaches the engine).
• Without tools + non-text response_format: wrap json_schema/regex_schema/json_object into a Harmony final-channel structural_tag via _build_response_format_grammar(), no longer injecting into the prompt.
• Grammar construction fails: fall back to the original _convert_response_format_to_harmony (inject system prompt + clear).
• Both tools and response_format present: tool grammar takes priority (consistent with vLLM).

Bug fixes (found during review)

• AllowedToolChoice crash: __init__ tool filtering used not isinstance(tool_choice, str) then accessed .function.name, crashing on AllowedToolChoice (type='allowed_tools'). Fixed with getattr(tool_choice, 'type') == 'function'.
• Dead BaseModel.schema branch: _build_response_format_grammar had an elif hasattr(schema, 'schema') branch that always picked up the deprecated BaseModel.schema method (a bound method, not None) when json_schema had no inner schema, producing invalid grammar. Removed.
• Tool grammar failure left response_format intact: when _build_tool_grammar returned None, a non-text response_format was left on the request, which normalize_chat_request would pass through to the engine — re-enabling json/regex modes that conflict with Harmony tool-call constraints. Now falls back to prompt injection and clears.

Tests (tests/test_lmdeploy/serve/parsers/test_gpt_oss_parser.py)

• Regression tests for all three bug fixes.
• TestGptOssResponseFormatGrammarConversion: verify json_schema/regex_schema/json_object → structural_tag conversion, grammar-failure fallback, and BaseModel.schema leak prevention.
• TestGptOssToolGrammarInjection: verify required/auto/specific function/none/allowed_tools/priority/failure-fallback.
• Consolidated the existing test suite: merged redundant fixtures, removed dead helpers, parametrized overlapping cases, dropped tests covering code paths unchanged by this PR.

BC-breaking

None. Adding structural_tag to ResponseFormat is purely additive. The legacy _convert_response_format_to_harmony is retained as fallback.

Checklist

1. Pre-commit hooks pass (ruff, docformatter, clang-format, etc.).
2. 29/29 tests pass in test_gpt_oss_parser.py.
3. No new external dependency — uses vendored xgrammar's get_model_structural_tag.
4. No documentation changes needed (response_format docstring intentionally unchanged).

[Copilot]

Pull request overview

This PR integrates xgrammar structural_tag guided generation with the existing GPT-OSS Harmony response parser so that tool-calling and structured outputs can be enforced via hard grammar constraints (instead of Harmony-native prompt injection), while keeping a legacy prompt-injection fallback when grammar construction is unavailable.

Changes:

• Extend the OpenAI protocol ResponseFormat to support type='structural_tag' and carry a structural_tag payload.
• Update the GPT-OSS Harmony parser to build and attach Harmony-compatible structural_tag grammars for (a) tool calling and (b) non-text response_format modes, with a prompt-injection fallback.
• Revise and expand unit tests to validate structural_tag conversion/injection and the fallback behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
lmdeploy/serve/openai/protocol.py	Adds structural_tag to ResponseFormat so requests can carry structural-tag grammars end-to-end.
lmdeploy/serve/parsers/_openai_harmony.py	Implements structural_tag grammar generation for tool calling and response_format, plus legacy fallback injection.
tests/test_lmdeploy/serve/parsers/test_gpt_oss_parser.py	Updates tests to assert structural_tag behavior and legacy fallback when grammar building fails.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

lmdeploy/serve/parsers/_openai_harmony.py:109

• ResponseFormat.type now supports 'structural_tag', but _maybe_inject_tool_grammar() treats any non-text response_format as needing conversion and will fall back to prompt injection (which clears response_format) when type=='structural_tag'. That makes client-supplied structural tags effectively unusable.

                if fmt is not None and getattr(fmt, 'type', 'text') != 'text':
                    self._convert_response_format_to_harmony()
            return
        if fmt is not None and getattr(fmt, 'type', 'text') != 'text':
            grammar = self._build_response_format_grammar(fmt)

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

[windreamer]

Thanks for the review! @lvhan028

Fully addressed in d5d8a5f (force-pushed): the structural_tag extension to ResponseFormat has been removed entirely. protocol.py is no longer touched by this PR — ResponseFormat.type stays at text / json_object / json_schema / regex_schema per the OpenAI spec.

What changed

The structural_tag is now a purely internal mechanism: the Harmony parser builds the grammar server-side (via xgrammar.get_model_structural_tag("harmony", ...) for tool calling, or a final-channel wrapper for json_schema/regex_schema/json_object) and attaches it to the request's response_format dict after protocol validation, so it flows to the engine through the existing response_format → gen_config pipeline untouched. normalize_chat_request now passes dict-shaped response_format through as-is (response_parser.py, +4 lines) since it may hold this internal payload.

I also verified the vLLM convention mentioned in the review: vLLM handles Harmony constraints server-side in vllm/parser/harmony.py (_adjust_output_format / _apply_structural_tag), rewriting constraints into a Harmony-aware structural_tag internally and clearing response_format — the same shape as this PR now does. (vLLM's client-facing extra_body field structured_outputs exists, but adding it here would expand lmdeploy's protocol surface, which this PR doesn't need — happy to follow up separately if we want client-supplied tags.)

Other review items (from Copilot, fixed in prior commits, kept in this rework)

• AllowedToolChoice crash in __init__ tool filtering — fixed with getattr(tool_choice, 'type') == 'function'
• Dead BaseModel.schema branch in _build_response_format_grammar — removed
• Tool grammar failure no longer leaves response_format intact — falls back to prompt injection and clears

All 29 tests in test_gpt_oss_parser.py pass; engine files (turbomind.py, guided_process.py) are untouched since the engine already supported structural_tag grammars.

[Copilot]

🟡 Changes recommended

Unresolved moderate issues affect allowed-tool handling and structural-tag request validation.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (4)

lmdeploy/serve/parsers/_openai_harmony.py:203

• The documented fallback is not honored for exceptions raised during grammar construction or serialization: only the xgrammar imports are inside try, so a validation/API error from JSONSchemaFormat (and likewise the regex/structural-tag constructors below) escapes GptOssResponseParser.__init__ instead of calling _convert_response_format_to_harmony(). Keep the conversion and model_dump_json() work under the exception handler, as _build_tool_grammar() does.

        if fmt_type == 'json_schema':
            # normalize dumps JsonSchema via by_alias: inner schema lives at
            # key 'schema' (OpenAI wire format).
            raw = (fmt.get('json_schema') or {}).get('schema')

lmdeploy/serve/parsers/_openai_harmony.py:207

• An empty regex is a valid value for the regex_schema: str field, but or '.*' converts it into an unconstrained wildcard. That weakens the promised hard constraint for this edge case; distinguish None from '' when choosing the default.

        elif fmt_type == 'json_object':

lmdeploy/serve/parsers/_openai_harmony.py:211

• The new structural_tag response-format type is treated as unsupported here, so _maybe_inject_tool_grammar falls through to legacy prompt injection, serializes the grammar as prose, and clears it. A caller that supplies an already-built structural tag therefore loses hard grammar enforcement; pass this type through unchanged before handling the legacy formats.

            return None

lmdeploy/serve/parsers/_openai_harmony.py:50

• When tool_choice is AllowedToolChoice and top-level request.tools is omitted, allowed_tools.tools is the source of the allowed definitions (the base parser populates tools from it), but this branch discards them as None. The subsequent has_tools check skips _build_tool_grammar, so GPT-OSS cannot emit any requested allowed tool; preserve/render those definitions here.

                rendered_tools = None
            elif getattr(request.tool_choice, 'type', None) == 'function':

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite