Use model token IDs for guidance markers

[Baiju Meswani (baijumeswani)]

What this fixes

Foundry Local builds an llguidance grammar when a request requires a tool call.

Previously, Foundry inserted marker text directly into the grammar:

toolcall: <tool_call> functioncall </tool_call>

In llguidance, <tool_call> means a named tokenizer special token. This fails for models that use the same marker as
an ordinary token or ordinary text.

What changed

Foundry now uses the boundary token IDs published by ORT GenAI:

• BOT — beginning of tool call
• EOT — end of tool call
• BOR — beginning of reasoning
• EOR — end of reasoning

When a published ID matches the configured marker, Foundry uses llguidance's exact numeric token syntax:

toolcall: <[248058]> functioncall <[248059]>

This works whether the tokenizer marks the token as special or ordinary.

If an ID is unavailable, invalid, or does not match an overridden marker, Foundry uses an escaped quoted string:

toolcall: "<tool_call>" functioncall "</tool_call>"

The quoted form allows llguidance to tokenize the marker as ordinary text, including markers represented by multiple
tokens.

Tool-call and reasoning start/end boundaries are resolved independently. The implementation contains no model names
or hardcoded token IDs.

Why not mark every marker as special?

Changing tokenizer metadata affects more than grammar construction. ORT GenAI normally decodes with
skip_special_tokens=true, so changing an ordinary marker to special: true can remove it from decoded output.

Using numeric IDs preserves the model package's tokenizer behavior while still constraining the exact published
token.

Required tool-call failures

If Foundry generates guidance for a required tool call and the runtime cannot apply it, the request now fails clearly
instead of continuing with unconstrained generation.

Invalid caller-provided guidance remains an invalid-argument error on both Generator and Engine backends.

Example

Request:

Use the multiply tool to calculate 17 times 23. Do not answer directly.

Result:

{
  "finish_reason": "tool_calls",
  "tool_calls": [
    {
      "type": "function",
      "function": {
        "name": "multiply",
        "arguments": "{\"a\":17,\"b\":23}"
      }
    }
  ]
}

After the tool returns 391, the next turn answers 17 × 23 = 391.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
foundry-local	Ready	Preview	Sep 11, 2026 1:21am UTC

[Copilot]

🔵 Needs a closer look

The grammar helper can render invalid numeric-token syntax for negative or sentinel IDs.

Pull request overview

Updates chat guidance to use exact published token IDs, with escaped-literal fallback and safer required-tool handling across Generator and Engine backends.

Changes:

• Added numeric-token grammar rendering and literal escaping.
• Propagated boundary IDs and reasoning state through chat flows.
• Expanded grammar, planning, and Engine tests.

File summaries

File	Description
sdk_v2/cpp/test/internal_api/toolcalling/grammar_test.cc	Tests numeric-token and literal rendering.
sdk_v2/cpp/test/internal_api/chat/search_options_test.cc	Tests guidance planning and reasoning state.
sdk_v2/cpp/test/internal_api/chat/dynamic_engine_chat_test.cc	Updates Engine turn tests.
sdk_v2/cpp/src/inferencing/generative/toolcalling/tool_call_context.h	Adds boundary token ID fields.
sdk_v2/cpp/src/inferencing/generative/toolcalling/grammar.h	Declares grammar rendering APIs.
sdk_v2/cpp/src/inferencing/generative/toolcalling/grammar.cc	Implements token rendering and escaping.
sdk_v2/cpp/src/inferencing/generative/genai_model_instance.h	Declares tokenizer encoding support.
sdk_v2/cpp/src/inferencing/generative/genai_model_instance.cc	Implements tokenizer encoding support.
sdk_v2/cpp/src/inferencing/generative/chat/search_options.h	Defines guidance planning interfaces.
sdk_v2/cpp/src/inferencing/generative/chat/search_options.cc	Handles guidance planning and errors.
sdk_v2/cpp/src/inferencing/generative/chat/onnx_engine_chat_stream.cc	Propagates reasoning state.
sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_generator.cc	Resolves markers and applies guidance.
sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_engine.h	Extends Engine guidance interfaces.
sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_engine.cc	Integrates Engine guidance handling.
sdk_v2/cpp/src/inferencing/generative/chat/chat_session.cc	Resolves published boundary IDs.

Review details

Suppressed comments (1)

sdk_v2/cpp/src/inferencing/generative/toolcalling/grammar.cc:59

• This treats any engaged ID as valid, so a sentinel or malformed negative ID produces <[-1]>, which is not valid llguidance numeric-token syntax. ResolveMarkerTokenId filters negatives on the current ChatSession path, but this public helper can also be called with a directly populated ToolCallContext; only nonnegative IDs should use exact-token rendering, with invalid IDs falling back to the escaped marker literal.

  if (token_id.has_value()) {
    return "<[" + std::to_string(*token_id) + "]>";

• Files reviewed: 15/15 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.