Skip to content

fix(template): replace rejected responses after the last user message - #10127

Open
Excelius-Wang wants to merge 1 commit into
modelscope:mainfrom
Excelius-Wang:fix/rejected-tool-round
Open

Excelius-Wang wants to merge 1 commit into
modelscope:mainfrom
Excelius-Wang:fix/rejected-tool-round

Conversation

@Excelius-Wang

Copy link
Copy Markdown
Contributor

Agent preference data using tool messages currently inherits chosen tool calls and results when building rejected_messages from rejected_response. Keep the documented replacement boundary at the last user message by adding an opt-out for tool boundaries to the existing helper. Its default behavior, including the last-round loss fix in #9923, remains unchanged.

Validation: 10 tests and 12 subtests passed, including an actual JSONL/OpenAI tool-call loading regression and the existing loss-scale tests. Local CPU encoding checks covered 48 official-example variants and 192 cases from three public preference datasets; all match explicit rejected-message references after the fix. Six two-step tiny-model DPO runs produced identical losses and trained weights for compact versus explicit representations on each dataset. Changed-file pre-commit checks and git diff --check passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant