Skip to content

fix(bridge): decode every batch row in generate_stream - #1757

Merged
jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
emerardd:fix/generate-stream-batch-decode
Sep 8, 2026
Merged

fix(bridge): decode every batch row in generate_stream#1757
jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
emerardd:fix/generate-stream-batch-decode

Conversation

@emerardd

@emerardd emerardd commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Description

TransformerBridge.generate_stream(..., return_type="str") decoded tokens[0] only, so a batched stream yielded the first sequence's text and the rest of the batch was unreachable. Generation itself was fine — all rows are produced, and return_type="tokens" yields the full [batch, seq_len] tensor — only the decode step dropped them.

This decodes every row and keeps generate()'s unwrap convention: a bare string for a one-row batch, one string per row otherwise. The yield type widens to Generator[Union[torch.Tensor, str, List[str]], None, None].

prompts = ["The capital of France is", "My favourite colour is"]
bridge.generate(prompts, max_new_tokens=4, do_sample=False, verbose=False)
# ['The capital of France is the capital of the', 'My favourite colour is the red. I']
generate_stream(prompts, ...) Before After
yield 'The capital of France is the capital of the' ['The capital of France is the capital of the', 'My favourite colour is the red. I']

Single-string and single-row streaming are unchanged, which is why tests/acceptance/model_bridge/test_generate_stream.py::test_stream_returns_strings passed throughout.

HookedTransformer.generate_stream does not share the bug — it never decodes and always yields tensors, even for return_type="str". That is a separate inconsistency, so no mirroring change (AGENTS.md §2) is required here.

Fixes #1756

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

Tests added — both fail on dev and pass with the fix:

  • tests/unit/model_bridge/test_generate_stream_batch_decode.py — stubbed token stream, no model load; pins the batched list[str] yield and the single-row bare-string unwrap.
  • tests/integration/model_bridge/test_generate_stream_batched.py — distilgpt2 end-to-end; each yield carries one string per prompt and the first yield echoes each row's own prompt.

generate_stream(return_type="str") decoded tokens[0] only, so a batched
stream yielded the first sequence's text and the rest of the batch was
unreachable — generation itself was fine, only the decode step dropped
the other rows.

Decode each row and keep generate()'s unwrap convention: a bare string
for a one-row batch, one string per row otherwise. Covered at the unit
tier with a stubbed token stream and at the integration tier against
distilgpt2.

Fixes TransformerLensOrg#1756
@emerardd
emerardd force-pushed the fix/generate-stream-batch-decode branch from f7b9432 to 65c5205 Compare September 8, 2026 15:04
@jlarson4

jlarson4 commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Great work on this @emerardd! Assuming Full Code Coverage passes, I will merge

@jlarson4
jlarson4 merged commit 7edc5dd into TransformerLensOrg:dev Sep 8, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants