Skip to content

Keep render tool results as Preview paths; hydrate before the model - #4971

Merged
moshiii-wei merged 7 commits into
mainfrom
moshi/fix/agents-render-image-refs
Nov 2, 2026
Merged

moshiii-wei merged 7 commits into
mainfrom
moshi/fix/agents-render-image-refs

Conversation

@moshiii-wei

@moshiii-wei moshiii-wei commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Keep bash/stream/event render tool results as Preview: paths + details.render FileRefs instead of inlining base64 image blocks (fixes gateway NDJSON truncation / VQA Err).
  • Hydrate cached preview images request-scoped in GenerationTask.beforeRequest so the model still sees them.

Test plan

  • npm run typecheck in dimos/agents
  • preview-refs / render / clip / points / boundaries / intake tests
  • Full npm test (remaining flakes unrelated to this change)
  • Greptile 5/5

…he model.

Stops base64 image blocks from riding bash/stream/event tool output into session history and gateway NDJSON (VQA Err truncation). GenerationTask.beforeRequest reloads cached previews for the LLM only; drop retired npm dimcode from install-test-clis.

@github-actions github-actions Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go2 replay realtime (arm64)

Details
Benchmark suite Current: aaf8323 Previous: 2e7688b Ratio
peak memory 1748.297 MB 1716.07 MB 1.02
peak threads 364 threads 374 threads 0.97
network (transport) 2724.091 MB 2725.848 MB 1.00
disk write 1.758 MB 1.738 MB 1.01
instructions 137.219 G 136.431 G 1.01

This comment was automatically generated by workflow using github-action-benchmark.

@codecov

codecov Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

@@            Coverage Diff             @@
##             main    #4971      +/-   ##
==========================================
+ Coverage   83.00%   83.08%   +0.08%     
==========================================
  Files        1752     1752              
  Lines      169315   169315              
  Branches    13955    13955              
==========================================
+ Hits       140542   140681     +139     
+ Misses      25017    24978      -39     
+ Partials     3756     3656     -100     
Components Coverage Δ
Tests 96.44% <ø> (+0.03%) ⬆️
Flag Coverage Δ
OS-macos-latest 77.46% <ø> (+<0.01%) ⬆️
OS-ubuntu-24.04-arm 77.25% <ø> (-0.01%) ⬇️
OS-ubuntu-latest 77.63% <ø> (-0.01%) ⬇️
Py-3.10 77.62% <ø> (ø)
Py-3.11 77.61% <ø> (-0.01%) ⬇️
Py-3.12 77.62% <ø> (+<0.01%) ⬆️
SelfHosted-Linux 40.08% <ø> (+<0.01%) ⬆️
SelfHosted-macOS 39.80% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 28 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Drop the evals CLI and comment-only edits so the change is just Preview paths + beforeRequest hydrate.
@moshiii-wei

Copy link
Copy Markdown
Contributor Author

@greptileai

@greptile-apps

greptile-apps Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] No blocking findings remain. The outstanding photo-clip encoding concern is non-blocking.

Findings

  1. P2 Photos become larger PNG requests ▶

Summary

Keeps rendered tool results as text paths and structured file references, then adds images only to the next model request.

  • Bash and stream results keep preview paths and restore images for model requests.

Reviews (6) · Last reviewed commit: "Seal verified clip frame bytes into a te..." · Reviewed by Greptile

Comment thread dimos/agents/harness/events/message.ts Resolved
Addresses Greptile P1s: stop reading arbitrary Preview: paths, rebuild model images from trusted sources when the cache is gone, and keep event exports on the draft so truncation cannot drop them.
@moshiii-wei

Copy link
Copy Markdown
Contributor Author

@greptileai

@greptile-apps

greptile-apps Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Want your agent to iterate on Greptile's feedback? Try greploops.

Closes the remaining Greptile cache-lifetime P1: when RenderCache drops a contact sheet, restore the model image by re-rendering the trusted source instead of omitting it.
@moshiii-wei

Copy link
Copy Markdown
Contributor Author

@greptileai

@moshiii-wei

Copy link
Copy Markdown
Contributor Author

@greptileai please re-review — clip rebuild now verifies each view.clip.frames source.sha256 via readVerified before re-rendering (c4dabc5).

The renderer must consume the same hashed bytes recorded on the tool result, not re-read the live tree after a separate check.
@moshiii-wei

Copy link
Copy Markdown
Contributor Author

@greptileai please re-review — clip rebuild now seals readVerified frame bytes into a temp index (with those sha256s) and renders only that copy, so the renderer never re-reads the live tree (5858134).

@moshiii-wei
moshiii-wei marked this pull request as ready for review November 1, 2026 18:04
@github-actions github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Nov 1, 2026
@moshiii-wei
moshiii-wei enabled auto-merge (squash) November 1, 2026 18:04
@github-actions github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Nov 2, 2026
@github-actions github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Nov 2, 2026
@moshiii-wei
moshiii-wei merged commit c3e490e into main Nov 2, 2026
41 checks passed
@moshiii-wei
moshiii-wei deleted the moshi/fix/agents-render-image-refs branch November 2, 2026 22:07

This branch was previously deployed

1 inactive deployment
cachix — aaf83238 Deployed Nov 2, 2026 by moshiii-wei via cachix-build-macos #15457
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-merge Required CI checks have passed on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants