r/pallas
1
0
Fork 0
Code Issues 2 Pull Requests Packages Projects Releases Wiki Activity

feat: forward tool-result images to the MCP caller in send_message results #5

Merged
r merged 1 commits from feat/tool-result-image-passthrough into main 2026-08-02 13:14:58 +00:00
Conversation 0 Commits 1 Files Changed 7 +359 -5
r commented 2026-08-02 02:47:56 +00:00
Owner
Copy Link

Problem

Pallas multimodal support was inbound-only. agent.send() returns only the final assistant text, so ImageContent produced by downstream tools during the agentic loop — playwright screenshots, rommie desktop captures — reached the agent's own vision model but never crossed the MCP boundary. Daedalus (and any lead agent driving a sub-agent) saw a text-only CallToolResult; the live stream reduced result images to a "+N images" preview. Screenshots never reached Robert.

Fix

  • New pallas.image_passthrough module: a per-request after_tool_call hook (same merge/restore composition as assistant_stream / loop_guard) collects every ImageContent block from the turn's tool results.
  • send_message now returns a FastMCP ToolResult of [final text, *images] when images were collected; turns with no images return the plain string as before.
  • The -> str return annotation becomes -> str | ToolResult, which drops FastMCP's auto-generated {"result": string} output schema. Verified no consumer reads structuredContent (Daedalus reads result.content only, and already renders image blocks — chat.py extracts type == "image").
  • Images cascade hop-by-hop up delegation chains with no extra wiring: a lead receives the sub-agent's images as ordinary tool-result content, so its own hook forwards them again.

Config & observability

  • Per-agent agents.yaml knob max_result_images (default 8, keeps the most recent, 0 disables passthrough) — plumbed through request_limits like the other per-agent limits.
  • New pallas_result_images_total{agent} counter; drops are logged.

Verification

  • 67/67 tests pass (9 new in tests/test_image_passthrough.py).
  • FastMCP 3.4.2 round-trip test: ToolResult passes through verbatim, union annotation yields outputSchema: None.
  • Live sandbox e2e (branch pallas via PYTHONPATH, temp topology): direct Dolores call returned [text, image/png]; Harper → Dolores delegation returned the same image intact through both hops; pallas_result_images_total counted dolores=2, harper=1.

Found along the way (fixed in the open Dolores PRs, not here)

playwright's browser_take_screenshot omits the image from its result whenever a filename argument is passed — Dolores was helpfully naming files, so no image ever existed to forward. Also, a screenshot taken immediately after a cold browser launch can fail with Unable to capture screenshot (surface-readiness race; retry succeeds). Both addressed via prompt guidance in kottos#2 / mentor#3 / iolaus#4.

Release train

v0.6.0 tagged on pre-change main as the rollback point. Post-merge: tag v0.7.0, then pin pallas_rel in ouranos/virgo inventories when ready (currently main). Version bumped to 0.7.0 in this PR.

🤖 Generated with Claude Code

## Problem Pallas multimodal support was inbound-only. `agent.send()` returns only the final assistant text, so `ImageContent` produced by downstream tools during the agentic loop — playwright screenshots, rommie desktop captures — reached the agent's own vision model but never crossed the MCP boundary. Daedalus (and any lead agent driving a sub-agent) saw a text-only `CallToolResult`; the live stream reduced result images to a `"+N images"` preview. Screenshots never reached Robert. ## Fix - New `pallas.image_passthrough` module: a per-request `after_tool_call` hook (same merge/restore composition as `assistant_stream` / `loop_guard`) collects every `ImageContent` block from the turn's tool results. - `send_message` now returns a FastMCP `ToolResult` of `[final text, *images]` when images were collected; turns with no images return the plain string as before. - The `-> str` return annotation becomes `-> str | ToolResult`, which drops FastMCP's auto-generated `{"result": string}` output schema. Verified no consumer reads `structuredContent` (Daedalus reads `result.content` only, and already renders image blocks — `chat.py` extracts `type == "image"`). - Images cascade hop-by-hop up delegation chains with no extra wiring: a lead receives the sub-agent's images as ordinary tool-result content, so its own hook forwards them again. ## Config & observability - Per-agent `agents.yaml` knob `max_result_images` (default 8, keeps the most recent, `0` disables passthrough) — plumbed through `request_limits` like the other per-agent limits. - New `pallas_result_images_total{agent}` counter; drops are logged. ## Verification - 67/67 tests pass (9 new in `tests/test_image_passthrough.py`). - FastMCP 3.4.2 round-trip test: `ToolResult` passes through verbatim, union annotation yields `outputSchema: None`. - **Live sandbox e2e** (branch pallas via `PYTHONPATH`, temp topology): direct Dolores call returned `[text, image/png]`; Harper → Dolores delegation returned the same image intact through both hops; `pallas_result_images_total` counted dolores=2, harper=1. ## Found along the way (fixed in the open Dolores PRs, not here) playwright's `browser_take_screenshot` **omits the image from its result whenever a `filename` argument is passed** — Dolores was helpfully naming files, so no image ever existed to forward. Also, a screenshot taken immediately after a cold browser launch can fail with `Unable to capture screenshot` (surface-readiness race; retry succeeds). Both addressed via prompt guidance in kottos#2 / mentor#3 / iolaus#4. ## Release train `v0.6.0` tagged on pre-change main as the rollback point. Post-merge: tag `v0.7.0`, then pin `pallas_rel` in ouranos/virgo inventories when ready (currently `main`). Version bumped to 0.7.0 in this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
r added 1 commit 2026-08-02 02:47:57 +00:00
feat: forward tool-result images to the MCP caller in send_message results 1b05504207 
fast-agent's agent.send() returns only the final assistant text, so
ImageContent produced by downstream tools during the agentic loop
(playwright screenshots, rommie desktop captures) reached the agent's own
vision model but never crossed the MCP boundary — Daedalus and lead agents
saw text-only results.

A per-request after_tool_call hook (pallas.image_passthrough, same
composition pattern as assistant_stream / loop_guard) collects every
ImageContent block from the turn's tool results; send_message then returns
a FastMCP ToolResult of [final text, *images]. Turns with no images return
the plain string — wire shape unchanged (the str-only output schema is
dropped so the union return passes through cleanly; no consumer read
structuredContent). Images cascade hop-by-hop up delegation chains with no
extra wiring: verified live playwright → dolores → harper → MCP client,
image intact at each hop.

New per-agent agents.yaml knob max_result_images (default 8, keeps most
recent, 0 disables) and pallas_result_images_total counter. Version 0.7.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
r merged commit 5af198911b into main 2026-08-02 13:14:58 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
r
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: r/pallas#5
Delete Branch "feat/tool-result-image-passthrough"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?