feat: forward tool-result images to the MCP caller in send_message results
fast-agent's agent.send() returns only the final assistant text, so ImageContent produced by downstream tools during the agentic loop (playwright screenshots, rommie desktop captures) reached the agent's own vision model but never crossed the MCP boundary — Daedalus and lead agents saw text-only results. A per-request after_tool_call hook (pallas.image_passthrough, same composition pattern as assistant_stream / loop_guard) collects every ImageContent block from the turn's tool results; send_message then returns a FastMCP ToolResult of [final text, *images]. Turns with no images return the plain string — wire shape unchanged (the str-only output schema is dropped so the union return passes through cleanly; no consumer read structuredContent). Images cascade hop-by-hop up delegation chains with no extra wiring: verified live playwright → dolores → harper → MCP client, image intact at each hop. New per-agent agents.yaml knob max_result_images (default 8, keeps most recent, 0 disables) and pallas_result_images_total counter. Version 0.7.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -111,7 +111,7 @@ server.py main()
|
||||
|---|---|
|
||||
| `pallas.server` | CLI entry point, configuration loading, agent lifecycle orchestration, model registration |
|
||||
| `pallas.registry` | Starlette app serving `GET /.well-known/mcp/server.json` — builds the agent catalogue from `agents.yaml` + `fastagent.config.yaml` |
|
||||
| `pallas.multimodal_server` | `MultimodalAgentMCPServer` — `AgentMCPServer` subclass adding image attachment support and conversation history prompts |
|
||||
| `pallas.multimodal_server` | `MultimodalAgentMCPServer` — `AgentMCPServer` subclass adding image attachment support, tool-result image passthrough, and conversation history prompts |
|
||||
| `pallas.health` | Two-layer health: startup LLM preflight validation + runtime `get_health` MCP tool with downstream server probing |
|
||||
|
||||
---
|
||||
@@ -195,6 +195,7 @@ agents:
|
||||
| `agents.<name>.depends_on` | no | List of agent names that must start and become ready before this agent |
|
||||
| `agents.<name>.max_iterations` | no | Hard cap on agentic-loop turns per `send_message`. Default: `15`. fast-agent returns a partial answer once exceeded |
|
||||
| `agents.<name>.loop_repeat_threshold` | no | Halt the loop after this many consecutive identical `(tool, args) → result` rounds. Default: `3`. `0` disables the guard |
|
||||
| `agents.<name>.max_result_images` | no | Cap on tool-result images forwarded in the final `send_message` result (most recent kept). Default: `8`. `0` disables image passthrough |
|
||||
|
||||
### `fastagent.config.yaml` Extensions
|
||||
|
||||
@@ -447,6 +448,14 @@ Each agent's MCP tool accepts:
|
||||
|
||||
When `images` is provided, the message is sent as a `PromptMessageExtended` containing both `TextContent` and `ImageContent` parts — the agent's underlying model must support vision.
|
||||
|
||||
### Tool-Result Image Passthrough
|
||||
|
||||
Images work in both directions. fast-agent's `agent.send()` returns only the final assistant text, so images produced by downstream tools during the agentic loop (playwright screenshots, rommie desktop captures) would otherwise reach the agent's own vision model but never the MCP caller. A per-request `after_tool_call` hook (`pallas.image_passthrough`) collects every `ImageContent` block from the turn's tool results; at end of turn `send_message` returns a `CallToolResult` whose content is the assistant's text block followed by the collected images. Turns that produce no images return the plain string, unchanged from previous releases.
|
||||
|
||||
Only the most recent `max_result_images` (default `8`) are forwarded — screenshots are usually taken to show final state, and an unbounded loop of full-desktop captures would balloon the HTTP response. Forwarded images are counted in `pallas_result_images_total`; drops are logged.
|
||||
|
||||
Because a lead agent receives a sub-agent's images as ordinary tool-result content, the same hook on the lead's own turn forwards them again — images cascade hop-by-hop up the delegation chain (playwright → dolores → lead → Daedalus) with no extra wiring.
|
||||
|
||||
### Conversation History Prompt
|
||||
|
||||
For agents with `instance_scope != "request"`, a `{agent}_history` prompt is registered that returns the full conversation history as FastMCP `Message` objects. This allows clients to retrieve the stored context.
|
||||
@@ -606,6 +615,7 @@ scrape_configs:
|
||||
| `pallas_llm_provider_up` | gauge | `provider` | `1` when the active LLM provider passed its last preflight or runtime re-probe |
|
||||
| `pallas_agent_health_status` | gauge | `agent` | Aggregate from the last `get_health`: `1`=ok, `0.5`=degraded, `0`=error |
|
||||
| `pallas_agent_loop_aborted_total` | counter | `agent`, `reason` | Agentic loops force-stopped by a runtime guard. `reason` ∈ `repeat` (identical-tool-call loop detected) |
|
||||
| `pallas_result_images_total` | counter | `agent` | Tool-result images forwarded to the MCP caller in `send_message` results |
|
||||
|
||||
Standard process metrics (RSS, CPU, GC, open FDs) are emitted by `prometheus-client`'s default collectors on the same endpoint.
|
||||
|
||||
@@ -683,6 +693,7 @@ This avoids the brittle pattern of inferring capabilities from model name substr
|
||||
| `pallas.multimodal_server` | `multimodal_server.py` | `MultimodalAgentMCPServer` — extends `AgentMCPServer` with image support, conversation history prompts, bearer token propagation |
|
||||
| `pallas.health` | `health.py` | LLM provider preflight validation, downstream MCP server probing, `get_health` tool registration |
|
||||
| `pallas.loop_guard` | `loop_guard.py` | Per-request `ToolRunnerHooks` that halt the agentic loop on repeated-identical tool calls |
|
||||
| `pallas.image_passthrough` | `image_passthrough.py` | Per-request `ToolRunnerHooks` that collect tool-result images so `send_message` can return them in the final `CallToolResult` |
|
||||
| `pallas.log` | `log.py` | JSON log configuration, third-party traceback capture, Rich-TUI-safe handler attachment |
|
||||
| `pallas._fastagent_patch` | `_fastagent_patch.py` | Monkey-patches fast-agent at import time: per-request bearer forwarding via `httpx.Auth`, diagnostic trace-capture wrappers around `send_request` / `session.call_tool` / `_execute_on_server` |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user