r/pallas
1
0
Fork 0
Code Issues 2 Pull Requests Packages Projects Releases Wiki Activity

🐾 fix(mantle): survive Mantle's 20k output ceiling on tool-heavy turns #8

Merged
r merged 1 commits from feature/mantle-max-tokens-shim into main 2026-08-04 20:54:12 +00:00
Conversation 0 Commits 1 Files Changed 3 +197 -21
r commented 2026-08-04 20:49:44 +00:00
Owner
Copy Link

Problem

Alan (Mentor, Taurus prod) fails on daedalus__revise_workspace_file with Streaming completed but tool call never finished: toolu_bdrk_… after ~700 s. Root cause chain, established from Taurus Loki + fast-agent 0.7.22 source:

  1. fast-agent requests max_tokens=128000 (ModelDatabase for Opus 4.8), but Mantle clamps output at 20 000 tokens server-side — streams observed completing at exactly Output tokens: 20000.
  2. fast-agent unconditionally sends the fine-grained-tool-streaming-2025-05-14 beta on tool-bearing requests. Under that beta, hitting the ceiling mid-tool_use block ends the stream without content_block_stop.
  3. _process_stream sees a cleanly-ended stream with an open tool block → RuntimeError → 2 retries × ~2 min streaming into the same wall + backoff = the observed ~700 s failures. The tool call never reaches Daedalus.
  4. A large revise_workspace_file turn (thinking + prose + 25 KB body_markdown last) reliably crosses 20 000, so big revises fail deterministically. Text-only turns crossing the ceiling truncate silently.

Fix

Two new Mantle shims (auto-installed by the existing maybe_install base-url detection):

  • Shim 3 — fine-grained tool streaming opt-out: patch AnthropicLLM.supports_direct_anthropic_beta to refuse fine_grained_tool_streaming. A cutoff then closes blocks properly and lands in fast-agent's graceful stop_reason=max_tokens handling.
  • Shim 4 — maxTokens clamp: clamp default maxTokens to MANTLE_MAX_OUTPUT_TOKENS = 20_000 so the model stops itself at the limit instead of the gateway cutting the stream.

Docs: docs/bedrock.md Mantle section now describes all four shims and the per-turn output ceiling.

Verification

  • tests/test_mantle_shims.py: 21 tests (new coverage for both shims: opt-out behaviour, other betas untouched, clamp over/under/None, idempotency, maybe_install ordering).
  • Full suite: 90 passed on fast-agent 0.7.22 (repo .venv upgraded from stale 0.7.2 to the pyproject pin).

Notes

  • This makes the failure fast and explicit rather than a 12-minute retry storm; it cannot raise the 20 k ceiling itself. The durable companion fix is patch-style revise in Daedalus (separate PR) so tool inputs stay small.
  • If Mantle ignores the beta header (i.e. its truncation behaviour is native), Shim 4 still converts the failure into a graceful model-side stop; Shim 3 is then inert but harmless.

🤖 Generated with Claude Code

## Problem Alan (Mentor, Taurus prod) fails on `daedalus__revise_workspace_file` with `Streaming completed but tool call never finished: toolu_bdrk_…` after ~700 s. Root cause chain, established from Taurus Loki + fast-agent 0.7.22 source: 1. fast-agent requests `max_tokens=128000` (ModelDatabase for Opus 4.8), but **Mantle clamps output at 20 000 tokens server-side** — streams observed completing at exactly `Output tokens: 20000`. 2. fast-agent unconditionally sends the `fine-grained-tool-streaming-2025-05-14` beta on tool-bearing requests. Under that beta, hitting the ceiling **mid-`tool_use` block ends the stream without `content_block_stop`**. 3. `_process_stream` sees a cleanly-ended stream with an open tool block → `RuntimeError` → 2 retries × ~2 min streaming into the same wall + backoff = the observed ~700 s failures. The tool call never reaches Daedalus. 4. A large `revise_workspace_file` turn (thinking + prose + 25 KB `body_markdown` last) reliably crosses 20 000, so big revises fail deterministically. Text-only turns crossing the ceiling truncate *silently*. ## Fix Two new Mantle shims (auto-installed by the existing `maybe_install` base-url detection): - **Shim 3 — fine-grained tool streaming opt-out**: patch `AnthropicLLM.supports_direct_anthropic_beta` to refuse `fine_grained_tool_streaming`. A cutoff then closes blocks properly and lands in fast-agent's graceful `stop_reason=max_tokens` handling. - **Shim 4 — `maxTokens` clamp**: clamp default `maxTokens` to `MANTLE_MAX_OUTPUT_TOKENS = 20_000` so the model stops itself at the limit instead of the gateway cutting the stream. Docs: `docs/bedrock.md` Mantle section now describes all four shims and the per-turn output ceiling. ## Verification - `tests/test_mantle_shims.py`: 21 tests (new coverage for both shims: opt-out behaviour, other betas untouched, clamp over/under/None, idempotency, `maybe_install` ordering). - Full suite: **90 passed** on fast-agent 0.7.22 (repo `.venv` upgraded from stale 0.7.2 to the pyproject pin). ## Notes - This makes the failure *fast and explicit* rather than a 12-minute retry storm; it cannot raise the 20 k ceiling itself. The durable companion fix is patch-style revise in Daedalus (separate PR) so tool inputs stay small. - If Mantle ignores the beta header (i.e. its truncation behaviour is native), Shim 4 still converts the failure into a graceful model-side stop; Shim 3 is then inert but harmless. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
r added 1 commit 2026-08-04 20:49:45 +00:00
🐾 fix(mantle): survive Mantle's 20k output ceiling on tool-heavy turns b5a3aa214b 
Two new Mantle shims, auto-installed alongside the existing pair:

- Opt out of the fine-grained-tool-streaming beta. Under it, an
  output-token cutoff mid-tool_use ends the stream without
  content_block_stop; fast-agent raises "Streaming completed but tool
  call never finished" and burns its retry ladder against the same
  wall (the observed ~700s Alan revise_workspace_file failures on
  Taurus).

- Clamp default maxTokens to Mantle's observed 20 000-token server
  ceiling, so the model stops gracefully (proper block close +
  stop_reason=max_tokens) instead of being cut off by the gateway.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
r merged commit 537f3c7963 into main 2026-08-04 20:54:12 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
r
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: r/pallas#8
Delete Branch "feature/mantle-max-tokens-shim"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?