pallas/docs/pallas.md

Pallas — Technical Reference

Pallas is the generic runtime that turns fast-agent agent definitions into StreamableHTTP MCP servers. It is completely deployment-agnostic: all environment-specific values (agent names, ports, hosts, model) live in the calling project's configuration files, not in Pallas itself.

---

Solution Architecture

Pallas occupies the middle tier of a three-layer MCP architecture. It bridges a web-facing client (Daedalus) and a constellation of specialised downstream MCP servers.

┌──────────────────────────────────┐
│  Daedalus                        │  Web UI / FastAPI / MCP client
│  Workspace management, chat,     │  Discovers agents via registry
│  health monitoring, progress     │  Calls agent tools via MCP
└──────────┬───────────────────────┘
           │ MCP over Streamable HTTP
           ▼
┌──────────────────────────────────┐
│  Pallas (FastAgent MCP Bridge)   │  Python runtime
│                                  │
│  ┌─ Registry  (port N)          │  GET /.well-known/mcp/server.json
│  ├─ Agent: Research  (port N+1) │  Chains, routers, sub-agents
│  ├─ Agent: Engineering (port N+2)│  Orchestrators, tool pipelines
│  └─ Agent: Orchestrator (N+3)   │  Delegates across agents
│                                  │
│  Each agent exposes:             │
│    • send_message tool           │
│    • get_health tool             │
│    • {agent}_history prompt      │
└──────────┬───────────────────────┘
           │ MCP over Streamable HTTP
           ▼
┌──────────────────────────────────┐
│  Downstream MCP Servers          │
│                                  │
│  Argos        — web search       │
│  Neo4j        — knowledge graph  │
│  Mnemosyne    — content library  │
│  Kernos       — shell execution  │
│  Gitea        — repository mgmt  │
│  Grafana      — monitoring       │
│  Rommie       — system management│
└──────────────────────────────────┘

Daedalus → Pallas

Interaction	Mechanism
Agent discovery	GET {registry}/.well-known/mcp/server.json — plain HTTP, returns all agents with MCP endpoint URLs
Agent communication	MCP tools/call on send_message — text + optional images
Health monitoring	MCP tools/call on get_health — programmatic, no LLM invocation
Progress feedback	MCP notifications/progress — streamed over SSE during long-running tool calls
Conversation history	MCP prompts/get on {agent}_history — retrieves stored message history

Pallas → Downstream

Pallas agents call downstream MCP servers via standard MCP tool calls. Each agent declares its servers in its fast-agent definition (servers=["argos", "neo4j_cypher", ...]). The server URLs and auth headers are configured in the consuming project's fastagent.config.yaml.

Mnemosyne's Role

Mnemosyne provides a content-type-aware knowledge graph with hybrid search (vector + full-text + graph). Agents with mnemosyne in their servers list gain access to tools for searching documents, browsing libraries and collections, retrieving items, and traversing the concept graph. It complements Neo4j (graph topology and relationships) with content-focused retrieval and re-ranking.

Why MCP End-to-End

Pallas is the protocol boundary — MCP above (from Daedalus) and MCP below (to downstream servers). This eliminates any MCP→REST→MCP translation layer. A single fast.start_server(transport="http") call exposes a complete agent as a StreamableHTTP MCP endpoint, giving Daedalus:

• Tool discovery via session.list_tools()
• Native streaming via MCP Streamable HTTP / SSE
• Health checks as ordinary tool calls — no separate API surface
• Progress notifications built into the protocol

---

Pallas Internal Architecture

Pallas is four modules, composed at startup:

server.py main()
  │
  ├─ _load_deployment_config()         parse agents.yaml
  ├─ _build_agents_table()             {name: (module, port)}
  ├─ _build_agent_deps()               dependency graph
  │
  ├─ _start_all()  or  _run_single()
  │    │
  │    ├─ _preflight()
  │    │    ├─ _register_unknown_models()   model registration
  │    │    └─ validate_llm_providers()     LLM API key + model checks
  │    │
  │    ├─ start subagents (depends_on)
  │    ├─ wait for subagent readiness
  │    ├─ start top-level agents
  │    │    │
  │    │    └─ _start_agent(name)
  │    │         ├─ import agent module
  │    │         ├─ MultimodalAgentMCPServer(...)
  │    │         ├─ _resolve_downstream_servers()
  │    │         ├─ _preflight_mcp_servers()     warn on missing auth
  │    │         ├─ register_health_tool()
  │    │         └─ server.run_async()
  │    │
  │    └─ run_registry()               Starlette app on registry port
  │
  └─ asyncio.run(...)

Module	Purpose
pallas.server	CLI entry point, configuration loading, agent lifecycle orchestration, model registration
pallas.registry	Starlette app serving GET /.well-known/mcp/server.json — builds the agent catalogue from agents.yaml + fastagent.config.yaml
pallas.multimodal_server	MultimodalAgentMCPServer — AgentMCPServer subclass adding image attachment support and conversation history prompts
pallas.health	Two-layer health: startup LLM preflight validation + runtime get_health MCP tool with downstream server probing

---

Installation

pip install git+ssh://git@git.helu.ca:22022/r/pallas.git

Or as a project dependency:

dependencies = [
    "pallas-mcp @ git+ssh://git@git.helu.ca:22022/r/pallas.git",
]

Requires Python ≥ 3.13. Key dependencies: fast-agent-mcp, httpx, pyyaml, starlette, uvicorn.

---

Project Layout

Pallas reads configuration from the working directory at runtime. A consuming project looks like:

my-project/
├── agents/
│   ├── __init__.py
│   └── jarvis.py              # FastAgent definitions
├── agents.yaml                # Deployment topology
├── fastagent.config.yaml      # FastAgent + model config
├── fastagent.secrets.yaml     # API keys (gitignored)
└── .env                       # Secret values (gitignored)

Pallas itself contains no agent definitions, model names, ports, or hostnames. Everything is injected by the consuming project.

---

Configuration Reference

agents.yaml

Single source of truth for deployment topology.

name: my-project               # log prefixes and registry names
version: "1.0.0"               # published in registry entries
host: my-host.example.com      # hostname for registry URLs
namespace: com.example.project  # reverse-domain prefix for registry names
registry_port: 8200             # port for the registry server

agents:
  jarvis:
    module: agents.jarvis       # importable Python module path
    port: 8201                  # StreamableHTTP port for this agent
    title: Jarvis               # human-readable name (registry)
    description: "My assistant" # one-line description (registry)
    depends_on: [research]      # optional: start these agents first

  research:
    module: agents.research
    port: 8250
    title: Research Agent
    description: "Web search and knowledge graph"

Field	Required	Description
name	yes	Project name — used in log prefixes ([my-project]) and CLI help
version	no	Semver string published in registry entries. Default: "1.0.0"
host	no	Hostname used in registry remotes[].url. Default: "localhost"
namespace	no	Reverse-domain prefix for registry server.name (e.g. com.example/jarvis)
registry_port	no	Port for the registry server. Default: 24200
agents.<name>.module	yes	Importable Python module path containing a fast instance
agents.<name>.port	yes	Port for this agent's StreamableHTTP MCP server
agents.<name>.title	no	Display name in registry. Default: name.title()
agents.<name>.description	no	Description in registry
agents.<name>.depends_on	no	List of agent names that must start and become ready before this agent

fastagent.config.yaml Extensions

Pallas reads two keys beyond the standard fast-agent config:

default_model: openai.my-model-name

model_capabilities:
  vision: false
  context_window: 200000
  max_output_tokens: 32000

Key	Description
default_model	provider.model-name format. The provider prefix (anthropic or openai) determines which LLM provider is active for health checks.
model_capabilities.vision	true registers the model with multimodal tokenization; false registers as text-only. Default: false
model_capabilities.context_window	Context window size in tokens. Default: 131072
model_capabilities.max_output_tokens	Max output token limit. Default: 16384

Capabilities are declared explicitly rather than inferred from model name — naming conventions vary across model families, making regex heuristics brittle. These values are both used to register unknown models with fast-agent's ModelDatabase and published in the registry response.

fastagent.secrets.yaml

anthropic:
  api_key: "${ANTHROPIC_API_KEY}"
openai:
  api_key: "${OPENAI_API_KEY}"
  base_url: "${OPENAI_BASE_URL}"

${ENV_VAR} placeholders are expanded at runtime from environment variables.

.env

Pallas loads .env from the working directory into os.environ without overwriting existing variables. This supports both local development and systemd deployments:

ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
OPENAI_BASE_URL=http://my-llm-server:8080/v1

OPENAI_BASE_URL defaults to https://api.openai.com/v1 if unset. For local llama-cpp, vLLM, or other OpenAI-compatible servers, set it to their endpoint.

Environment Variables

Variable	Default	Purpose
PALLAS_AGENTS_CONFIG	agents.yaml	Override path to deployment config

---

Running Pallas

CLI

pallas                     # start all agents + registry
pallas --agent jarvis      # start a single agent (no registry)
python -m pallas.server    # equivalent to `pallas`

Startup Sequence

All agents mode (pallas):

1. Load agents.yaml, build agents table and dependency graph
2. Preflight — register unknown models with ModelDatabase, validate LLM provider API keys and model availability
3. Start the registry server on registry_port
4. Start subagents (agents listed in other agents' depends_on)
5. Wait for each subagent to become ready (HTTP probe on /mcp, 60s timeout)
6. Start top-level agents (everything not a subagent)
7. All servers run concurrently via asyncio.gather

Single agent mode (pallas --agent <name>):

1. Load agents.yaml
2. Preflight
3. Start the named agent (no registry, no dependency resolution)

Per-Agent Startup

For each agent:

1. Import the agent module (agents.<name>) and obtain its fast instance
2. Enter fast.run() context — initialises the fast-agent runtime
3. Create a MultimodalAgentMCPServer wrapping the primary agent instance
4. Resolve downstream MCP server configs from the fast-agent configuration
5. Warn if any downstream auth headers reference unset environment variables
6. Register the get_health MCP tool with downstream server info
7. Bind to 0.0.0.0:<port> and serve StreamableHTTP

---

Daedalus Integration

This section describes the contract from Pallas's perspective. The full client-side specification is in docs/pallas_integration.md.

Registration Flow

1. Daedalus stores a registry URL (e.g. http://puck.incus:23030)
2. Fetches GET {url}/.well-known/mcp/server.json
3. Discovers all agents with their MCP endpoint URLs, titles, and descriptions
4. Creates connections to each agent

Health Polling

Daedalus calls get_health on each connected agent at a configurable interval (default 60s). The response maps to UI indicators:

status	Daedalus behaviour
ok	Green badge, normal operation
degraded	Yellow badge + warning banner showing message. Chat allowed.
error	Red badge. Chat disabled.

Progress Notifications

Long-running agent tool calls (agentic loops, sub-agent delegation) emit MCP notifications/progress on the SSE stream. Daedalus must include a progressToken in the _meta of tools/call requests to opt in:

result = await session.call_tool(
    "jarvis",
    arguments={"message": user_input},
    request_params={"_meta": {"progressToken": str(uuid4())}},
)

Progress notification fields:

Field	Description
progressToken	Matches the token sent in the request
progress	Monotonically increasing step counter
total	null = indeterminate (loop in progress), 1.0 = sub-task finished
message	Status text: {server}/{tool}: started|completed|failed or {agent} step N (llm|tool)

Without a progressToken, Pallas skips all progress notifications and the client receives nothing until the final result.

Chat Blocking

If the target agent's cached health is error, Daedalus returns HTTP 503 and disables the message input. degraded shows a warning but allows chat.

---

Registry Server

Endpoint

GET {host}:{registry_port}/.well-known/mcp/server.json

Plain HTTP — not MCP. No authentication. Returns application/json.

Response Structure

Built dynamically from agents.yaml + fastagent.config.yaml:

{
  "servers": [
    {
      "server": {
        "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
        "name": "com.example.project/jarvis",
        "title": "Jarvis",
        "description": "My assistant agent",
        "version": "1.0.0",
        "remotes": [
          { "type": "streamable-http", "url": "http://my-host.example.com:8201/mcp" }
        ],
        "capabilities": {
          "model": "my-model-name",
          "vision": false,
          "context_window": 200000,
          "max_output_tokens": 32000
        }
      },
      "_meta": {
        "io.modelcontextprotocol.registry/official": {
          "status": "active",
          "updatedAt": "2026-01-01T00:00:00Z",
          "isLatest": true
        }
      }
    }
  ]
}

Registry Name Construction

{namespace}/{slug} — where slug is the agent key with underscores replaced by hyphens. Example: namespace com.example.project + agent key tech_research → com.example.project/tech-research.

Capabilities

If model_capabilities is defined in fastagent.config.yaml, each registry entry includes a capabilities object with model name, vision support, context window, and max output tokens. This allows clients to make informed decisions about what an agent can handle.

---

Multimodal Support

MultimodalAgentMCPServer extends fast-agent's AgentMCPServer with image attachment support.

send_message Tool

Each agent's MCP tool accepts:

Parameter	Type	Required	Description
message	str	yes	Text message to the agent
images	list[dict]	no	Base64-encoded images: [{"data": "...", "mime_type": "image/png"}]

When images is provided, the message is sent as a PromptMessageExtended containing both TextContent and ImageContent parts — the agent's underlying model must support vision.

Conversation History Prompt

For agents with instance_scope != "request", a {agent}_history prompt is registered that returns the full conversation history as FastMCP Message objects. This allows clients to retrieve the stored context.

Bearer Token Propagation

The server captures the authenticated bearer token from the incoming MCP request's Authorization: Bearer … header via fastmcp.server.dependencies.get_http_request() (FastMCP's get_access_token() returns None because Pallas runs without the auth middleware). Two consumers read it:

• LLM-provider passthrough — the token is also pushed into the request_bearer_token ContextVar for the agent's LLM provider key manager to pick up automatically (used by HuggingFace and any other token-passthrough providers). The ContextVar works here because the LLM call runs in a child task of the request handler.
• Downstream MCP servers (opt-in) — outgoing MCP calls inherit the same bearer when the downstream server is marked forward_inbound_auth: true in fastagent.config.yaml. Without that flag, the inbound bearer is not forwarded to MCP transport calls — server_config.headers is the only header source.

The forwarding is per-server so a FastAgent attached to both a credentialed downstream (e.g. Mnemosyne) and an unrelated public server doesn't leak the bearer to the latter.

Why a simple ContextVar forward isn't enough

fast-agent's MCPConnectionManager runs each downstream transport inside a long-lived anyio.TaskGroup created at manager startup. TaskGroup.start_soon snapshots the owner's contextvars.Context at spawn time — the request-handler's context is invisible to the transport task. A straight request_bearer_token.get() inside _prepare_headers_and_auth therefore always resolves to None even when the inbound handler has set the token a few frames up. The persistent connection is additionally reused across requests, so the first-call context (often empty) would be cached forever.

Pallas works around this in pallas._fastagent_patch by maintaining a process-wide _pending_bearers registry keyed by id(server_config). multimodal_server.send_message calls publish_bearer(cfg, token) for every opted-in downstream the agent is allowed to reach; the patched _prepare_headers_and_auth looks it up there (with the ContextVar as a fallback for non-persistent probe paths); and the request handler's finally block calls revoke_bearer(cfg) to clear the entry. Per-request bearers therefore survive the task-group boundary without any mutation of shared config.

Example:

mcp:
  servers:
    mnemosyne:
      transport: http
      url: "https://mnemosyne.example/mcp/"
      forward_inbound_auth: true   # inbound bearer rides outbound
    weather:
      transport: http
      url: "https://weather.example/mcp/"
      # no flag → outbound calls go unauthenticated

When the agent receives a request with Authorization: Bearer X, mnemosyne will see Authorization: Bearer X on the outbound call; weather will see no Authorization header. If mnemosyne.headers.Authorization is set explicitly, that wins (the inbound bearer is not overwritten on top of an explicit header).

---

Health System

Two-layer health checking: startup preflight validates LLM providers before agents launch, and a runtime get_health tool reports ongoing status.

Startup Preflight

Runs once before any agents start. Validates all LLM providers that have API keys configured.

Provider	Active (default_model matches)	Key set, not active
Anthropic	GET /v1/models/{model} — confirms model exists and key is valid	GET /v1/models/claude-sonnet-4-5 — verifies API access
OpenAI	GET {base_url}/models — lists models, confirms configured model is present	GET {base_url}/models — lists available models

• Warn-only — never blocks startup. Agents start regardless.
• 5-second timeout per provider API call.
• Loads .env before checking.

Runtime get_health Tool

Registered on each agent's MCP server. Checks:

1. Downstream MCP servers — sends an MCP initialize handshake to each server URL. Uses initialize because it is the only MCP method that works without a pre-established session. After success, sends DELETE with the returned Mcp-Session-Id to tear down the session cleanly. 3-second timeout.

2. Active LLM provider — includes the preflight result for the provider that default_model points to. Only the active provider affects health status.

Response Format

{ "status": "ok", "timestamp": "2026-01-01T00:00:00Z" }

{
  "status": "degraded",
  "timestamp": "2026-01-01T00:00:00Z",
  "message": "Unreachable: neo4j_cypher; LLM: openai: model 'bad-model' not found"
}

Status	Meaning
ok	All downstream servers reachable and active LLM provider healthy
degraded	One or more downstream servers unreachable, or active LLM provider failed

---

Model Registration

Pallas registers models not in fast-agent's built-in ModelDatabase at startup, using the explicit capability declarations from fastagent.config.yaml.

The process:

1. Read default_model and model_capabilities from config
2. Extract the model name (portion after the provider prefix dot)
3. Check if ModelDatabase already knows this model — if so, skip
4. Register with ModelDatabase.register_runtime_model_params():
   • vision: true → multimodal tokenization (QWEN_MULTIMODAL)
   • vision: false → text-only tokenization (TEXT_ONLY)
   • context_window and max_output_tokens from config (with sensible defaults)

This avoids the brittle pattern of inferring capabilities from model name substrings, which breaks for custom or fine-tuned models with non-standard names.

---

Module Reference

Module	File	Purpose
pallas.server	server.py	CLI entry point (pallas command), configuration loading, agent lifecycle orchestration, dependency ordering, model registration
pallas.registry	registry.py	Starlette app serving GET /.well-known/mcp/server.json — agent catalogue built from config
pallas.multimodal_server	multimodal_server.py	MultimodalAgentMCPServer — extends AgentMCPServer with image support, conversation history prompts, bearer token propagation
pallas.health	health.py	LLM provider preflight validation, downstream MCP server probing, get_health tool registration
pallas.log	log.py	JSON log configuration, third-party traceback capture, Rich-TUI-safe handler attachment
pallas._fastagent_patch	_fastagent_patch.py	Monkey-patches fast-agent at import time: per-request bearer forwarding via httpx.Auth, diagnostic trace-capture wrappers around send_request / session.call_tool / _execute_on_server

---

Incidents & Lessons Learned

The Pallas↔Mnemosyne bearer-forwarding rollout surfaced a chain of bugs that ranged from "obvious in hindsight" to "you have to go read the fast-agent source to see why". None of the individual symptoms pointed at the true cause — each had a plausible scapegoat — which is why the actual fix was to install structured diagnostics first and work the problem end-to-end. This section captures the findings so the next person to touch this code (likely future me) does not have to re-derive them.

1. Per-request bearer across an anyio.TaskGroup boundary

Symptom. Per-turn JWTs minted by Daedalus and sent as Authorization: Bearer … to Pallas never reached Mnemosyne; Mnemosyne saw either no Authorization header at all, or — worse, intermittently — a bearer from a previous turn against an unrelated workspace.

Cause. fast-agent's MCPConnectionManager runs each downstream transport inside a long-lived anyio.TaskGroup created at manager startup. TaskGroup.start_soon snapshots the owner's contextvars.Context at spawn time, so any request_bearer_token.set(…) done in the request handler a few frames up is invisible to the transport task. The persistent connection additionally caches its handshake context — so the bearer observed on the first call (often empty during a health-probe-triggered warm-up) gets reused forever.

Why the first attempt didn't help. We initially set the bearer via a contextvars.ContextVar and tried to have _prepare_headers_and_auth read it. It almost works — until any reconnect, retry, or persistent stream, at which point the cached snapshot wins.

Fix (pallas._fastagent_patch). Maintain a process-wide _pending_bearers: dict[int, str] keyed by id(server_config), guarded by a threading.Lock. multimodal_server.send_message calls publish_bearer(cfg, token) for every opted-in downstream before spawning any tool call; the patched _prepare_headers_and_auth pulls the token from the registry (ContextVar used as a fallback for non-persistent probe paths); a finally in the request handler calls revoke_bearer(cfg) to clear the entry. Per-request bearers therefore survive the task-group boundary without mutating any shared config object.

Bonus gotcha. The opt-in was originally keyed off a custom forward_inbound_auth: true field on the server block, read via fast-agent's pydantic config model. Pydantic's nested-model validation silently dropped unknown keys, so the flag never appeared on the parsed config. Workaround: scan fastagent.config.yaml directly for the flag at module import time (pallas._fastagent_patch._FORWARD_SERVERS) rather than rely on the parsed config object.

Bonus gotcha 2. httpx caches auth handshake headers on persistent connections. A plain mutation of server_config.headers["Authorization"] in the request handler only affects new connections. The forwarding patch works by providing a custom httpx.Auth subclass (_DynamicBearerAuth) that looks up the bearer on every request, not by mutating headers — this is why the override is auth_flow (the generic non-async flow), not async_auth_flow.

2. install() idempotency shadowing newly-added patches

Symptom. After adding two new diagnostic monkey-patches (_patch_session_call_tool, _patch_execute_on_server) and reinstalling pallas-mcp into the Kottos venv, the trace-capture records refused to appear in pallas.log. Four repro cycles, five log rotations, no evidence that the new code was running.

Cause. install() had a single top-level guard on _prepare_headers_and_auth._pallas_forward_patched. Once the bearer-forwarding patch was applied on first import, every subsequent install() call returned early — skipping the three later _patch_*() helpers entirely. The patches were present in the installed file; they were never executed.

Lesson. A shared idempotency guard at the top of an install()-style function is a liability as soon as the function grows past one patch. The fix (commit 082b611) moves each patch's guard to a per-target sentinel attribute on the target method (target._pallas_trace_patched = True), checked inside each helper. install() now calls every helper unconditionally; duplicate installs are cheap and harmless.

Bonus gotcha. install() runs at module-import time, which in Pallas happens before pallas.log.setup_logging() attaches the file handler. Any logger.info("patch installed") inside install() is emitted into the default handler and lost. "No 'patch installed' line in the log" is not evidence that the patch didn't install — only the runtime firing of the wrapper (e.g. forward.applied …) is a reliable presence marker.

3. FastMCP on_call_tool context shape: message.name, not message.params.name

Symptom. Once bearer forwarding worked, Harper's Mnemosyne tool calls came back to fast-agent as the literal string "object NoneType can't be used in 'await' expression". The tool result was visible in the OpenAI request payload of the next turn as {"role":"tool", "content":"object NoneType can't be used in 'await' expression"}. No traceback anywhere in Pallas or Mnemosyne.

Cause. mnemosyne/mcp_server/auth.py:MCPAuthMiddleware._extract_tool_name read context.message.params.name, but inside an on_call_tool hook FastMCP's MiddlewareContext[CallToolRequestParams] exposes .name and .arguments directly on context.message — the type parameter is already the params object. The extractor always returned None, which:

• silently skipped the _PUBLIC_TOOLS = {"get_health"} bypass so even the public health probe went through JWT validation; and
• made the per-tool ACL token.can_use_tool(None) short-circuit.

The NoneType await error string itself came from somewhere downstream of the middleware — the middleware still unconditionally awaited call_next(context). The most likely path was await self._tools.get(None)(...) in the FastMCP dispatch (None lookup returns None, then await None(...) raises the TypeError).

Fix (mnemosyne commit e0fa825). Read context.message.name directly; fall back to message.params.name only as a legacy safety net. Verified against fastmcp's own Middleware.on_call_tool signature (MiddlewareContext[mt.CallToolRequestParams]) and four independent docs examples.

Diagnostic helper. The commit also added _call_next_with_trace around await call_next(context) so any future exception inside FastMCP dispatch is captured with a full logger.exception traceback before propagating — and so the success path logs the result type, which doubles as a canary for "the middleware actually ran".

4. Rich-TUI corruption by DEBUG-level third-party loggers

Symptom. fast-agent go in an interactive session was unusable: massive blobs of plain-text DEBUG:openai._base_client:Sending HTTP Request: … and DEBUG:sse_starlette.sse:chunk: … lines splattered over the Rich chat UI on every redraw.

Cause. Two layers stacked up:

• Pallas's original setup_logging() set the root logger to whatever logger.level was configured. With logger.level: debug in kottos/fastagent.config.yaml (set intentionally for Pallas diagnostics), every third-party library inherited DEBUG and started emitting.
• Pallas attached a StreamHandler(stream=sys.__stderr__) to both root and pallas loggers so DEBUG records would "survive Rich's console takeover". This did solve the Rich-swallowing problem, but swapped it for a worse one: every library's DEBUG record now bypassed the Rich Live display and leaked through every TUI repaint.

Fix (commits dde7d4f + 89870f4).

• PALLAS_LOG_STDERR env var gates the stderr handler. Off by default. Interactive users get a clean TUI + rotating file sink; systemd/journal deployments set PALLAS_LOG_STDERR=1.
• Root-logger level is decoupled from Pallas's own level. Default: max(configured_level, INFO). Pallas's pallas.* loggers still honour logger.level: debug, but third-party libraries stay at INFO unless PALLAS_ROOT_LOG_LEVEL=DEBUG is set explicitly.
• openai, openai._base_client, anthropic, anthropic._base_client, sse_starlette, sse_starlette.sse, mcp, mcp.client, mcp.server, httpx, httpcore pinned at WARNING individually — belt-and-braces against any future re-enablement of root DEBUG.

5. Logging configuration knobs (current state)

Env var / config	Default	Effect
PALLAS_LOG_LEVEL	INFO	Level for the pallas.* logger tree and the rotating file sink
fastagent.config.yaml logger.level	fallback for PALLAS_LOG_LEVEL	Unified knob — flipping fast-agent's level also flips Pallas's diagnostic level
PALLAS_ROOT_LOG_LEVEL	max(pallas_level, INFO)	Level for the root logger (controls third-party library output). Rarely needs to be changed.
PALLAS_LOG_STDERR	unset (off)	Attach a JSON StreamHandler to sys.__stderr__. Enable for systemd/journal; leave off in Rich TUI sessions.
PALLAS_LOG_FILE	~/.local/state/pallas/pallas.log	Rotating JSON log file. 10 MB × 5 backups.

The rotating file sink is always on. It's what catches tracebacks from fast-agent, fastmcp, the MCP SDK, and our own trace wrappers regardless of how Rich is interacting with the terminal. Tail with jq for structured access:

tail -n 100 -f ~/.local/state/pallas/pallas.log | jq -r '"\(.time) \(.level) \(.logger) \(.message)"'

When diagnosing a downstream-MCP issue, grep pallas.forward.trace in that file: any uncaught exception inside send_request, session.call_tool, or _execute_on_server appears there with full traceback, even when fast-agent's aggregator turns it into a terse CallToolResult(isError=True) by the time the agent loop sees it.