5c178bb7bdabd4995f6d3f18ea44513b9b985f4e
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 5c178bb7bd |
feat(auth): rate-limit the unauthenticated /auth/* edge
Blanket per-endpoint limits would have been the wrong shape here. Every REST/WS/MCP surface is owner-only — an unauthenticated request is rejected by resolve_bearer/is_owner before any handler runs — so limiting them would mostly throttle the single legitimate operator, and real spend control for outbound calls is already max_concurrent_calls in gateway.make_call. What is genuinely exposed is the handful of /auth/* routes that must answer before an identity exists. /auth/callback and /auth/refresh-callback each make an outbound token exchange with Casdoor on every request; /auth/me opens a DB session and runs a token lookup. All are free to trigger and none are cheap to serve. /auth/logout is left unlimited — it builds a redirect URL and does no I/O. Not a defence against credential guessing: PATs are secrets.token_urlsafe(32) (256 bits) compared by SHA-256 digest, so brute force was never the threat. This is about unauthenticated work an attacker controls. Fixed-window, in-process, no new dependency — one operator and one process make a shared counter store infrastructure without a purpose. The bucket store is bounded and evicts oldest-first, since an unbounded map keyed by source address would itself be the exhaustion vector. The limiter keys on the socket peer and deliberately ignores X-Forwarded-For. That header is attacker-controlled unless a trusted proxy overwrites it, and this app establishes no such trust; keying on it would let one client present as thousands and make the limiter worse than useless. Behind the estate's reverse proxy the limit is therefore per-proxy, not per-caller — correct for exhaustion and honest about what it can enforce. Per-caller limits need an explicit trusted-proxy config, noted in CLAUDE.md so it isn't added silently. Verified against a real server: exactly 30 requests pass, then 429 with Retry-After: 60, while an owner-gated route serves 40/40. The 429s appear in the JSON access log with queryable status_code and client_addr, so an attack is visible in Loki. The wiring test identifies the dependency by qualname rather than string search, and was mutation-checked by removing the limit from /auth/me. Also documents 401/403/429 in the API reference — 401 and 403 have existed since auth landed but were never in the status-code table. Phase 4 is now complete. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
|||
| 1644999bcb |
feat(logging): structured JSON logs, uvicorn access log included
Hold Slayer's logs are shipped to Loki by the host's Alloy agent, which reads container stdout. Text lines arrive there as an opaque blob: filtering on a status code meant regex over a formatted string. This adds LOG_FORMAT=json (default "text", so local dev stays readable) rendering one JSON object per line. Two parts were less obvious than a format= argument would suggest, and both are why this is a module rather than a basicConfig tweak: Uvicorn attaches its own handlers to `uvicorn` and `uvicorn.access` with propagate=False, so configuring only the root logger would have left the access log — the highest-volume, most useful stream — as colourised text next to our JSON. configure_logging clears those handlers and re-enables propagation, and is called both at import (for startup config checks) and in lifespan (uvicorn configures itself after importing the app). The __main__ path passes log_config=None so uvicorn never applies its own. The access record's payload lives in record.args as a 5-tuple, not in the message. Formatting it would throw the structure away and force Loki to parse it back out, so the tuple is unpacked into real fields and status_code is emitted as a number for range filtering. Also drops uvicorn's `color_message` extra, an ANSI-coloured duplicate of the message that generic extra-promotion would otherwise copy into every startup line — the same unreadable-in-Grafana problem recently fixed for the lab's Asterisk logs. Verified against a real uvicorn server: 39/39 lines valid JSON, zero ANSI escapes, no duplicates, access lines structured with correct status codes; text mode unchanged. Thread name is included off the main thread, since "which execution context logged this" is the first question when debugging across the asyncio/Sippy/PJSUA2 boundary. SecretStr extras stay masked. README Phase 4 item ticked; LOG_FORMAT and the previously-undocumented LOG_LEVEL added to the config table and .env.example. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
|||
| 2a05be27bf |
feat(sip): add PJSUA2 engine — audio finally reaches the classifier
The gateway can now hear. Verified end to end against the Asterisk lab: speech classifies as live_human, hold music as music, the speech→music transition tracks the dialplan, and DTMF reaches a real IVR (Asterisk logged "caller pressed 1 -> accounts" and branched). RTP stats show 0% packet loss. PJSUAEngine implements the existing SIPEngine interface, so the gateway, call manager and hold-slayer service are unchanged. It is selected with SIP_ENGINE=pjsua2; the default stays "sippy" while this is proven, and the mock remains opt-in as before. Why a new engine rather than fixing the old path: PJSUA2 exposes no standalone RTP media object, so it will not surface media for a dialog it does not own. Owning the dialog is the price of owning the media. Sippy keeps the SBC roles it is good at — device registration, routing, leg bridging — and SippyEngine remains fully functional for signalling; it simply cannot carry media, which its media branch now says plainly instead of calling a method that could never work. The safety invariants are untouched. This engine is reachable only through gateway.make_call, which refuses emergency numbers and enforces the concurrency cap before any SIP action. No new dial path was introduced. MediaPipeline.add_remote_stream(host, port) is replaced by attach_call_media(stream_id, audio_media), called from onCallMediaState — the one place PJSUA2 hands out RTP-backed media. Taps requested before media comes up are attached when it does, so the classifier never misses the start of a call. Three crash/lifetime bugs found by running against the real bindings, none of which any unit test would have caught: - pj.Call and pj.Account objects finalised after libDestroy() abort the process on a native assertion, exactly as media ports do. Both are now dropped and collected before the pipeline destroys the endpoint. - PJSUA2 keeps delivering callbacks during interpreter teardown, when module globals may already be cleared. The callbacks alias what they need locally and swallow everything: a raise there escapes into C++ and takes the worker thread with it. - hangup() only queues the BYE, so shutdown deleted the account with a call still active and left the far end on an unclosed dialog. stop() now waits briefly for the teardown to complete. Threading follows the established rule: PJSUA2 worker threads reach the loop only through _post_from_pj → run_coroutine_threadsafe, and any thread PJSUA2 did not create registers itself before touching a PJSUA2 object. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
|||
| 7979e70705 |
feat(media): implement PJSUA2 audio capture port; document the media-plane refactor
Implements the tap half of the media path and records why the other half requires moving call placement into PJSUA2. MediaPipeline.create_tap was a stub: it logged "🎤 Audio tap created" and returned a tap that nothing ever fed, so the classifier received no audio on a live call. It now builds a real pj.AudioMediaPort subclass whose onFrameReceived converts the SWIG ByteVector to PCM bytes and fans it out to every tap on the stream. One capture port per stream, shared by all taps: a second port on the same stream would be mixed back into the conference bridge and the call would echo. Thread safety is the constraint here. onFrameReceived runs on a PJSUA2 worker thread — a third execution context beside the asyncio loop and the Sippy ED thread — and touches nothing but AudioTap.feed, which hops to the owning loop via call_soon_threadsafe. An exception escaping into PJSUA2's C++ callback would tear down the worker thread and silently kill media for every call, so the handler catches and logs once per port rather than on every 20ms frame. Also fixes a hard crash found while testing this against real PJSUA2: a media port finalised after Endpoint.libDestroy() calls pjmedia_conf_remove_port against a freed conference bridge and aborts the process on a native assertion. Ports are now released in remove_stream while the bridge still exists, and stop() forces a collection before libDestroy — dropping the last Python reference is not sufficient on its own. Verified against the real bindings: frames fan out to multiple taps, cross the thread boundary intact, and shutdown is clean. add_remote_stream remains a stub, and deliberately so. PJSUA2 exposes no standalone RTP media object — every AudioMedia subclass in the Python bindings is a file player, recorder, tone generator or capture port, and RTP is reachable only via pj.Call.getAudioMedia() on a dialog PJSUA2 itself owns. A design where Sippy owns the dialog can never obtain media from PJSUA2, so that function cannot be written against this API. docs/architecture.md now explains this and records the resolution: PJSUA2 places the trunk call while Sippy keeps the SBC roles (device registration, routing, leg bridging), with the emergency guard and concurrency cap staying first in gateway.make_call regardless of which library dials. The architecture doc also had drift unrelated to media: it described the thread boundary as asyncio.run_in_executor() when the real mechanism is run_coroutine_threadsafe / ED2.callFromThread, claimed two execution contexts where there are three, and cited MediaPipeline.add_stream() and SippyEngine.bridge() — neither of which exists. Corrected, with the data flow now showing the emergency guard and concurrency cap in their real positions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
|||
| 204203e3b0 |
fix(sip): correct five Sippy API mismatches that broke every real call
SippyEngine had never successfully placed a call or registered a trunk. Every failure was masked by broad exception handlers that logged and marked the leg terminated, so the gateway reported "ringing" and then ended the call rather than surfacing the fault. None of it was visible to the test suite, which runs exclusively on MockSIPEngine. Found by pointing the gateway at a local Asterisk instance (tests/lab) — each fix uncovered the next. 1. Trunk registration used kwargs the installed sippy (2.3.0) does not accept (auth_name/auth_password → user/passw), called register() instead of doregister(), and passed aor/contact as strings where SipRegistrationAgent calls .getCopy() and mutates .username/.port, so SipURL objects are required. It also posted registered=True at *send* time. Registration is asynchronous, so a rejected REGISTER would still have reported success — and /health treats a registered trunk as a condition for "healthy". Now wired to sippy's rok_cb/rfail_cb, so the rejection status line (typically a bad trunk password) reaches the operator. The Contact also fell back to loopback when the SIP bind is 0.0.0.0; a wildcard address is not somewhere a trunk can send an INVITE. 2. The INVITE passed SDP as a `body` kwarg. CCEventTry takes no such argument: UacStateIdle unpacks exactly six fields from the data tuple and expects the SDP as a MsgBody in position four. callingID/calledID are bare usernames — sippy builds the URIs itself from nh_address. 3. _sip_logger was absent from the global config. SipTransactionManager dereferences it on every message, so the first SIP packet in either direction raised KeyError inside the ED thread. 4. SippyCallController was not callable. Sippy invokes event_cb(event, ua) with CCEvent objects; the class only exposed on_* methods that nothing called. Added __call__ to dispatch CCEventRing/Connect/Disconnect/Fail to the existing handlers, guarding the body because an exception escaping into the ED dispatcher would hang the leg silently. 5. The UA was constructed without credentials, so sippy could not answer the 401/407 challenge that any authenticating trunk sends. Every outbound call died on the challenge. Verified end to end against Asterisk 22.10.1: 180 Ringing → Connected → 23s of audio → clean teardown, with the dialplan executing and audio playing in real time. Not fixed here, and still blocking media: MediaPipeline.create_tap is a stub that logs success and returns a tap nothing ever feeds, so the classifier receives no audio on a live call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
|||
| f7a11f2f20 |
Stage 5: Alembic migrations, durable call rows, one transcript truth
Alembic replaces create_all as the schema authority: async env.py against Base.metadata (CLI and in-app entry paths share it via config.attributes["connection"]), an autogenerated baseline of the create_all-era schema, and init_db now runs upgrade head — stamping the baseline first on a pre-Alembic database so existing deployments adopt cleanly. create_all remains for tests only. Calls are durable from the start: CallManager gains an on_call_created hook (wired to persist_call_on_create) that inserts an in_progress CallRecord the moment a call is created; persist_call_on_end finalizes that same row. A SIGKILL mid-call now leaves an in_progress row instead of erasing the call from history (verified live against the dev database). One transcript representation: ActiveCall.transcript_chunks holds TranscriptEntry (t_offset_ms, speaker, text) — add_transcript stamps real offsets from connect time, receptionist passes speaker instead of encoding it into "caller: ..." strings, persisted chunks carry real seek offsets, and the dead CallRecord.transcript Text column is dropped by migration. Device.is_online migrates String → Boolean (with a USING cast for existing rows). Model de-triplication: CallResponse/CallStatusResponse build via from_call classmethods (one ActiveCall→response mapping); DeviceStatus deleted — can_receive_call is a computed field on Device and the list endpoint returns the domain model; all row↔dict and row↔domain mapping now lives in call_persistence.py (record_summary/record_detail/chunk_to_dict + device row functions). New tests/test_data_layer.py: upgrade-head-matches-models, pre-Alembic adoption, durable in_progress rows, end-without-create fallback, transcript offsets, consolidated response models. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
|||
| 4048ce1db6 |
Stage 4: honest health, explicit error policy, event-bus integrity
Engine mode is now explicit: USE_MOCK_SIP=true is the only way to get the mock engine; an unconfigured trunk fails startup with guidance instead of silently degrading. Root-caused why the engine always ran mock: nested pydantic-settings never read .env (no env_file on the sub-settings classes) — all 8 now declare it. /health stops lying: reports engine mode (sippy|mock), a live DB SELECT 1, trunk registration state with reason, and TTS/STT availability from their last real request; "healthy" now requires ready + db + sippy + registered trunk. Error policy: leaf services (tts/transcription/llm_client) raise and track availability; call-loop callers catch, publish EventType.ERROR naming the failed service, and apply an explicit fallback. Persistence writes get one bounded 3x exponential retry, then an ERROR log — no more silent data loss. Event bus: a full subscriber queue drops its oldest event (counted) instead of silently evicting the subscription; subscribe(replay_last=N) delivers the advertised history replay, used by /ws/events (25). Receptionist correctness: a matched TAKE_MESSAGE rule beats the LLM; voicemail polls for early hangup and stops/transcribes/hangs up in finally; RecordingSession finally keeps its leg_ids so taps detach. Dead code removed: models/contact.py + Contact table, dtmf_buffer, transcribe_stream stub, SMS stub in notification.py. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
|||
| 67a00defc3 |
refactor: composition root in lifespan, break core↔services cycle, shared data layer
The gateway was the composition root, device registry, inbound-call policy, and call-operations service in one class, with core↔services circular imports papered over by function-local imports, wiring done by assigning private attributes, and MCP tools duplicating REST query logic against their own sessions. Composition: - main.py's lifespan now builds every service and wires them by constructor/registration. gateway.from_config() is gone; core/ no longer imports services/ anywhere — the cycle is dead. - Inbound-call policy moved to ReceptionistService.on_inbound_call (routing evaluation, reject/answer, screening dispatch); wired as the engine's on_incoming_call by the lifespan. Receptionist deps (tts/transcription/recording/routing) are constructor-injected — no more gateway._tts reach-through or importing hold_slayer's private _get_llm (now services.llm_client.get_llm, shared). - Hold-slayer launch goes through a mode-handler registry (register_mode_handler); the gateway no longer knows the service's type. CallManager takes on_call_ended in its constructor. - build_sip_engine() is a pure function taking explicit callbacks. - api/routing.py uses the routing service from app.state via a proper dependency instead of gateway._routing. Shared data layer: - db.session_scope() is the one session convention (get_db wraps it). - services/call_persistence.py gains the query/write functions and the single StoredCallFlow→CallFlow mapper; api/call_flows.py, api/call_history.py, and the six DB-touching MCP tools are thin wrappers over them — the two surfaces can't drift. - legs_for_call() replaces the three private _call_legs scans (gateway transfer/hangup, REST dtmf, MCP dtmf). 7 new tests (mode-handler launch, on_call_ended hook, receptionist inbound answer/reject, call-flow CRUD round-trip and history routes against real SQLite through the shared layer). aiosqlite added to dev deps for that. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
|||
| 5880b59872 |
fix: enforce thread ownership at the Sippy/PJSUA2 boundary
Three thread domains were mutating shared dicts with no locks: Sippy's ED thread wrote _legs/_registered_devices directly from SIP handlers, the asyncio loop wrote them from make_call/hangup, and run_in_executor(None, ...) had default-pool threads driving sippy UA objects. AudioTap.feed() pushed into an asyncio.Queue (not thread-safe) from the PJSUA2 thread. New ownership rule, enforced structurally: - The asyncio loop owns all app-visible state; the only mutator is the new _on_engine_event funnel. Sippy handlers extract plain strings on the ED thread and post via run_coroutine_threadsafe. - The ED thread owns sippy objects plus _ed_ua_to_leg/_ed_leg_to_ua; loop-side commands (INVITE/BYE/DTMF/trunk register) hop over via ED2.callFromThread. UA references no longer live on SipCallLeg. - AudioTap captures its loop and feed() hops via call_soon_threadsafe. - Fix ED import: installed sippy 2.x exposes ED2, not ED — the old import could never start the event loop. Also: - Wire the never-connected on_leg_state_change callback: outbound ringing/connected/terminated now reaches CallManager; a call ends when its last leg terminates (transfers keep it alive). Adds CallManager.unmap_leg/legs_for_call. - AudioClassifier.classify(): async entry that runs the FFT work in asyncio.to_thread and updates history on the loop — all four hold_slayer call sites now route through it, fixing both the loop-blocking and the 2-of-4 history gap. DTMF Goertzel loop replaced by the equivalent vectorized DFT-bin power. - Task hygiene: gateway.spawn() tracks hold-slayer/receptionist tasks and stop() cancels them; recording safety-timeout task is retained and cancelled on stop_recording; engine tracks incoming-call dispatch tasks. 10 new tests: funnel events from a foreign thread, auto-answer fallback, AudioTap cross-thread feed, classifier history, leg-state → call status (including no stomping of ON_HOLD), stop() cancellation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
|||
| 94fb6cd79d |
feat: mount MCP server, add bearer auth, and guard outbound calls
The MCP server was created but never mounted — no client could reach
it. Mount it at /mcp/ over streamable HTTP with a combined lifespan,
resolving the gateway lazily so mounting happens at app construction.
Security and safety for the agent surface:
- One static API_TOKEN (SecretStr) enforced across REST (dependency),
WebSocket (query param/header before accept), and MCP
(StaticTokenVerifier). Startup refuses tokenless non-loopback binds.
- Emergency numbers (911/9911/112) always refused on make_call, plus a
MAX_CONCURRENT_CALLS cap; ValueError surfaces as 400/ToolError.
- Safe defaults: debug off, no credential in default DATABASE_URL,
SIP/LLM/TTS secrets as SecretStr.
Cleanups:
- Delete broken learn_call_flow tool (wrong ctor args, nonexistent
method) and the never-fed CallAnalytics service; keep
call_flow_learner for proper wiring later.
- Trim dial_plan to what is actually used (emergency guard, extension
allocation); delete the unreferenced matcher/normaliser.
- Register call_history before calls so /api/calls/history is no
longer shadowed by /api/calls/{call_id}.
- fastmcp pinned >=3.0 (http_app + StaticTokenVerifier).
New tests: MCP in-memory client (tool surface, lazy gateway, emergency
refusal, call cap) and API security (401 paths, route order, mount).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|||
| 63f1a270bb |
feat: add call history API endpoints and TTS service client
Adds read-only access to persisted call records for the dashboard and implements a client for the Rhema text-to-speech service. - api/call_history.py: New router providing paged call lists and detailed call records with transcript metadata. - services/tts.py: Async client for OpenAI-compatible TTS endpoints (Rhema/Kokoro) used for call-flow steps. |
|||
| ecf37658ce |
feat: add initial Hold Slayer AI telephony gateway implementation
Complete project scaffolding and core implementation of an AI-powered telephony system that calls companies, navigates IVR menus, waits on hold, and transfers to the user when a human answers. Key components: - FastAPI server with REST API, WebSocket, and MCP (SSE) interfaces - SIP/VoIP call management via PJSUA2 with RTP audio streaming - LLM-powered IVR navigation using OpenAI/Anthropic with tool calling - Hold detection service combining audio analysis and silence detection - Real-time STT (Whisper/Deepgram) and TTS (OpenAI/Piper) pipelines - Call recording with per-channel and mixed audio capture - Event bus (asyncio pub/sub) for real-time client updates - Web dashboard with live call monitoring - SQLite persistence via SQLAlchemy with call history and analytics - Notification support (email, SMS, webhook, desktop) - Docker Compose deployment with Opal VoIP and Opal Media containers - Comprehensive test suite with unit, integration, and E2E tests - Simplified .gitignore and full project documentation in README |
