# Hold Slayer 🔥 **An AI-powered telephony gateway that calls companies, navigates IVR menus, waits on hold, and transfers you when a human picks up.** You give it a phone number and an intent ("dispute a charge on my December statement"). It dials the number through your SIP trunk, navigates the phone tree, sits through the hold music, and rings your desk phone the instant a live person answers. You never hear Vivaldi again. > [!CAUTION] > **Emergency calling — 911** > Outbound calls to emergency numbers (`911`, `9911`, `112`) via the > REST API or MCP tools are **always refused** — an AI agent must never > place an emergency call, and API calls carry no E911 location data. > Do not rely on this system as any part of your means of reaching > emergency services; keep a phone with provider-registered E911 > service available. ## Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ FastAPI Server │ │ │ │ ┌──────────┐ ┌──────────┐ ┌───────────┐ ┌──────────────┐ │ │ │ REST API │ │WebSocket │ │MCP Server │ │ Dashboard │ │ │ │ /api/v1/*│ │ /ws/* │ │ (HTTP) │ │ / │ │ │ └────┬─────┘ └────┬─────┘ └─────┬─────┘ └──────────────┘ │ │ │ │ │ │ │ ┌────┴──────────────┴──────────────┴────┐ │ │ │ Event Bus │ │ │ │ (asyncio Queue pub/sub per client) │ │ │ └────┬──────────────┬──────────────┬────┘ │ │ │ │ │ │ │ ┌────┴─────┐ ┌─────┴─────┐ ┌────┴──────────┐ │ │ │ Call │ │ Hold │ │ Services │ │ │ │ Manager │ │ Slayer │ │ (LLM, STT, │ │ │ │ │ │ │ │ Recording, │ │ │ │ │ │ │ │ Analytics, │ │ │ │ │ │ │ │ Notify) │ │ │ └────┬─────┘ └─────┬─────┘ └──────────────┘ │ │ │ │ │ │ ┌────┴──────────────┴───────────────────┐ │ │ │ Sippy B2BUA Engine │ │ │ │ (SIP calls, DTMF, conference bridge) │ │ │ └────┬──────────────────────────────────┘ │ │ │ │ └───────┼─────────────────────────────────────────────────────────┘ │ ┌────┴────┐ │SIP Trunk│ ──→ PSTN └─────────┘ ``` ## What's Implemented ### Core Engine - **Sippy B2BUA Engine** (`core/sippy_engine.py`) — SIP call control, DTMF, bridging, conference, trunk registration. Signalling only: no audio reaches the classifier on this path - **PJSUA2 SIP Engine** (`core/pjsua_engine.py`) — Places the call itself so it owns the dialog, which is the only way PJSUA2 will surface RTP. Select with `SIP_ENGINE=pjsua2`; see [docs/architecture.md](docs/architecture.md#media-plane-why-pjsua2-places-the-call) - **PJSUA2 Media Pipeline** (`core/media_pipeline.py`) — Audio routing, capture ports, conference bridge, WAV playback (stub mode until the `pjsua2` bindings are installed — see note below) - **Call Manager** (`core/call_manager.py`) — Active call state tracking, lifecycle management - **Event Bus** (`core/event_bus.py`) — Async pub/sub with per-subscriber queues, type filtering, history ### Hold Slayer - **IVR Navigation** (`services/hold_slayer.py`) — Follows stored call flows step-by-step through phone menus, including SPEAK steps that synthesize speech via TTS - **Audio Classifier** (`services/audio_classifier.py`) — Real-time waveform analysis: silence, tones, DTMF, music, speech detection - **Call Flow Learner** (`services/call_flow_learner.py`) — Builds reusable call flows from exploration data, merges new discoveries - **LLM Fallback** — When a LISTEN step has no hardcoded DTMF, the LLM analyzes the transcript and picks the right menu option ### AI Receptionist & Smart Routing - **AI Receptionist** (`services/receptionist.py`) — Answers inbound calls, greets via TTS, captures the caller's intent with STT + LLM, then routes to a device or takes a voicemail - **Smart Routing** (`services/routing.py`) — Caller-pattern (glob), DNIS, time-of-day (with tz + midnight wrap), per-device DND, and ring-chain priority. Rules win over the LLM on conflict. - **TTS** (`services/tts.py`) — [Rhema](https://github.com/heluca/rhema) (OpenAI-compatible `/v1/audio/speech`) — synthesizes Kokoro voices for the SPEAK step and receptionist prompts ### Intelligence Layer - **LLM Client** (`services/llm_client.py`) — OpenAI-compatible API client (Ollama, vLLM, LM Studio, OpenAI) with JSON parsing, retry, stats - **Transcription** (`services/transcription.py`) — Speaches/Whisper STT integration for live call transcription - **Recording** (`services/recording.py`) — WAV recording with date-organized storage, dual-channel support, persisted to the `recordings` table - **Call Persistence** (`services/call_persistence.py`) — Writes completed calls + transcript chunks to the database on hangup - **Notifications** (`services/notification.py`) — WebSocket + SMS alerts for human detection, call failures, hold status ### API Surface - **REST API** — Call management, call history, transcripts, recordings, routing rules, device DND, call flow CRUD - **WebSocket** — Real-time call events, transcripts, classification updates, receptionist state transitions - **MCP Server** — 15 tools + 3 resources for AI assistant integration (make calls, send DTMF, get transcripts, manage flows), served over streamable HTTP at `/mcp/` - **Dashboard** — SvelteKit UI served at `/` with live monitor, call history with transcript playback, and a routing-rules editor ### Data Models - **Call** — Active call state with classification history, transcript chunks, hold time tracking - **Call Flow** — Stored IVR trees with steps (DTMF, LISTEN, HOLD, TRANSFER, SPEAK) - **Routing Rule** — Match (caller pattern, DNIS, time range) + action (ring_device, ring_chain, take_message, reject, dnd) - **Transcript Chunk** — Per-call STT segments with speaker tag and timestamp offset (for click-to-seek playback) - **Recording** — WAV file metadata (path, duration, size) per call - **Events** — 30+ typed events (call lifecycle, hold slayer, audio, device, system, receptionist, routing) - **Device** — SIP phone/softphone registration, priority, DND - **Contact** — Phone number management with routing preferences ## Project Structure ``` hold-slayer/ ├── main.py # FastAPI app + lifespan (service wiring) ├── config.py # Pydantic settings from .env ├── core/ │ ├── gateway.py # Top-level gateway orchestrator │ ├── dial_plan.py # Emergency-number guard + number normalisation │ ├── sip_engine.py # SIPEngine ABC + MockSIPEngine │ ├── sippy_engine.py # Sippy B2BUA SIP engine (signalling only) │ ├── pjsua_engine.py # PJSUA2 SIP engine (call control + media) │ ├── media_pipeline.py # PJSUA2 audio routing │ ├── logging_config.py # Text/JSON log formatting │ ├── rate_limit.py # Fixed-window limiter for the /auth/* edge │ ├── call_manager.py # Active call state management │ └── event_bus.py # Async pub/sub event bus ├── services/ │ ├── hold_slayer.py # IVR navigation + hold detection + SPEAK │ ├── receptionist.py # AI Receptionist state machine │ ├── routing.py # Smart routing (rules, DND, ring chain) │ ├── tts.py # Rhema TTS client (OpenAI-compatible) │ ├── audio_classifier.py # Waveform analysis (music/speech/DTMF) │ ├── call_flow_learner.py # Auto-learns IVR trees from calls │ ├── call_persistence.py # Writes calls + transcript chunks on hangup │ ├── llm_client.py # OpenAI-compatible LLM client │ ├── transcription.py # Speaches/Whisper STT │ ├── recording.py # Call recording management │ └── notification.py # WebSocket + SMS notifications ├── api/ │ ├── calls.py # Call management endpoints │ ├── call_history.py # History, transcript, recording playback │ ├── call_flows.py # Call flow CRUD │ ├── devices.py # Device registration │ ├── routing.py # Routing rules CRUD + per-device DND │ ├── websocket.py # Real-time event stream │ └── deps.py # FastAPI dependency injection ├── dashboard/ # SvelteKit UI (built to dashboard/build) │ └── src/routes/ │ ├── +page.svelte # Live monitor │ ├── history/ # Call history list │ ├── calls/[call_id]/ # Detail page + transcript playback │ └── routing/ # Rules editor + DND toggles ├── mcp_server/ │ └── server.py # MCP tools + resources (15 tools) ├── models/ │ ├── call.py # Call state models │ ├── call_flow.py # IVR tree models │ ├── routing.py # Routing rule / match / action models │ ├── events.py # Event type definitions │ ├── device.py # Device models │ └── contact.py # Contact models ├── db/ │ └── database.py # SQLAlchemy async (PostgreSQL + Alembic) └── tests/ ├── test_audio_classifier.py # 18 tests — waveform analysis ├── test_call_flows.py # 10 tests — call flow models ├── test_hold_slayer.py # 20 tests — IVR nav, EventBus, CallManager ├── test_services.py # 27 tests — LLM, notifications, recording, │ # analytics, learner, EventBus ├── test_tts.py # 4 tests — Rhema TTS client ├── test_routing.py # 8 tests — rules evaluator └── test_receptionist.py # 7 tests — receptionist decision logic ``` ## Quick Start ### 1. Install ```bash python -m venv .venv source .venv/bin/activate pip install -e ".[dev]" ``` > [!NOTE] > The PJSUA2 media pipeline needs the `pjsua2` Python bindings, which are > **not pip-installable** — they're built from pjproject (`./configure && > make && make install` with `--enable-shared` and the Python SWIG target). > Without them the media layer runs in stub mode (signaling only): audio > routing, recording and playback become no-ops that still return success. > > **See [docs/pjsua2-build.md](docs/pjsua2-build.md)** for the verified > procedure (pjproject 2.17, no `sudo` required). It includes the `patchelf` > RPATH step, without which the bindings compile and install but fail to import. ### 2. Configure ```bash cp .env.example .env # Edit .env with your SIP trunk credentials, LLM endpoint, etc. # Required: DATABASE_URL, plus either the Casdoor SSO settings # (CASDOOR_* + OWNER_NAME) or CASDOOR_ENABLED=false with HOST=127.0.0.1. ``` The gateway is **owner-only**. The browser dashboard signs in via **Casdoor SSO** (short-lived JWT); MCP and CLI clients use a **Personal Access Token** (`hs_pat_…`) minted from the dashboard's *API Tokens* menu. Both are presented as `Authorization: Bearer ` (WebSocket and `