hold-slayer/README.md

Hold Slayer 🔥

An AI-powered telephony gateway that calls companies, navigates IVR menus, waits on hold, and transfers you when a human picks up.

You give it a phone number and an intent ("dispute a charge on my December statement"). It dials the number through your SIP trunk, navigates the phone tree, sits through the hold music, and rings your desk phone the instant a live person answers. You never hear Vivaldi again.

Caution

Emergency calling — 911 Outbound calls to emergency numbers (911, 9911, 112) via the REST API or MCP tools are always refused — an AI agent must never place an emergency call, and API calls carry no E911 location data. Do not rely on this system as any part of your means of reaching emergency services; keep a phone with provider-registered E911 service available.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        FastAPI Server                           │
│                                                                 │
│  ┌──────────┐  ┌──────────┐  ┌───────────┐  ┌──────────────┐  │
│  │ REST API │  │WebSocket │  │MCP Server │  │  Dashboard   │  │
│  │ /api/*   │  │ /ws/*    │  │ (SSE)     │  │  /dashboard  │  │
│  └────┬─────┘  └────┬─────┘  └─────┬─────┘  └──────────────┘  │
│       │              │              │                            │
│  ┌────┴──────────────┴──────────────┴────┐                     │
│  │             Event Bus                  │                     │
│  │   (asyncio Queue pub/sub per client)   │                     │
│  └────┬──────────────┬──────────────┬────┘                     │
│       │              │              │                            │
│  ┌────┴─────┐  ┌─────┴─────┐  ┌────┴──────────┐               │
│  │   Call   │  │   Hold    │  │   Services    │               │
│  │ Manager  │  │  Slayer   │  │ (LLM, STT,   │               │
│  │          │  │           │  │  Recording,   │               │
│  │          │  │           │  │  Analytics,   │               │
│  │          │  │           │  │  Notify)      │               │
│  └────┬─────┘  └─────┬─────┘  └──────────────┘               │
│       │              │                                          │
│  ┌────┴──────────────┴───────────────────┐                     │
│  │         Sippy B2BUA Engine            │                     │
│  │  (SIP calls, DTMF, conference bridge) │                     │
│  └────┬──────────────────────────────────┘                     │
│       │                                                         │
└───────┼─────────────────────────────────────────────────────────┘
        │
   ┌────┴────┐
   │SIP Trunk│ ──→ PSTN
   └─────────┘

What's Implemented

Core Engine

• Sippy B2BUA Engine (core/sippy_engine.py) — SIP call control, DTMF, bridging, conference, trunk registration
• PJSUA2 Media Pipeline (core/media_pipeline.py) — Audio routing, recording ports, conference bridge, WAV playback
• Call Manager (core/call_manager.py) — Active call state tracking, lifecycle management
• Event Bus (core/event_bus.py) — Async pub/sub with per-subscriber queues, type filtering, history

Hold Slayer

• IVR Navigation (services/hold_slayer.py) — Follows stored call flows step-by-step through phone menus, including SPEAK steps that synthesize speech via TTS
• Audio Classifier (services/audio_classifier.py) — Real-time waveform analysis: silence, tones, DTMF, music, speech detection
• Call Flow Learner (services/call_flow_learner.py) — Builds reusable call flows from exploration data, merges new discoveries
• LLM Fallback — When a LISTEN step has no hardcoded DTMF, the LLM analyzes the transcript and picks the right menu option

AI Receptionist & Smart Routing

• AI Receptionist (services/receptionist.py) — Answers inbound calls, greets via TTS, captures the caller's intent with STT + LLM, then routes to a device or takes a voicemail
• Smart Routing (services/routing.py) — Caller-pattern (glob), DNIS, time-of-day (with tz + midnight wrap), per-device DND, and ring-chain priority. Rules win over the LLM on conflict.
• TTS (services/tts.py) — Rhema (OpenAI-compatible /v1/audio/speech) — synthesizes Kokoro voices for the SPEAK step and receptionist prompts

Intelligence Layer

• LLM Client (services/llm_client.py) — OpenAI-compatible API client (Ollama, vLLM, LM Studio, OpenAI) with JSON parsing, retry, stats
• Transcription (services/transcription.py) — Speaches/Whisper STT integration for live call transcription
• Recording (services/recording.py) — WAV recording with date-organized storage, dual-channel support, persisted to the recordings table
• Call Persistence (services/call_persistence.py) — Writes completed calls + transcript chunks to the database on hangup
• Notifications (services/notification.py) — WebSocket + SMS alerts for human detection, call failures, hold status

API Surface

• REST API — Call management, call history, transcripts, recordings, routing rules, device DND, call flow CRUD
• WebSocket — Real-time call events, transcripts, classification updates, receptionist state transitions
• MCP Server — 14 tools + 3 resources for AI assistant integration (make calls, send DTMF, get transcripts, manage flows), served over streamable HTTP at /mcp/
• Dashboard — SvelteKit UI served at /dashboard with live monitor, call history with transcript playback, and a routing-rules editor

Data Models

• Call — Active call state with classification history, transcript chunks, hold time tracking
• Call Flow — Stored IVR trees with steps (DTMF, LISTEN, HOLD, TRANSFER, SPEAK)
• Routing Rule — Match (caller pattern, DNIS, time range) + action (ring_device, ring_chain, take_message, reject, dnd)
• Transcript Chunk — Per-call STT segments with speaker tag and timestamp offset (for click-to-seek playback)
• Recording — WAV file metadata (path, duration, size) per call
• Events — 30+ typed events (call lifecycle, hold slayer, audio, device, system, receptionist, routing)
• Device — SIP phone/softphone registration, priority, DND
• Contact — Phone number management with routing preferences

Project Structure

hold-slayer/
├── main.py                      # FastAPI app + lifespan (service wiring)
├── config.py                    # Pydantic settings from .env
├── core/
│   ├── gateway.py               # Top-level gateway orchestrator
│   ├── sippy_engine.py          # Sippy B2BUA SIP engine
│   ├── media_pipeline.py        # PJSUA2 audio routing
│   ├── call_manager.py          # Active call state management
│   └── event_bus.py             # Async pub/sub event bus
├── services/
│   ├── hold_slayer.py           # IVR navigation + hold detection + SPEAK
│   ├── receptionist.py          # AI Receptionist state machine
│   ├── routing.py               # Smart routing (rules, DND, ring chain)
│   ├── tts.py                   # Rhema TTS client (OpenAI-compatible)
│   ├── audio_classifier.py      # Waveform analysis (music/speech/DTMF)
│   ├── call_flow_learner.py     # Auto-learns IVR trees from calls
│   ├── call_persistence.py      # Writes calls + transcript chunks on hangup
│   ├── llm_client.py            # OpenAI-compatible LLM client
│   ├── transcription.py         # Speaches/Whisper STT
│   ├── recording.py             # Call recording management
│   └── notification.py          # WebSocket + SMS notifications
├── api/
│   ├── calls.py                 # Call management endpoints
│   ├── call_history.py          # History, transcript, recording playback
│   ├── call_flows.py            # Call flow CRUD
│   ├── devices.py               # Device registration
│   ├── routing.py               # Routing rules CRUD + per-device DND
│   ├── websocket.py             # Real-time event stream
│   └── deps.py                  # FastAPI dependency injection
├── dashboard/                   # SvelteKit UI (built to dashboard/build)
│   └── src/routes/
│       ├── +page.svelte         # Live monitor
│       ├── history/             # Call history list
│       ├── calls/[call_id]/     # Detail page + transcript playback
│       └── routing/             # Rules editor + DND toggles
├── mcp_server/
│   └── server.py                # MCP tools + resources (10 tools)
├── models/
│   ├── call.py                  # Call state models
│   ├── call_flow.py             # IVR tree models
│   ├── routing.py               # Routing rule / match / action models
│   ├── events.py                # Event type definitions
│   ├── device.py                # Device models
│   └── contact.py               # Contact models
├── db/
│   └── database.py              # SQLAlchemy async (PostgreSQL/SQLite)
└── tests/
    ├── test_audio_classifier.py # 18 tests — waveform analysis
    ├── test_call_flows.py       # 10 tests — call flow models
    ├── test_hold_slayer.py      # 20 tests — IVR nav, EventBus, CallManager
    ├── test_services.py         # 27 tests — LLM, notifications, recording,
    │                            #             analytics, learner, EventBus
    ├── test_tts.py              #  4 tests — Rhema TTS client
    ├── test_routing.py          #  8 tests — rules evaluator
    └── test_receptionist.py     #  7 tests — receptionist decision logic

Quick Start

1. Install

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

Note

The PJSUA2 media pipeline needs the pjsua2 Python bindings, which are not pip-installable — they're built from pjproject (./configure && make && make install with --enable-shared and the Python SWIG target). Without them the media layer runs in stub mode (signaling only).

2. Configure

cp .env.example .env
# Edit .env with your SIP trunk credentials, LLM endpoint, etc.
# Required: DATABASE_URL, and API_TOKEN unless HOST=127.0.0.1
openssl rand -hex 32   # → API_TOKEN

All REST, WebSocket, and MCP access requires Authorization: Bearer $API_TOKEN (WebSocket also accepts ?token=...). An empty token is only permitted when bound to loopback.

3. Build the dashboard (optional but recommended)

cd dashboard
npm install
npm run build
cd ..

The gateway serves the built UI at /dashboard automatically when dashboard/build/ exists. Skip this step if you only need the REST/WS API.

4. Run

uvicorn main:app --host 0.0.0.0 --port 8000

5. Test

pytest tests/ -v

Usage

REST API

Launch Hold Slayer on a number:

curl -X POST http://localhost:8000/api/calls/hold-slayer \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "number": "+18005551234",
    "intent": "dispute Amazon charge from December 15th",
    "call_flow_id": "chase_bank_main",
    "transfer_to": "sip_phone"
  }'

Check call status:

curl http://localhost:8000/api/calls/call_abc123

Browse call history (persisted in the database):

curl http://localhost:8000/api/calls/history?limit=50
curl http://localhost:8000/api/calls/call_abc123/transcript
curl -O http://localhost:8000/api/calls/call_abc123/recording   # WAV

Create a smart-routing rule:

curl -X POST http://localhost:8000/api/routing/rules \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Block tollfree at night",
    "priority": 10,
    "enabled": true,
    "match": {
      "caller_pattern": "+1800*",
      "time_range": {"start": "22:00", "end": "06:00", "tz": "America/Toronto", "days": [0,1,2,3,4,5,6]}
    },
    "action": {"type": "reject", "message": "Office is closed."}
  }'

Toggle Do Not Disturb on a device:

curl -X PATCH http://localhost:8000/api/routing/devices/dev_abc123/dnd \
  -H "Content-Type: application/json" \
  -d '{"enabled": true}'

WebSocket — Real-Time Events

const ws = new WebSocket(`ws://localhost:8000/ws/events?token=${API_TOKEN}`);
ws.onmessage = (msg) => {
  const event = JSON.parse(msg.data);
  // event.type: "human_detected", "hold_detected", "ivr_step", etc.
  // event.call_id: which call this is about
  // event.data: type-specific payload
};

MCP — AI Assistant Integration

The MCP server is served over streamable HTTP at /mcp/ (note the trailing slash) and authenticates with the same bearer token:

claude mcp add hold-slayer --transport http http://localhost:8000/mcp/ \
  --header "Authorization: Bearer $API_TOKEN"

It exposes 14 tools and 3 resources (gateway://status, gateway://call-flows, gateway://active-calls):

Tool	Description
make_call	Dial a real number through the SIP trunk (emergency numbers refused)
hangup	Hang up an active call
transfer_call	Transfer an active call to a device
send_dtmf	Send touch-tone digits to navigate menus
get_call_status	Check current state of a call
get_call_transcript	Get live transcript of a call
get_call_recording	Get recording metadata and file path
list_active_calls	List all calls in progress
list_devices	List registered devices and status
gateway_status	Trunk, devices, active calls, uptime
get_call_flow	Look up a stored IVR flow for a number
create_call_flow	Store a new IVR call flow
get_call_summary	Stored summary and action items for a call
search_call_history	Search past calls by number or intent

How It Works

Outbound (Hold Slayer)

1. You request a call — via REST API, MCP tool, or dashboard
2. Gateway dials out — Sippy B2BUA places the call through your SIP trunk
3. Audio classifier listens — Real-time waveform analysis detects IVR prompts, hold music, ringing, silence, and live speech
4. Transcription runs — Speaches/Whisper converts audio to text in real-time
5. IVR navigator decides — If a stored call flow exists, it follows the steps (including SPEAK steps that synthesize speech via Rhema TTS). If not, the LLM analyzes the transcript and picks the right menu option
6. Hold detection — When hold music is detected, the system waits patiently and monitors for transitions
7. Human detection — The classifier detects the transition from music/silence to live speech
8. Transfer — Your desk phone rings. Pick up and you're talking to the agent. Zero hold time.

Inbound (AI Receptionist + Smart Routing)

1. SIP INVITE arrives — Sippy surfaces it to the gateway instead of auto-answering
2. Routing rules evaluate — Caller pattern, DNIS, and time-of-day rules run in priority order. A reject or dnd action declines the call immediately.
3. Receptionist answers — TTS plays the greeting; the call's audio tap captures the caller's response
4. Intent capture — The utterance is transcribed and the LLM extracts intent, urgency, and a recommended action (ring / message / reject)
5. Final decision — Routing rules win on conflict; otherwise the LLM's recommendation is followed
6. Route or take a message — ring_chain tries devices in priority order (skipping any in DND); if nobody picks up (or the action is take_message), the receptionist records up to 90s, transcribes it, and emits a RECEPTIONIST_MESSAGE_SAVED event

Configuration

All configuration is via environment variables (see .env.example):

Variable	Description	Default
DATABASE_URL	PostgreSQL connection string	— (required)
API_TOKEN	Static bearer token for REST/WS/MCP	— (required unless HOST=127.0.0.1)
MAX_CONCURRENT_CALLS	Cap on simultaneous outbound calls	4
SIP_TRUNK_HOST	Your SIP provider hostname	—
SIP_TRUNK_USERNAME	SIP auth username	—
SIP_TRUNK_PASSWORD	SIP auth password	—
SIP_TRUNK_DID	Your phone number (E.164)	—
GATEWAY_SIP_PORT	Port for device registration	5080
SPEACHES_URL	Speaches/Whisper STT endpoint	http://localhost:22070
LLM_BASE_URL	OpenAI-compatible LLM endpoint	http://localhost:11434/v1
LLM_MODEL	Model name for IVR analysis	llama3
TTS_BASE_URL	Rhema TTS endpoint (OpenAI-compatible)	http://localhost:8000
TTS_MODEL	TTS model ID	speaches-ai/Kokoro-82M-v1.0-ONNX
TTS_VOICE	Default Kokoro voice	af_heart
TTS_API_KEY	Optional bearer token for Rhema	—
RECEPTIONIST_ENABLED	Answer inbound calls with the AI receptionist	true
RECEPTIONIST_GREETING_TEMPLATE	Spoken greeting	"Hi, you've reached Robert's line. Who's calling, and what's this about?"
RECEPTIONIST_MESSAGE_MAX_SECONDS	Voicemail cap	90

Tech Stack

• Python 3.13 + asyncio — Single-process async architecture
• FastAPI — REST API + WebSocket server
• SvelteKit — Dashboard UI (built static, served by FastAPI at /dashboard)
• Sippy B2BUA — SIP call control and DTMF
• PJSUA2 — Media pipeline, conference bridge, recording, WAV playback
• Speaches (Whisper) — Speech-to-text
• Rhema (Kokoro) — Text-to-speech (OpenAI-compatible /v1/audio/speech)
• Ollama / vLLM / OpenAI — LLM for IVR menu analysis and receptionist intent capture
• SQLAlchemy — Async database (PostgreSQL or SQLite)
• MCP (Model Context Protocol) — AI assistant integration

Documentation

Full documentation is in /docs:

• Architecture — System design, data flow, threading model
• Core Engine — SIP engine, media pipeline, call manager, event bus
• Hold Slayer Service — IVR navigation, hold detection, human detection
• Audio Classifier — Waveform analysis, feature extraction, classification
• Services — LLM client, transcription, recording, analytics, notifications
• Call Flows — Call flow model, step types, auto-learner
• API Reference — REST endpoints, WebSocket, request/response schemas
• MCP Server — MCP tools and resources for AI assistants
• Configuration — All environment variables, deployment options
• Development — Setup, testing, contributing

Build Phases

Phase 1: Core Engine ✅

• Extract EventBus to dedicated module with typed filtering
• Implement Sippy B2BUA SIP engine (signaling, DTMF, bridging)
• Implement PJSUA2 media pipeline (conference bridge, audio tapping, recording)
• Call manager with active call state tracking
• Gateway orchestrator wiring all components

Phase 2: Intelligence Layer ✅

• LLM client (OpenAI-compatible — Ollama, vLLM, LM Studio, OpenAI)
• Hold Slayer IVR navigation with LLM fallback for LISTEN steps
• Call Flow Learner — auto-builds reusable IVR trees from exploration
• Recording service with date-organized WAV storage
• Audio classifier with spectral analysis, DTMF detection, hold-to-human transition

Phase 3: API & Integration ✅

• REST API — calls, call flows, devices, DTMF
• WebSocket real-time event streaming
• MCP server with 14 tools + 3 resources, mounted at /mcp/ (streamable HTTP)
• Notification service (WebSocket + SMS)
• Service wiring in main.py lifespan

Phase 4: Production Hardening 🚧

• Alembic database migrations
• API authentication — static bearer token across REST/WS/MCP
• Emergency-number guard + concurrent-call cap on outbound calls
• Rate limiting on API endpoints
• Structured JSON logging
• Health check endpoints for all dependencies
• Graceful degradation (classifier works without STT, etc.)
• Docker Compose (Hold Slayer + PostgreSQL)

Phase 5: Additional Services 🚧

• AI Receptionist — answer inbound calls, screen callers, take messages
• Smart Routing — time-of-day rules, device priority, DND
• TTS/Speech — play prompts into calls (SPEAK step support, Rhema/Kokoro)
• Spam Filter — detect robocalls using caller ID + audio patterns
• Noise Cancellation — RNNoise integration in media pipeline

Phase 6: Dashboard & UX 🚧

• Web dashboard with real-time call monitor
• Call history with transcript playback (click-to-seek)
• Routing rules editor + per-device DND toggles
• Call flow visual editor (drag-and-drop IVR tree builder)
• Analytics dashboard with hold time graphs
• Mobile app (or PWA) for on-the-go control

License

MIT