hold-slayer/docs/architecture.md

Architecture

Hold Slayer is a single-process async Python application built on FastAPI. It acts as an intelligent B2BUA (Back-to-Back User Agent) sitting between your SIP trunk (PSTN access) and your desk phone/softphone.

Two SIP engines, selected by SIP_ENGINE. sippy (the default) signals only — PJSUA2 will not surface an RTP stream for a dialog it does not own, so no audio reaches the classifier on that path. pjsua2 (core/pjsua_engine.py) places the call itself and is the only mode where audio reaches the classifier; it is opt-in while being proven against the lab. Read Media plane: why PJSUA2 places the call before changing anything in core/.

System Diagram

┌─────────────────────────────────────────────────────────────────┐
│                        FastAPI Server                           │
│                                                                 │
│  ┌──────────┐  ┌──────────┐  ┌───────────┐  ┌──────────────┐  │
│  │ REST API │  │WebSocket │  │MCP Server │  │  Dashboard   │  │
│  │ /api/v1/*│  │ /ws/*    │  │ (HTTP)    │  │      /       │  │
│  └────┬─────┘  └────┬─────┘  └─────┬─────┘  └──────────────┘  │
│       │              │              │                            │
│  ┌────┴──────────────┴──────────────┴────┐                     │
│  │             Event Bus                  │                     │
│  │   (asyncio Queue pub/sub per client)   │                     │
│  └────┬──────────────┬──────────────┬────┘                     │
│       │              │              │                            │
│  ┌────┴─────┐  ┌─────┴─────┐  ┌────┴──────────┐               │
│  │   Call   │  │   Hold    │  │   Services    │               │
│  │ Manager  │  │  Slayer   │  │ (LLM, STT,   │               │
│  │          │  │           │  │  Recording,   │               │
│  │          │  │           │  │  Analytics,   │               │
│  │          │  │           │  │  Notify)      │               │
│  └────┬─────┘  └─────┬─────┘  └──────────────┘               │
│       │              │                                          │
│  ┌────┴──────────────┴───────────────────┐                     │
│  │            SIP Engine                  │                     │
│  │  signalling + call control             │                     │
│  └────┬──────────────────────────────────┘                     │
│       │                                                         │
│  ┌────┴──────────────────────────────────┐                     │
│  │          Media Pipeline (PJSUA2)       │                     │
│  │  RTP, conference bridge, taps, record  │                     │
│  └────┬──────────────────────────────────┘                     │
│       │                                                         │
└───────┼─────────────────────────────────────────────────────────┘
        │
   ┌────┴────┐
   │SIP Trunk│ ──→ PSTN
   └─────────┘

Component Overview

Presentation Layer

Component	File	Protocol	Purpose
REST API	api/calls.py, api/call_flows.py, api/devices.py	HTTP	Call management, CRUD, configuration
WebSocket	api/websocket.py	WS	Real-time event streaming to clients
MCP Server	mcp_server/server.py	Streamable HTTP at /mcp/	AI assistant tool integration

Orchestration Layer

Component	File	Purpose
Gateway	core/gateway.py	Top-level orchestrator — owns all services, routes calls
Call Manager	core/call_manager.py	Active call state, lifecycle, transcript tracking
Event Bus	core/event_bus.py	Async pub/sub connecting everything together

Intelligence Layer

Component	File	Purpose
Hold Slayer	services/hold_slayer.py	IVR navigation, hold monitoring, human detection
Audio Classifier	services/audio_classifier.py	Real-time waveform analysis (music/speech/DTMF/silence)
LLM Client	services/llm_client.py	OpenAI-compatible LLM for IVR menu decisions
Transcription	services/transcription.py	Speaches/Whisper STT for live audio
Call Flow Learner	services/call_flow_learner.py	Builds reusable IVR trees from exploration data

Infrastructure Layer

Component	File	Purpose
Sippy Engine	core/sippy_engine.py	SIP signalling (INVITE, BYE, REGISTER, DTMF)
Media Pipeline	core/media_pipeline.py	PJSUA2 RTP media, conference bridge, taps, recording
Recording	services/recording.py	WAV file management and storage
Analytics	services/call_analytics.py	Call metrics, hold time stats, trends
Notifications	services/notification.py	WebSocket + SMS alerts
Database	db/database.py	SQLAlchemy async (PostgreSQL, Alembic migrations)

Data Flow — Hold Slayer Call

1. User Request
   POST /api/v1/calls/hold-slayer { number, intent, call_flow_id }
         │
2. Gateway.make_call()
   ├── is_emergency_number()          → REFUSE 911/112 (before anything else)
   ├── concurrency cap check          → refuse past max_concurrent_calls
   ├── CallManager.create_call()      → track state
   └── sip_engine.make_call()         → place the call, media follows
         │
3. HoldSlayer.run_with_flow() or run_exploration()
   ├── AudioClassifier.classify()    → analyze 3s audio windows
   │   ├── silence? → wait
   │   ├── ringing? → wait
   │   ├── DTMF? → detect tones
   │   ├── music? → HOLD_DETECTED event
   │   └── speech? → transcribe + decide
   │
   ├── TranscriptionService.transcribe() → STT on speech audio
   │
   ├── LLMClient.analyze_ivr_menu() → pick menu option (fallback)
   │   └── sip_engine.send_dtmf()   → press the button
   │
   └── detect_hold_to_human_transition()
       └── HUMAN_DETECTED! → transfer
           │
4. Transfer
   ├── SippyEngine.bridge_calls()      → join the two call legs
   ├── MediaPipeline.bridge_streams()  → bridge RTP in the conf bridge
   ├── EventBus.publish(TRANSFER_STARTED)
   └── NotificationService → "Pick up your phone!"
         │
5. Real-Time Updates (throughout)
   EventBus.publish() → WebSocket clients
                      → MCP server resources
                      → Notification service
                      → Analytics tracking

The emergency guard and the concurrency cap are the first two steps of make_call for a reason, and their order is load-bearing — see .claude/rules/call-safety.md.

Threading Model

The README's "single-process async" is a simplification. There are three execution contexts, and the boundaries between them are the highest-leverage invariant in the codebase.

asyncio loop (main thread)          Sippy ED thread         PJSUA2 worker threads
├── FastAPI (uvicorn)               └── ED2 dispatcher      └── media / RTP
├── EventBus                            ├── SIP signalling      └── onFrameReceived
├── CallManager                         ├── UA objects
├── HoldSlayer                          └── DTMF relay
├── AudioClassifier
├── TranscriptionService
├── LLMClient
├── NotificationService
└── RecordingService

Crossing the boundaries — one funnel each way:

Direction	Mechanism	Notes
Sippy ED → loop	_post_from_ed → asyncio.run_coroutine_threadsafe → _on_engine_event	The single funnel where Sippy-thread events mutate loop state
loop → Sippy ED	_run_on_sippy → ED2.callFromThread	Anything touching a Sippy UA object
PJSUA2 worker → loop	AudioTap.feed → loop.call_soon_threadsafe	The only thing a PJSUA2 callback may touch

onFrameReceived runs on a PJSUA2 worker thread every 20 ms. It must call nothing but AudioTap.feed; reaching into pipeline state, the event bus, or a Sippy object from there is a data race. An exception escaping into PJSUA2's C++ callback tears down the worker thread and silently kills media for every call, which is why the capture port catches and logs once rather than per frame.

Full detail: .claude/rules/concurrency-threads.md.

Design Decisions

Media plane: why PJSUA2 places the call

The original split was Sippy signals, PJSUA2 carries media. It does not work, for a reason that is not obvious until you try it:

PJSUA2 exposes no standalone RTP media object. Every AudioMedia subclass in the Python bindings is a file player, recorder, tone generator, or capture port. RTP is reachable only through pj.Call.getAudioMedia(), after onCallMediaState fires on a dialog PJSUA2 itself owns. There is no "give me an AudioMedia for this remote host:port" API to call.

So a design where Sippy owns the dialog can never obtain a media stream from PJSUA2. MediaPipeline.add_remote_stream() is not unfinished work — it is a function that cannot be written against this API. The consequence is that audio never reaches the classifier: create_tap builds a valid capture port with nothing to attach it to.

The resolution: PJSUA2 places the call; Sippy keeps every other role.

Concern	Owner
Emergency guard, concurrency cap	gateway.make_call — unchanged, still first
Trunk registration	PJSUA2 Account
Outbound INVITE / answer / hangup	PJSUA2 Call
RTP, conference bridge, taps, recording	PJSUA2 media
DTMF	PJSUA2 Call.dialDtmf (RFC 2833)
Device registration, routing, leg bridging	Sippy / gateway
Inbound call dispatch	PJSUA2 Account.onIncomingCall

Sippy remains the SBC-shaped layer — it is where device registrations, routing decisions and B2BUA leg-joining live. What moves is the raw dialog for a trunk call, because owning the dialog is the price of owning the media.

Alternatives considered and rejected:

• Terminate RTP ourselves (aiortc or raw sockets) and feed PCM into AudioTap directly, keeping Sippy on the wire. Preserves the split, but means owning jitter buffering, packet loss concealment and ulaw/alaw transcoding — precisely the work PJSUA2 exists to do.
• A loopback pj.Call mirroring each real leg, so PJSUA2 has a dialog it owns. Avoids touching call placement, but adds a phantom call per real call and the SDP juggling is fragile.

Safety note for this refactor: is_emergency_number() stays the first check in gateway.make_call, above the concurrency cap and above any SIP action, regardless of which library dials. A new outbound path that reaches the SIP layer without passing that guard is a serious regression even if every test passes.

Why asyncio Queue-based EventBus?

• Single process — no need for Redis/RabbitMQ cross-process messaging
• Zero dependencies — pure asyncio, no external services to deploy
• Per-subscriber queues — slow consumers don't block fast publishers
• Dead subscriber cleanup — full queues are automatically removed
• Event history — late joiners can catch up on recent events

If scaling to multiple gateway processes becomes necessary, the EventBus interface can be backed by Redis pub/sub without changing consumers.

Why OpenAI-compatible LLM API?

The LLM client uses raw HTTP (httpx) against any OpenAI-compatible endpoint. This means:

• Ollama (local, free) — http://localhost:11434/v1
• LM Studio (local, free) — http://localhost:1234/v1
• vLLM (local, fast) — http://localhost:8000/v1
• OpenAI (cloud) — https://api.openai.com/v1

No SDK dependency. No vendor lock-in. Switch models by changing one env var.

Testing against a fake PSTN

tests/lab/ runs an Asterisk instance that answers calls, plays an IVR, holds with music and connects a "human" — so the gateway has something real to dial that is not the PSTN. SIP_TRUNK_HOST is just an address, so the production code path runs unmodified; while it points at the lab there is no route to the PSTN at all. See tests/lab/README.md.