Robert Helewka
7979e70705
Implements the tap half of the media path and records why the other half requires moving call placement into PJSUA2. MediaPipeline.create_tap was a stub: it logged "🎤 Audio tap created" and returned a tap that nothing ever fed, so the classifier received no audio on a live call. It now builds a real pj.AudioMediaPort subclass whose onFrameReceived converts the SWIG ByteVector to PCM bytes and fans it out to every tap on the stream. One capture port per stream, shared by all taps: a second port on the same stream would be mixed back into the conference bridge and the call would echo. Thread safety is the constraint here. onFrameReceived runs on a PJSUA2 worker thread — a third execution context beside the asyncio loop and the Sippy ED thread — and touches nothing but AudioTap.feed, which hops to the owning loop via call_soon_threadsafe. An exception escaping into PJSUA2's C++ callback would tear down the worker thread and silently kill media for every call, so the handler catches and logs once per port rather than on every 20ms frame. Also fixes a hard crash found while testing this against real PJSUA2: a media port finalised after Endpoint.libDestroy() calls pjmedia_conf_remove_port against a freed conference bridge and aborts the process on a native assertion. Ports are now released in remove_stream while the bridge still exists, and stop() forces a collection before libDestroy — dropping the last Python reference is not sufficient on its own. Verified against the real bindings: frames fan out to multiple taps, cross the thread boundary intact, and shutdown is clean. add_remote_stream remains a stub, and deliberately so. PJSUA2 exposes no standalone RTP media object — every AudioMedia subclass in the Python bindings is a file player, recorder, tone generator or capture port, and RTP is reachable only via pj.Call.getAudioMedia() on a dialog PJSUA2 itself owns. A design where Sippy owns the dialog can never obtain media from PJSUA2, so that function cannot be written against this API. docs/architecture.md now explains this and records the resolution: PJSUA2 places the trunk call while Sippy keeps the SBC roles (device registration, routing, leg bridging), with the emergency guard and concurrency cap staying first in gateway.make_call regardless of which library dials. The architecture doc also had drift unrelated to media: it described the thread boundary as asyncio.run_in_executor() when the real mechanism is run_coroutine_threadsafe / ED2.callFromThread, claimed two execution contexts where there are three, and cited MediaPipeline.add_stream() and SippyEngine.bridge() — neither of which exists. Corrected, with the data flow now showing the emergency guard and concurrency cap in their real positions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
255 lines
14 KiB
Markdown
255 lines
14 KiB
Markdown
# Architecture
|
|
|
|
Hold Slayer is a single-process async Python application built on FastAPI. It
|
|
acts as an intelligent B2BUA (Back-to-Back User Agent) sitting between your SIP
|
|
trunk (PSTN access) and your desk phone/softphone.
|
|
|
|
> **Media plane in transition.** The gateway currently signals with Sippy and
|
|
> intends PJSUA2 to carry media, but PJSUA2 will not surface an RTP stream for
|
|
> a dialog it does not own — so no audio ever reaches the classifier. The fix
|
|
> moves *call placement* into PJSUA2 while Sippy keeps the SBC roles. See
|
|
> [Media plane: why PJSUA2 places the call](#media-plane-why-pjsua2-places-the-call)
|
|
> before changing anything in `core/`.
|
|
|
|
## System Diagram
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ FastAPI Server │
|
|
│ │
|
|
│ ┌──────────┐ ┌──────────┐ ┌───────────┐ ┌──────────────┐ │
|
|
│ │ REST API │ │WebSocket │ │MCP Server │ │ Dashboard │ │
|
|
│ │ /api/v1/*│ │ /ws/* │ │ (HTTP) │ │ / │ │
|
|
│ └────┬─────┘ └────┬─────┘ └─────┬─────┘ └──────────────┘ │
|
|
│ │ │ │ │
|
|
│ ┌────┴──────────────┴──────────────┴────┐ │
|
|
│ │ Event Bus │ │
|
|
│ │ (asyncio Queue pub/sub per client) │ │
|
|
│ └────┬──────────────┬──────────────┬────┘ │
|
|
│ │ │ │ │
|
|
│ ┌────┴─────┐ ┌─────┴─────┐ ┌────┴──────────┐ │
|
|
│ │ Call │ │ Hold │ │ Services │ │
|
|
│ │ Manager │ │ Slayer │ │ (LLM, STT, │ │
|
|
│ │ │ │ │ │ Recording, │ │
|
|
│ │ │ │ │ │ Analytics, │ │
|
|
│ │ │ │ │ │ Notify) │ │
|
|
│ └────┬─────┘ └─────┬─────┘ └──────────────┘ │
|
|
│ │ │ │
|
|
│ ┌────┴──────────────┴───────────────────┐ │
|
|
│ │ SIP Engine │ │
|
|
│ │ signalling + call control │ │
|
|
│ └────┬──────────────────────────────────┘ │
|
|
│ │ │
|
|
│ ┌────┴──────────────────────────────────┐ │
|
|
│ │ Media Pipeline (PJSUA2) │ │
|
|
│ │ RTP, conference bridge, taps, record │ │
|
|
│ └────┬──────────────────────────────────┘ │
|
|
│ │ │
|
|
└───────┼─────────────────────────────────────────────────────────┘
|
|
│
|
|
┌────┴────┐
|
|
│SIP Trunk│ ──→ PSTN
|
|
└─────────┘
|
|
```
|
|
|
|
## Component Overview
|
|
|
|
### Presentation Layer
|
|
|
|
| Component | File | Protocol | Purpose |
|
|
|-----------|------|----------|---------|
|
|
| REST API | `api/calls.py`, `api/call_flows.py`, `api/devices.py` | HTTP | Call management, CRUD, configuration |
|
|
| WebSocket | `api/websocket.py` | WS | Real-time event streaming to clients |
|
|
| MCP Server | `mcp_server/server.py` | Streamable HTTP at `/mcp/` | AI assistant tool integration |
|
|
|
|
### Orchestration Layer
|
|
|
|
| Component | File | Purpose |
|
|
|-----------|------|---------|
|
|
| Gateway | `core/gateway.py` | Top-level orchestrator — owns all services, routes calls |
|
|
| Call Manager | `core/call_manager.py` | Active call state, lifecycle, transcript tracking |
|
|
| Event Bus | `core/event_bus.py` | Async pub/sub connecting everything together |
|
|
|
|
### Intelligence Layer
|
|
|
|
| Component | File | Purpose |
|
|
|-----------|------|---------|
|
|
| Hold Slayer | `services/hold_slayer.py` | IVR navigation, hold monitoring, human detection |
|
|
| Audio Classifier | `services/audio_classifier.py` | Real-time waveform analysis (music/speech/DTMF/silence) |
|
|
| LLM Client | `services/llm_client.py` | OpenAI-compatible LLM for IVR menu decisions |
|
|
| Transcription | `services/transcription.py` | Speaches/Whisper STT for live audio |
|
|
| Call Flow Learner | `services/call_flow_learner.py` | Builds reusable IVR trees from exploration data |
|
|
|
|
### Infrastructure Layer
|
|
|
|
| Component | File | Purpose |
|
|
|-----------|------|---------|
|
|
| Sippy Engine | `core/sippy_engine.py` | SIP signalling (INVITE, BYE, REGISTER, DTMF) |
|
|
| Media Pipeline | `core/media_pipeline.py` | PJSUA2 RTP media, conference bridge, taps, recording |
|
|
| Recording | `services/recording.py` | WAV file management and storage |
|
|
| Analytics | `services/call_analytics.py` | Call metrics, hold time stats, trends |
|
|
| Notifications | `services/notification.py` | WebSocket + SMS alerts |
|
|
| Database | `db/database.py` | SQLAlchemy async (PostgreSQL, Alembic migrations) |
|
|
|
|
## Data Flow — Hold Slayer Call
|
|
|
|
```
|
|
1. User Request
|
|
POST /api/v1/calls/hold-slayer { number, intent, call_flow_id }
|
|
│
|
|
2. Gateway.make_call()
|
|
├── is_emergency_number() → REFUSE 911/112 (before anything else)
|
|
├── concurrency cap check → refuse past max_concurrent_calls
|
|
├── CallManager.create_call() → track state
|
|
└── sip_engine.make_call() → place the call, media follows
|
|
│
|
|
3. HoldSlayer.run_with_flow() or run_exploration()
|
|
├── AudioClassifier.classify() → analyze 3s audio windows
|
|
│ ├── silence? → wait
|
|
│ ├── ringing? → wait
|
|
│ ├── DTMF? → detect tones
|
|
│ ├── music? → HOLD_DETECTED event
|
|
│ └── speech? → transcribe + decide
|
|
│
|
|
├── TranscriptionService.transcribe() → STT on speech audio
|
|
│
|
|
├── LLMClient.analyze_ivr_menu() → pick menu option (fallback)
|
|
│ └── sip_engine.send_dtmf() → press the button
|
|
│
|
|
└── detect_hold_to_human_transition()
|
|
└── HUMAN_DETECTED! → transfer
|
|
│
|
|
4. Transfer
|
|
├── SippyEngine.bridge_calls() → join the two call legs
|
|
├── MediaPipeline.bridge_streams() → bridge RTP in the conf bridge
|
|
├── EventBus.publish(TRANSFER_STARTED)
|
|
└── NotificationService → "Pick up your phone!"
|
|
│
|
|
5. Real-Time Updates (throughout)
|
|
EventBus.publish() → WebSocket clients
|
|
→ MCP server resources
|
|
→ Notification service
|
|
→ Analytics tracking
|
|
```
|
|
|
|
The emergency guard and the concurrency cap are the first two steps of
|
|
`make_call` for a reason, and their order is load-bearing — see
|
|
[.claude/rules/call-safety.md](../.claude/rules/call-safety.md).
|
|
|
|
## Threading Model
|
|
|
|
The README's "single-process async" is a simplification. There are **three**
|
|
execution contexts, and the boundaries between them are the highest-leverage
|
|
invariant in the codebase.
|
|
|
|
```
|
|
asyncio loop (main thread) Sippy ED thread PJSUA2 worker threads
|
|
├── FastAPI (uvicorn) └── ED2 dispatcher └── media / RTP
|
|
├── EventBus ├── SIP signalling └── onFrameReceived
|
|
├── CallManager ├── UA objects
|
|
├── HoldSlayer └── DTMF relay
|
|
├── AudioClassifier
|
|
├── TranscriptionService
|
|
├── LLMClient
|
|
├── NotificationService
|
|
└── RecordingService
|
|
```
|
|
|
|
**Crossing the boundaries — one funnel each way:**
|
|
|
|
| Direction | Mechanism | Notes |
|
|
|---|---|---|
|
|
| Sippy ED → loop | `_post_from_ed` → `asyncio.run_coroutine_threadsafe` → `_on_engine_event` | The single funnel where Sippy-thread events mutate loop state |
|
|
| loop → Sippy ED | `_run_on_sippy` → `ED2.callFromThread` | Anything touching a Sippy UA object |
|
|
| PJSUA2 worker → loop | `AudioTap.feed` → `loop.call_soon_threadsafe` | The **only** thing a PJSUA2 callback may touch |
|
|
|
|
`onFrameReceived` runs on a PJSUA2 worker thread every 20 ms. It must call
|
|
nothing but `AudioTap.feed`; reaching into pipeline state, the event bus, or a
|
|
Sippy object from there is a data race. An exception escaping into PJSUA2's C++
|
|
callback tears down the worker thread and silently kills media for every call,
|
|
which is why the capture port catches and logs once rather than per frame.
|
|
|
|
Full detail: [.claude/rules/concurrency-threads.md](../.claude/rules/concurrency-threads.md).
|
|
|
|
## Design Decisions
|
|
|
|
### Media plane: why PJSUA2 places the call
|
|
|
|
The original split was *Sippy signals, PJSUA2 carries media*. It does not work,
|
|
for a reason that is not obvious until you try it:
|
|
|
|
**PJSUA2 exposes no standalone RTP media object.** Every `AudioMedia` subclass
|
|
in the Python bindings is a file player, recorder, tone generator, or capture
|
|
port. RTP is reachable only through `pj.Call.getAudioMedia()`, after
|
|
`onCallMediaState` fires on a dialog **PJSUA2 itself owns**. There is no
|
|
"give me an AudioMedia for this remote host:port" API to call.
|
|
|
|
So a design where Sippy owns the dialog can never obtain a media stream from
|
|
PJSUA2. `MediaPipeline.add_remote_stream()` is not unfinished work — it is a
|
|
function that cannot be written against this API. The consequence is that audio
|
|
never reaches the classifier: `create_tap` builds a valid capture port with
|
|
nothing to attach it to.
|
|
|
|
**The resolution: PJSUA2 places the call; Sippy keeps every other role.**
|
|
|
|
| Concern | Owner |
|
|
|---|---|
|
|
| Emergency guard, concurrency cap | `gateway.make_call` — unchanged, still first |
|
|
| Trunk registration | PJSUA2 `Account` |
|
|
| Outbound INVITE / answer / hangup | PJSUA2 `Call` |
|
|
| RTP, conference bridge, taps, recording | PJSUA2 media |
|
|
| DTMF | PJSUA2 `Call.dialDtmf` (RFC 2833) |
|
|
| Device registration, routing, leg bridging | Sippy / gateway |
|
|
| Inbound call dispatch | PJSUA2 `Account.onIncomingCall` |
|
|
|
|
Sippy remains the SBC-shaped layer — it is where device registrations, routing
|
|
decisions and B2BUA leg-joining live. What moves is the raw dialog for a trunk
|
|
call, because owning the dialog is the price of owning the media.
|
|
|
|
Alternatives considered and rejected:
|
|
|
|
- **Terminate RTP ourselves** (aiortc or raw sockets) and feed PCM into
|
|
`AudioTap` directly, keeping Sippy on the wire. Preserves the split, but
|
|
means owning jitter buffering, packet loss concealment and ulaw/alaw
|
|
transcoding — precisely the work PJSUA2 exists to do.
|
|
- **A loopback `pj.Call` mirroring each real leg**, so PJSUA2 has a dialog it
|
|
owns. Avoids touching call placement, but adds a phantom call per real call
|
|
and the SDP juggling is fragile.
|
|
|
|
> **Safety note for this refactor:** `is_emergency_number()` stays the first
|
|
> check in `gateway.make_call`, above the concurrency cap and above any SIP
|
|
> action, regardless of which library dials. A new outbound path that reaches
|
|
> the SIP layer without passing that guard is a serious regression even if
|
|
> every test passes.
|
|
|
|
### Why asyncio Queue-based EventBus?
|
|
|
|
- **Single process** — no need for Redis/RabbitMQ cross-process messaging
|
|
- **Zero dependencies** — pure asyncio, no external services to deploy
|
|
- **Per-subscriber queues** — slow consumers don't block fast publishers
|
|
- **Dead subscriber cleanup** — full queues are automatically removed
|
|
- **Event history** — late joiners can catch up on recent events
|
|
|
|
If scaling to multiple gateway processes becomes necessary, the EventBus
|
|
interface can be backed by Redis pub/sub without changing consumers.
|
|
|
|
### Why OpenAI-compatible LLM API?
|
|
|
|
The LLM client uses raw HTTP (httpx) against any OpenAI-compatible endpoint.
|
|
This means:
|
|
|
|
- **Ollama** (local, free) — `http://localhost:11434/v1`
|
|
- **LM Studio** (local, free) — `http://localhost:1234/v1`
|
|
- **vLLM** (local, fast) — `http://localhost:8000/v1`
|
|
- **OpenAI** (cloud) — `https://api.openai.com/v1`
|
|
|
|
No SDK dependency. No vendor lock-in. Switch models by changing one env var.
|
|
|
|
## Testing against a fake PSTN
|
|
|
|
`tests/lab/` runs an Asterisk instance that answers calls, plays an IVR, holds
|
|
with music and connects a "human" — so the gateway has something real to dial
|
|
that is not the PSTN. `SIP_TRUNK_HOST` is just an address, so the production
|
|
code path runs unmodified; while it points at the lab there is no route to the
|
|
PSTN at all. See [tests/lab/README.md](../tests/lab/README.md).
|