The gateway can now hear. Verified end to end against the Asterisk lab:
speech classifies as live_human, hold music as music, the speech→music
transition tracks the dialplan, and DTMF reaches a real IVR (Asterisk logged
"caller pressed 1 -> accounts" and branched). RTP stats show 0% packet loss.
PJSUAEngine implements the existing SIPEngine interface, so the gateway,
call manager and hold-slayer service are unchanged. It is selected with
SIP_ENGINE=pjsua2; the default stays "sippy" while this is proven, and the
mock remains opt-in as before.
Why a new engine rather than fixing the old path: PJSUA2 exposes no
standalone RTP media object, so it will not surface media for a dialog it
does not own. Owning the dialog is the price of owning the media. Sippy keeps
the SBC roles it is good at — device registration, routing, leg bridging —
and SippyEngine remains fully functional for signalling; it simply cannot
carry media, which its media branch now says plainly instead of calling a
method that could never work.
The safety invariants are untouched. This engine is reachable only through
gateway.make_call, which refuses emergency numbers and enforces the
concurrency cap before any SIP action. No new dial path was introduced.
MediaPipeline.add_remote_stream(host, port) is replaced by
attach_call_media(stream_id, audio_media), called from onCallMediaState —
the one place PJSUA2 hands out RTP-backed media. Taps requested before media
comes up are attached when it does, so the classifier never misses the start
of a call.
Three crash/lifetime bugs found by running against the real bindings, none of
which any unit test would have caught:
- pj.Call and pj.Account objects finalised after libDestroy() abort the
process on a native assertion, exactly as media ports do. Both are now
dropped and collected before the pipeline destroys the endpoint.
- PJSUA2 keeps delivering callbacks during interpreter teardown, when module
globals may already be cleared. The callbacks alias what they need locally
and swallow everything: a raise there escapes into C++ and takes the worker
thread with it.
- hangup() only queues the BYE, so shutdown deleted the account with a call
still active and left the far end on an unclosed dialog. stop() now waits
briefly for the teardown to complete.
Threading follows the established rule: PJSUA2 worker threads reach the loop
only through _post_from_pj → run_coroutine_threadsafe, and any thread PJSUA2
did not create registers itself before touching a PJSUA2 object.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Implements the tap half of the media path and records why the other half
requires moving call placement into PJSUA2.
MediaPipeline.create_tap was a stub: it logged "🎤 Audio tap created" and
returned a tap that nothing ever fed, so the classifier received no audio on
a live call. It now builds a real pj.AudioMediaPort subclass whose
onFrameReceived converts the SWIG ByteVector to PCM bytes and fans it out to
every tap on the stream.
One capture port per stream, shared by all taps: a second port on the same
stream would be mixed back into the conference bridge and the call would echo.
Thread safety is the constraint here. onFrameReceived runs on a PJSUA2 worker
thread — a third execution context beside the asyncio loop and the Sippy ED
thread — and touches nothing but AudioTap.feed, which hops to the owning loop
via call_soon_threadsafe. An exception escaping into PJSUA2's C++ callback
would tear down the worker thread and silently kill media for every call, so
the handler catches and logs once per port rather than on every 20ms frame.
Also fixes a hard crash found while testing this against real PJSUA2: a media
port finalised after Endpoint.libDestroy() calls pjmedia_conf_remove_port
against a freed conference bridge and aborts the process on a native
assertion. Ports are now released in remove_stream while the bridge still
exists, and stop() forces a collection before libDestroy — dropping the last
Python reference is not sufficient on its own.
Verified against the real bindings: frames fan out to multiple taps, cross the
thread boundary intact, and shutdown is clean.
add_remote_stream remains a stub, and deliberately so. PJSUA2 exposes no
standalone RTP media object — every AudioMedia subclass in the Python
bindings is a file player, recorder, tone generator or capture port, and RTP
is reachable only via pj.Call.getAudioMedia() on a dialog PJSUA2 itself owns.
A design where Sippy owns the dialog can never obtain media from PJSUA2, so
that function cannot be written against this API. docs/architecture.md now
explains this and records the resolution: PJSUA2 places the trunk call while
Sippy keeps the SBC roles (device registration, routing, leg bridging), with
the emergency guard and concurrency cap staying first in gateway.make_call
regardless of which library dials.
The architecture doc also had drift unrelated to media: it described the
thread boundary as asyncio.run_in_executor() when the real mechanism is
run_coroutine_threadsafe / ED2.callFromThread, claimed two execution contexts
where there are three, and cited MediaPipeline.add_stream() and
SippyEngine.bridge() — neither of which exists. Corrected, with the data flow
now showing the emergency guard and concurrency cap in their real positions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three thread domains were mutating shared dicts with no locks: Sippy's
ED thread wrote _legs/_registered_devices directly from SIP handlers,
the asyncio loop wrote them from make_call/hangup, and
run_in_executor(None, ...) had default-pool threads driving sippy UA
objects. AudioTap.feed() pushed into an asyncio.Queue (not
thread-safe) from the PJSUA2 thread.
New ownership rule, enforced structurally:
- The asyncio loop owns all app-visible state; the only mutator is the
new _on_engine_event funnel. Sippy handlers extract plain strings on
the ED thread and post via run_coroutine_threadsafe.
- The ED thread owns sippy objects plus _ed_ua_to_leg/_ed_leg_to_ua;
loop-side commands (INVITE/BYE/DTMF/trunk register) hop over via
ED2.callFromThread. UA references no longer live on SipCallLeg.
- AudioTap captures its loop and feed() hops via call_soon_threadsafe.
- Fix ED import: installed sippy 2.x exposes ED2, not ED — the old
import could never start the event loop.
Also:
- Wire the never-connected on_leg_state_change callback: outbound
ringing/connected/terminated now reaches CallManager; a call ends
when its last leg terminates (transfers keep it alive). Adds
CallManager.unmap_leg/legs_for_call.
- AudioClassifier.classify(): async entry that runs the FFT work in
asyncio.to_thread and updates history on the loop — all four
hold_slayer call sites now route through it, fixing both the
loop-blocking and the 2-of-4 history gap. DTMF Goertzel loop
replaced by the equivalent vectorized DFT-bin power.
- Task hygiene: gateway.spawn() tracks hold-slayer/receptionist tasks
and stop() cancels them; recording safety-timeout task is retained
and cancelled on stop_recording; engine tracks incoming-call
dispatch tasks.
10 new tests: funnel events from a foreign thread, auto-answer
fallback, AudioTap cross-thread feed, classifier history, leg-state →
call status (including no stomping of ON_HOLD), stop() cancellation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds read-only access to persisted call records for the dashboard
and implements a client for the Rhema text-to-speech service.
- api/call_history.py: New router providing paged call lists
and detailed call records with transcript metadata.
- services/tts.py: Async client for OpenAI-compatible TTS
endpoints (Rhema/Kokoro) used for call-flow steps.
Complete project scaffolding and core implementation of an AI-powered
telephony system that calls companies, navigates IVR menus, waits on
hold, and transfers to the user when a human answers.
Key components:
- FastAPI server with REST API, WebSocket, and MCP (SSE) interfaces
- SIP/VoIP call management via PJSUA2 with RTP audio streaming
- LLM-powered IVR navigation using OpenAI/Anthropic with tool calling
- Hold detection service combining audio analysis and silence detection
- Real-time STT (Whisper/Deepgram) and TTS (OpenAI/Piper) pipelines
- Call recording with per-channel and mixed audio capture
- Event bus (asyncio pub/sub) for real-time client updates
- Web dashboard with live call monitoring
- SQLite persistence via SQLAlchemy with call history and analytics
- Notification support (email, SMS, webhook, desktop)
- Docker Compose deployment with Opal VoIP and Opal Media containers
- Comprehensive test suite with unit, integration, and E2E tests
- Simplified .gitignore and full project documentation in README
