From e2051f7486cfaa2892994b74e46d2815c7231000 Mon Sep 17 00:00:00 2001 From: Robert Helewka Date: Thu, 30 Jul 2026 17:57:31 -0400 Subject: [PATCH] docs: document SIP_ENGINE and correct config tables against the models MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Started as the SIP_ENGINE row flagged in the last commit. Cross-checking the tables against config.py mechanically (rather than by eye) turned up more, including two entries that were actively wrong. Corrections: - GATEWAY_RTP_PORT_MIN/MAX and GATEWAY_HOST are documented in configuration.md but do not exist — no code reads them and they are absent from .env.example. Setting them today does nothing. Replaced with the real GATEWAY_SIP_ fields (host/port/domain). - GATEWAY_SIP_PORT was documented as 5080 in two places; the code and .env.example both say 5060. - DATABASE_URL was documented with a SQLite default. There is none, and startup exits if it is unset. Additions — every env var the models accept is now documented somewhere (verified bidirectionally: nothing in the models undocumented, nothing documented that the models reject): - Server section: HOST, PORT, DEBUG, LOG_LEVEL, LOG_FORMAT - Safety section: MAX_CONCURRENT_CALLS, USE_MOCK_SIP, SIP_ENGINE - Receptionist section (configuration.md had none, though seven vars exist) Structural staleness, from the PR #8 media-plane work: - core/pjsua_engine.py was absent from the component list and file tree; so were dial_plan.py (the emergency guard) and sip_engine.py. - architecture.md's banner still read "media plane in transition". The engine landed; it is now two selectable engines with the audio consequence stated. - Tech Stack described "single-process async architecture" — the simplification CLAUDE.md explicitly calls out. Now points at the threading model, since there are three execution contexts. - The Asterisk lab shipped in PR #8 with its own README but nothing linked to it. Linked from the test section and both doc indexes. - CLAUDE.md's "no structured JSON logging" gap is closed; test count was 146 across 16 files, now 189 across 19. The other listed gaps (no /metrics, no rate limiting, no health-probe log filter) were re-verified and still hold. Deliberately not hardcoding a test count in the README — that is the same staleness this commit is clearing up. All internal links and anchors verified to resolve; 189 tests pass; lint unchanged at its 216 baseline. Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 12 +++++++----- README.md | 37 +++++++++++++++++++++++++++++------- docs/README.md | 3 +++ docs/architecture.md | 11 ++++++----- docs/configuration.md | 44 ++++++++++++++++++++++++++++++++++++++----- 5 files changed, 85 insertions(+), 22 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 35e6072..808d92d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -68,7 +68,7 @@ REST / WS / MCP / Dashboard (asyncio, FastAPI) 4. **Consistent Patterns** — config via pydantic-settings sub-configs; MCP tools return formatted strings; REST returns Pydantic models; DB access via `session_scope()`. Match the neighbours. -5. **Actually Works** — `pytest tests/ -v` (146 tests across 16 files). A change +5. **Actually Works** — `pytest tests/ -v` (189 tests across 19 files). A change to call placement, routing, the classifier, or auth needs a test. The suite runs against SQLite (`aiosqlite`) and the mock SIP engine — no trunk, no Postgres required to test. @@ -203,10 +203,12 @@ Surfaced honestly so you don't rediscover them as surprises. The README's Phase 4/5/6 checklists track most of these; **don't fold fixes into unrelated work** — raise them. -- **No structured JSON logging.** Logging is plain `logging.basicConfig` in - [main.py](main.py); there's no `LOG_FORMAT`/JSON path (README Phase 4 has this - unchecked). If Heluca observability wants JSON logs shipped to a collector, - that's a deliberate piece of work, not a drive-by. +- ~~**No structured JSON logging.**~~ Done: `LOG_FORMAT=json` in + [core/logging_config.py](core/logging_config.py), applied at import and again + in `lifespan` because uvicorn installs its own handlers (`propagate=False`) + after importing the app. The access log is included, with `status_code` as a + number so Loki can range-filter it. Text remains the default; the Docker image + sets json. - **No `/metrics` endpoint and no Prometheus.** Unlike the metrics-bearing estate services, there's no exposition endpoint here yet. - **No health-probe access-log filter.** Every `/health` poll hits the access diff --git a/README.md b/README.md index a517f81..d12fdc2 100644 --- a/README.md +++ b/README.md @@ -52,8 +52,9 @@ You give it a phone number and an intent ("dispute a charge on my December state ## What's Implemented ### Core Engine -- **Sippy B2BUA Engine** (`core/sippy_engine.py`) — SIP call control, DTMF, bridging, conference, trunk registration -- **PJSUA2 Media Pipeline** (`core/media_pipeline.py`) — Audio routing, recording ports, conference bridge, WAV playback (stub mode until the `pjsua2` bindings are installed — see note below) +- **Sippy B2BUA Engine** (`core/sippy_engine.py`) — SIP call control, DTMF, bridging, conference, trunk registration. Signalling only: no audio reaches the classifier on this path +- **PJSUA2 SIP Engine** (`core/pjsua_engine.py`) — Places the call itself so it owns the dialog, which is the only way PJSUA2 will surface RTP. Select with `SIP_ENGINE=pjsua2`; see [docs/architecture.md](docs/architecture.md#media-plane-why-pjsua2-places-the-call) +- **PJSUA2 Media Pipeline** (`core/media_pipeline.py`) — Audio routing, capture ports, conference bridge, WAV playback (stub mode until the `pjsua2` bindings are installed — see note below) - **Call Manager** (`core/call_manager.py`) — Active call state tracking, lifecycle management - **Event Bus** (`core/event_bus.py`) — Async pub/sub with per-subscriber queues, type filtering, history @@ -99,8 +100,12 @@ hold-slayer/ ├── config.py # Pydantic settings from .env ├── core/ │ ├── gateway.py # Top-level gateway orchestrator -│ ├── sippy_engine.py # Sippy B2BUA SIP engine +│ ├── dial_plan.py # Emergency-number guard + number normalisation +│ ├── sip_engine.py # SIPEngine ABC + MockSIPEngine +│ ├── sippy_engine.py # Sippy B2BUA SIP engine (signalling only) +│ ├── pjsua_engine.py # PJSUA2 SIP engine (call control + media) │ ├── media_pipeline.py # PJSUA2 audio routing +│ ├── logging_config.py # Text/JSON log formatting │ ├── call_manager.py # Active call state management │ └── event_bus.py # Async pub/sub event bus ├── services/ @@ -213,6 +218,15 @@ uvicorn main:app --host 0.0.0.0 --port 8000 pytest tests/ -v ``` +No external services required — the suite runs against SQLite (`aiosqlite`) and +the mock SIP engine, so it needs neither a trunk nor PostgreSQL. + +For the parts a unit test cannot reach — real SIP signalling, RTP, IVR +navigation against a live switch — there is an **Asterisk lab**: a fake PSTN +that answers calls, plays hold music, and runs scripted IVR menus, so call +paths can be exercised without dialling a real number or incurring telephony +charges. See [tests/lab/README.md](tests/lab/README.md). + ## Docker A single image bundles the FastAPI process and the built dashboard (the node @@ -373,13 +387,19 @@ All configuration is via environment variables (see `.env.example`): | `OWNER_NAME` | Casdoor username of the single operator (owner) | — (required if SSO on) | | `PUBLIC_BASE_URL` | Public base URL for OAuth discovery (else derived from headers) | — | | `MAX_CONCURRENT_CALLS` | Cap on simultaneous outbound calls | `4` | +| `HOST` | Bind address (off-loopback requires SSO enabled) | `0.0.0.0` | +| `PORT` | Bind port | `8000` | +| `DEBUG` | SQLAlchemy echo + uvicorn reload | `false` | | `LOG_LEVEL` | Root log level (`debug`/`info`/`warning`/`error`) | `info` | | `LOG_FORMAT` | `text` (human-readable) or `json` (structured, for Loki) | `text` | +| `USE_MOCK_SIP` | Run the mock SIP engine — no real calls. Must be asked for | `false` | +| `SIP_ENGINE` | `sippy` (signalling only) or `pjsua2` (call control + media) | `sippy` | +| `NOTIFY_SMS_NUMBER` | SMS notification number (optional) | — | | `SIP_TRUNK_HOST` | Your SIP provider hostname | — | | `SIP_TRUNK_USERNAME` | SIP auth username | — | | `SIP_TRUNK_PASSWORD` | SIP auth password | — | | `SIP_TRUNK_DID` | Your phone number (E.164) | — | -| `GATEWAY_SIP_PORT` | Port for device registration | `5080` | +| `GATEWAY_SIP_PORT` | Port for device registration | `5060` | | `SPEACHES_URL` | Speaches/Whisper STT endpoint | `http://localhost:22070` | | `LLM_BASE_URL` | OpenAI-compatible LLM endpoint | `http://localhost:11434/v1` | | `LLM_MODEL` | Model name for IVR analysis | `llama3` | @@ -393,11 +413,11 @@ All configuration is via environment variables (see `.env.example`): ## Tech Stack -- **Python 3.12+** + **asyncio** — Single-process async architecture +- **Python 3.12+** + **asyncio** — Single process. Not single-threaded: the SIP stacks run their own event loops on separate OS threads, crossed only through defined funnels ([docs/architecture.md](docs/architecture.md#threading-model)) - **FastAPI** — REST API + WebSocket server - **SvelteKit** — Dashboard UI (built static, served by FastAPI at `/`) -- **Sippy B2BUA** — SIP call control and DTMF -- **PJSUA2** — Media pipeline, conference bridge, recording, WAV playback +- **Sippy B2BUA** — SIP call control and DTMF (`SIP_ENGINE=sippy`, signalling only) +- **PJSUA2** — Call control + media: conference bridge, capture ports, recording, WAV playback (`SIP_ENGINE=pjsua2`) - **Speaches** (Whisper) — Speech-to-text - **Rhema** (Kokoro) — Text-to-speech (OpenAI-compatible `/v1/audio/speech`) - **Ollama / vLLM / OpenAI** — LLM for IVR menu analysis and receptionist intent capture @@ -410,6 +430,9 @@ Full documentation is in [`/docs`](docs/README.md): - [Architecture](docs/architecture.md) — System design, data flow, threading model - [Core Engine](docs/core-engine.md) — SIP engine, media pipeline, call manager, event bus +- [Dial Plan](docs/dial-plan.md) — Number normalisation and the emergency-number guard +- [PJSUA2 Build](docs/pjsua2-build.md) — Building the bindings (not pip-installable) +- [Asterisk Lab](tests/lab/README.md) — The fake PSTN used for media validation - [Hold Slayer Service](docs/hold-slayer-service.md) — IVR navigation, hold detection, human detection - [Audio Classifier](docs/audio-classifier.md) — Waveform analysis, feature extraction, classification - [Services](docs/services.md) — LLM client, transcription, recording, analytics, notifications diff --git a/docs/README.md b/docs/README.md index 0b2c7b7..bbf6e79 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,6 +8,9 @@ Comprehensive documentation for the Hold Slayer AI telephony gateway. |----------|-------------| | [Architecture](architecture.md) | System architecture, component diagram, data flow | | [Core Engine](core-engine.md) | SIP engine, media pipeline, call manager, event bus | +| [Dial Plan](dial-plan.md) | Number normalisation and the emergency-number guard | +| [PJSUA2 Build](pjsua2-build.md) | Building the `pjsua2` bindings (not pip-installable) | +| [Asterisk Lab](../tests/lab/README.md) | The fake PSTN used for media validation | | [Hold Slayer Service](hold-slayer-service.md) | IVR navigation, hold detection, human detection, transfer | | [Audio Classifier](audio-classifier.md) | Waveform analysis, feature extraction, classification logic | | [Services](services.md) | LLM client, transcription, recording, analytics, notifications | diff --git a/docs/architecture.md b/docs/architecture.md index 1d35c00..4aa63a0 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -4,11 +4,12 @@ Hold Slayer is a single-process async Python application built on FastAPI. It acts as an intelligent B2BUA (Back-to-Back User Agent) sitting between your SIP trunk (PSTN access) and your desk phone/softphone. -> **Media plane in transition.** The gateway currently signals with Sippy and -> intends PJSUA2 to carry media, but PJSUA2 will not surface an RTP stream for -> a dialog it does not own — so no audio ever reaches the classifier. The fix -> moves *call placement* into PJSUA2 while Sippy keeps the SBC roles. See -> [Media plane: why PJSUA2 places the call](#media-plane-why-pjsua2-places-the-call) +> **Two SIP engines, selected by `SIP_ENGINE`.** `sippy` (the default) signals +> only — PJSUA2 will not surface an RTP stream for a dialog it does not own, so +> **no audio reaches the classifier** on that path. `pjsua2` +> (`core/pjsua_engine.py`) places the call itself and is the only mode where +> audio reaches the classifier; it is opt-in while being proven against the lab. +> Read [Media plane: why PJSUA2 places the call](#media-plane-why-pjsua2-places-the-call) > before changing anything in `core/`. ## System Diagram diff --git a/docs/configuration.md b/docs/configuration.md index 87ceb2e..68ae501 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -34,14 +34,36 @@ SSO-disabled-off-loopback. | `SIP_TRUNK_DID` | Your phone number (E.164) | — | Yes | | `SIP_TRUNK_TRANSPORT` | Transport protocol (`udp`, `tcp`, `tls`) | `udp` | No | +### Server + +| Variable | Description | Default | Required | +|----------|-------------|---------|----------| +| `HOST` | Bind address. Off-loopback requires `CASDOOR_ENABLED=true` | `0.0.0.0` | No | +| `PORT` | Bind port | `8000` | No | +| `DEBUG` | SQLAlchemy echo + uvicorn reload | `false` | No | +| `LOG_LEVEL` | Root log level (`debug`/`info`/`warning`/`error`) | `info` | No | +| `LOG_FORMAT` | `text` (human-readable) or `json` (structured, for Loki) | `text` | No | + +`LOG_FORMAT=json` renders one JSON object per line, including uvicorn's access +log — `method`, `path`, `status_code` (numeric, so it can be range-filtered) and +`client_addr` arrive as queryable fields rather than a formatted string. The +Docker image sets it; `text` is the default so local development stays readable. + +### Safety + +| Variable | Description | Default | Required | +|----------|-------------|---------|----------| +| `MAX_CONCURRENT_CALLS` | Cap on simultaneous outbound calls | `4` | No | +| `USE_MOCK_SIP` | Run the mock SIP engine — no real calls. Must be asked for explicitly; an unconfigured trunk without it fails startup | `false` | No | +| `SIP_ENGINE` | `sippy` (signalling only — no audio reaches the classifier) or `pjsua2` (call control + media) | `sippy` | No | + ### Gateway | Variable | Description | Default | Required | |----------|-------------|---------|----------| -| `GATEWAY_SIP_PORT` | Port for device SIP registration | `5080` | No | -| `GATEWAY_RTP_PORT_MIN` | Minimum RTP port | `10000` | No | -| `GATEWAY_RTP_PORT_MAX` | Maximum RTP port | `20000` | No | -| `GATEWAY_HOST` | Bind address | `0.0.0.0` | No | +| `GATEWAY_SIP_HOST` | Bind address for the device-registration listener | `0.0.0.0` | No | +| `GATEWAY_SIP_PORT` | Port for device SIP registration | `5060` | No | +| `GATEWAY_SIP_DOMAIN` | SIP domain devices register against | `gateway.local` | No | ### LLM @@ -65,7 +87,7 @@ SSO-disabled-off-loopback. | Variable | Description | Default | Required | |----------|-------------|---------|----------| -| `DATABASE_URL` | PostgreSQL or SQLite connection string | `sqlite+aiosqlite:///./hold_slayer.db` | No | +| `DATABASE_URL` | PostgreSQL connection string. Startup exits with a readable error if unset | — | Yes | ### Notifications @@ -73,6 +95,18 @@ SSO-disabled-off-loopback. |----------|-------------|---------|----------| | `NOTIFY_SMS_NUMBER` | Phone number for SMS alerts (E.164) | — | No | +### Receptionist + +| Variable | Description | Default | Required | +|----------|-------------|---------|----------| +| `RECEPTIONIST_ENABLED` | Answer inbound calls with the AI receptionist | `true` | No | +| `RECEPTIONIST_GREETING_TEMPLATE` | Spoken greeting | `"Hi, you've reached Robert's line. Who's calling, and what's this about?"` | No | +| `RECEPTIONIST_MESSAGE_PROMPT` | Spoken prompt before recording a message | `"Please leave your message after the tone."` | No | +| `RECEPTIONIST_LLM_PERSONA` | System prompt shaping the receptionist's decisions | See `config.py` | No | +| `RECEPTIONIST_LISTEN_TIMEOUT_S` | Seconds to wait for the caller to speak | `15.0` | No | +| `RECEPTIONIST_END_OF_UTTERANCE_SILENCE_S` | Silence marking the end of a turn | `1.2` | No | +| `RECEPTIONIST_MESSAGE_MAX_SECONDS` | Voicemail cap | `90` | No | + ### Audio Classifier | Variable | Description | Default | Required |