Robert Helewka
394e3fc920
Add comprehensive deployment and validation plan documenting a staged bring-up approach that gates each layer on its predecessor and defers PSTN testing until everything else is proven. Update .env.example to reflect current configuration: - Replace static API_TOKEN with Casdoor SSO + owner-minted PAT auth - Add Rhema TTS settings with port-collision warning - Add AI Receptionist settings for inbound calls - Reconcile GATEWAY_SIP_PORT to 5060 and default SIP domain
316 lines
16 KiB
Markdown
316 lines
16 KiB
Markdown
# Hold Slayer — Deployment & Validation Plan
|
||
|
||
Bring the gateway up in stages, proving each layer before the one above it can
|
||
lie about working. The ordering principle: **nothing touches the PSTN until
|
||
everything that can be validated without it has been.** A bug found on the trunk
|
||
costs money and rings a stranger's phone; the same bug found internally costs
|
||
nothing.
|
||
|
||
Phases gate on each other. Do not start a phase until its predecessor's exit
|
||
criteria are all green.
|
||
|
||
---
|
||
|
||
## Environment as found (2026-07-28)
|
||
|
||
Verified on `caliban`, not assumed:
|
||
|
||
| Fact | State | Consequence |
|
||
|---|---|---|
|
||
| `sippy` 2.3.0 | installed | SIP signaling is real |
|
||
| `pjsua2` | **not installed** | media pipeline is a **stub** — signaling works, audio does not |
|
||
| `USE_MOCK_SIP` | `true` | engine is `MockSIPEngine`; `/health` can never be `healthy` |
|
||
| `SIP_TRUNK_*` | placeholders (`sip.yourprovider.com`) | no trunk configured |
|
||
| `SIP_TRUNK_DID` | `+16472474242` | real DID already allocated |
|
||
| `GATEWAY_SIP_PORT` | `5060` | `.env.example` says 5080 — reconcile |
|
||
| `DATABASE_URL` | `portia.incus:5432/hold_slayer` | remote Postgres, reachable? unverified |
|
||
| Casdoor | enabled, `hold-slayer` app, `OWNER_NAME=r@helu.ca` | SSO path is configured |
|
||
| `SPEACHES_URL` | `pan.helu.ca:22070` | not reachable from this host as tested |
|
||
| `LLM_BASE_URL` | `nyx.helu.ca:29000` | not reachable from this host as tested |
|
||
| `TTS_*` | unset → defaults to `localhost:8000` | **collides with the app's own port** — must be set |
|
||
| `pytest` | 159 pass, 6 fail | 4 are `.env` bleed, 2 are real (see Phase 0) |
|
||
|
||
Two config landmines worth fixing before anything else: `TTS_BASE_URL` is unset
|
||
and defaults to `http://localhost:8000`, which is Hold Slayer's own port — TTS
|
||
calls would loop back into the gateway. And `API_TOKEN` still sits in `.env`
|
||
though the code now uses Casdoor + PATs; it is dead weight that suggests the old
|
||
auth path still exists.
|
||
|
||
---
|
||
|
||
## Phase 0 — Baseline: make the test suite an honest gate ✅ COMPLETE
|
||
|
||
**Done 2026-07-28.** Suite is 165 passed / 0 failed from the repo root with the
|
||
real `.env` present. Outcomes differed from the plan's guesses in one important
|
||
way: the classifier failure was a **real bug in `classify_chunk`**, not a
|
||
threshold or fixture problem — see item 3. One item is deferred with rationale
|
||
(`ruff`, below).
|
||
|
||
You cannot use a suite with known failures as a regression gate; every later
|
||
phase's "did I break something" check depends on a clean baseline.
|
||
|
||
**Do:**
|
||
1. Fix the `.env` bleed in `tests/test_oauth_metadata.py`. The 4 failures are
|
||
entirely because pydantic reads the real `.env` and `PUBLIC_BASE_URL=http://localhost:21081`
|
||
overrides the test's expected `http://test`. Confirmed: the same tests pass
|
||
when run from a directory without `.env`. Fix by monkeypatching
|
||
`get_settings().public_base_url = ""` in the `client` fixture, so tests derive
|
||
the base URL from request headers as intended.
|
||
2. Resolve `tests/test_hold_slayer.py::TestMockSIPEngine::test_trunk_status`.
|
||
**The test is wrong, not the code.** It asserts the mock trunk reports
|
||
`registered is True` after `start()`, but `MockSIPEngine.get_trunk_status()`
|
||
deliberately returns `registered: False` with
|
||
`reason: "No SIP trunk configured (mock mode)"` — which is exactly what the
|
||
`/health`-tells-the-truth invariant requires. Update the test to assert
|
||
`False` + the reason.
|
||
3. Triage `test_audio_classifier.py::test_complex_tone_as_music`. **Outcome: the
|
||
test was right and the code had a real bug.** Measured feature values for the
|
||
C-major chord: `music_score` = **0.850**, comfortably above the 0.7
|
||
`music_threshold` — music was detected confidently. But step 5's
|
||
`if speech_score > music_score` ran first and short-circuited, because
|
||
`speech_score` = **1.000**: the four speech bands are wide and overlapping
|
||
(flatness 0.223 ∈ (0.1,0.5), centroid 852 ∈ (500,4000), ZCR 0.039 ∈
|
||
(0.02,0.2), RMS 0.183 ∈ (0.01,0.5)) and a sustained musical chord satisfies
|
||
all four trivially. Verified structural, not a one-off — C major, A minor and
|
||
a bright chord *all* classified `live_human` despite scoring 0.85/0.65 music.
|
||
|
||
Fixed in `services/audio_classifier.py` by making the speech branch yield to a
|
||
confident music score:
|
||
`if speech_score > music_score and music_score < self.settings.music_threshold:`
|
||
One line, both thresholds stay meaningful, scorers untouched. Chords now
|
||
classify `music` (0.85) and all 18 classifier tests pass, so speech-like audio
|
||
still classifies as speech.
|
||
|
||
**This was the plan's highest-value find:** the bug is exactly the operational
|
||
failure that rings your desk for a hold queue. Worth re-validating against
|
||
*real* hold music in Phase 2 — the synthetic "bright chord" case scores 0.65,
|
||
below threshold, so it still reads as speech.
|
||
|
||
**Exit criteria:** `pytest tests/ -v` green from the repo root with the real
|
||
`.env` present ✅ (165 passed). `ruff check .` clean — **deferred, see below**.
|
||
|
||
**Deferred: `ruff check .` reports 217 errors — all pre-existing.** Verified
|
||
identical (217) with my changes stashed, so nothing here was introduced by
|
||
Phase 0, and the three files I touched add none. The backlog is stylistic and
|
||
repo-wide: 157 × `UP045` (`Optional[X]` → `X | None`), 18 × `F401` unused
|
||
imports, 14 × `E501`, spread across `services/` (77), `models/` (75) and
|
||
`core/` (49). Auto-fixing would rewrite annotations throughout `sippy_engine.py`
|
||
and `media_pipeline.py` — the thread-boundary and media code that currently
|
||
**cannot be exercised** (no PJSUA2, no trunk). That is a large unverifiable diff
|
||
in the highest-risk files, unrelated to making the suite an honest gate. Recommend
|
||
its own commit, ideally after Phase 1b restores the ability to actually run that
|
||
code.
|
||
|
||
---
|
||
|
||
## Phase 1a — Internal signaling with a registered softphone (no audio)
|
||
|
||
Prove the SIP stack, registration, dial plan, and call lifecycle with zero
|
||
media and zero PSTN. This is the phase that de-risks the most for the least.
|
||
|
||
**Config:**
|
||
```
|
||
USE_MOCK_SIP=false # real Sippy engine
|
||
SIP_TRUNK_HOST= # blank — engine skips _register_trunk()
|
||
GATEWAY_SIP_HOST=0.0.0.0
|
||
GATEWAY_SIP_PORT=5060
|
||
GATEWAY_SIP_DOMAIN=voip.helu.ca
|
||
```
|
||
Leaving `SIP_TRUNK_HOST` blank is deliberate and load-bearing: `SippyEngine.start()`
|
||
only calls `_register_trunk()` `if self._trunk_host`, so the gateway comes up
|
||
listening for devices with **no possible path to the PSTN**. That is the
|
||
strongest safety guarantee available in this phase — not policy, but absence of
|
||
a route.
|
||
|
||
**Setup:**
|
||
1. Register a softphone (Linphone or Zoiper on a laptop; Groundwire on mobile)
|
||
to `<caliban-ip>:5060`, any username, domain `voip.helu.ca`.
|
||
**Expect registration to succeed with any credentials** — see the security
|
||
finding below.
|
||
2. Confirm the device appears: `GET /api/v1/devices` and the `list_devices` MCP
|
||
tool, plus the `📱 SIP REGISTER` line in the log.
|
||
3. Create a `Device` row for it (`type: sip_phone`, `sip_uri` matching the
|
||
registered contact) so `call_device()` can target it.
|
||
|
||
**Validate:**
|
||
- REGISTER lands, appears in `_registered_devices`, refreshes on re-REGISTER,
|
||
and disappears on `expires=0` (hang up / unregister in the softphone).
|
||
- Inbound INVITE from the softphone to the gateway surfaces an `incoming_invite`
|
||
event and routes through the receptionist path.
|
||
- `gateway.call_device()` toward the softphone makes it **ring** — this
|
||
exercises `_call_sip_device`, SDP generation, and the loop→Sippy funnel.
|
||
- Answer, then hang up from each end in turn; confirm `terminated` propagates
|
||
and the leg is cleaned from `_legs`.
|
||
- DTMF from the softphone appears in the log (note: received DTMF currently has
|
||
no consumer — it logs and stops).
|
||
- `/health` reports `degraded` with `sip_trunk.registered: false`. **This is
|
||
correct** — do not chase a green light here.
|
||
|
||
**Security finding to address in this phase:** `_handle_incoming_register()`
|
||
replies `200 OK` to any REGISTER with no authentication challenge — no 401 with
|
||
a nonce, no digest verification. Any host that can reach port 5060 can register
|
||
as any AOR and become a transfer target. On a LAN-only bind for this phase
|
||
that's tolerable; **before Phase 3 exposes the gateway to a trunk it is not.**
|
||
Either add digest auth to the REGISTER path or firewall 5060 to known device
|
||
IPs. Flagging rather than folding in — it's a real change to the SIP path and
|
||
your call how to scope it.
|
||
|
||
**Exit criteria:** softphone registers, rings on `call_device`, both-direction
|
||
teardown is clean, no leaked legs after 10 call cycles, suite still green.
|
||
|
||
---
|
||
|
||
## Phase 1b — Gate: build PJSUA2
|
||
|
||
**Everything audio-shaped is blocked until this is done.** No TTS into a call,
|
||
no STT out of one, no recording, no hold-music classification on live audio.
|
||
`MediaPipeline` logs `⚠️ PJSUA2 not installed — media pipeline running in stub
|
||
mode` and every media call becomes a no-op that *returns successfully* — which
|
||
is the dangerous part, since it looks like it worked.
|
||
|
||
Build pjproject with Python bindings (`--enable-shared`, swig python2/3
|
||
bindings) into the `/home/robert/env/hold-slayer` venv. Budget real time for
|
||
this; it is a from-source build, not a pip install.
|
||
|
||
**Exit criteria:** `python -c "import pjsua2"` succeeds in the venv, and
|
||
`/health` / `media_pipeline.get_status()` reports `pjsua2_available: true`.
|
||
|
||
---
|
||
|
||
## Phase 2 — Audio, TTS, STT, and the classifier on internal calls
|
||
|
||
Now the softphone is a full test instrument: you can speak into the gateway and
|
||
hear it speak back, with no telephony charges.
|
||
|
||
**Config to fix first:**
|
||
```
|
||
TTS_BASE_URL=<real Rhema endpoint> # NOT localhost:8000 — that's this app
|
||
TTS_MODEL=speaches-ai/Kokoro-82M-v1.0-ONNX
|
||
SPEACHES_URL=http://pan.helu.ca:22070
|
||
```
|
||
|
||
**Validate leaf services standalone before wiring them into a call** — a failure
|
||
here is much easier to read outside the call path:
|
||
- `TTSService.synthesize()` returns PCM directly (a short script against the
|
||
service). Confirm `service.available` flips to `True` and `/health` reports
|
||
`tts: ok`.
|
||
- `TranscriptionService.transcribe()` against a known WAV returns expected text;
|
||
`/health` reports `stt: ok`.
|
||
- Both are reachable **from wherever the gateway actually runs** — neither
|
||
`pan.helu.ca:22070` nor `nyx.helu.ca:29000` answered from this host during
|
||
assessment. Resolve that before blaming the call path.
|
||
- LLM: `nyx.helu.ca:29000` with `Qwen3.6-35B-A3B-UD-Q4_K_XL`. Note the URL has
|
||
no `/v1` suffix while the default does — verify which the client expects.
|
||
|
||
**Then in-call, softphone ↔ gateway only:**
|
||
- Receptionist answers an inbound call from the softphone: you hear the TTS
|
||
greeting, speak, and your speech is transcribed. This single test exercises
|
||
TTS + STT + LLM + media in the real path.
|
||
- Recording writes a real file to `recordings/` with actual audio in it (check
|
||
the file plays, not just that it exists — the stub path created files too).
|
||
- Classifier: play hold music from a phone/laptop into the call and confirm
|
||
`MUSIC`; speak and confirm `LIVE_HUMAN`. This is the ground truth that
|
||
Phase 0's failing classifier test was gesturing at.
|
||
- Transcript persists to the DB on hangup and is readable via
|
||
`get_call_transcript`.
|
||
|
||
**Exit criteria:** a full receptionist conversation over the softphone with
|
||
audible TTS, accurate STT, correct classification, a playable recording, and a
|
||
persisted transcript.
|
||
|
||
---
|
||
|
||
## Phase 3 — MCP integration
|
||
|
||
Independent of audio, so it can run in parallel with 1b/2 if convenient.
|
||
|
||
- Mint a PAT via `api/tokens.py`, confirm `hs_pat_…` format.
|
||
- Connect an MCP client to `https://<host>/mcp/` and confirm OAuth discovery
|
||
(`/.well-known/oauth-protected-resource`) resolves against the real
|
||
`PUBLIC_BASE_URL`.
|
||
- Read-only tools first: `gateway_status`, `list_active_calls`, `list_devices`,
|
||
`search_call_history`, and the three resources.
|
||
- **Verify the auth boundary negatively**: a non-owner identity and a bogus PAT
|
||
both get 403/401 on `/mcp/`, REST, and WS. Owner-only is an invariant; prove
|
||
it rather than assuming it.
|
||
- `make_call` against the softphone (internal, no trunk) — confirms the MCP
|
||
path reaches the gateway.
|
||
- **Test the emergency guard through MCP specifically**: `make_call("911")`,
|
||
`"9911"`, `"112"`, `"+1911"`, and with whitespace/dashes must every one raise
|
||
`ToolError` before any SIP action. Do this while `SIP_TRUNK_HOST` is still
|
||
blank, so a guard failure cannot become an actual emergency call. **This is
|
||
the single most important test in the plan** — run it before Phase 4, never
|
||
after.
|
||
|
||
**Exit criteria:** all read tools return sane data, `make_call` rings the
|
||
softphone, every emergency variant is refused, non-owner is refused.
|
||
|
||
---
|
||
|
||
## Phase 4 — SIP trunk (first PSTN contact)
|
||
|
||
Only now does real money and a real network get involved.
|
||
|
||
1. Set `SIP_TRUNK_HOST/PORT/USERNAME/PASSWORD/TRANSPORT` for the provider; keep
|
||
`SIP_TRUNK_DID=+16472474242`.
|
||
2. Restart and watch registration: `_register_trunk()` posts `trunk_registered`,
|
||
`/health` should flip to **`healthy`** — real engine + registered trunk +
|
||
reachable DB. This is the first moment a green `/health` is meaningful.
|
||
3. **Re-run the entire emergency-guard suite from Phase 3** now that a real
|
||
route exists. The guard is what stands between an AI agent and a 911
|
||
dispatcher with no E911 location attached.
|
||
4. First outbound call: dial **your own mobile**, nothing else. Confirm ring,
|
||
answer, two-way audio, clean teardown, DB persistence.
|
||
5. First inbound: call the DID from your mobile, receptionist answers, routing
|
||
and transfer-to-softphone work.
|
||
6. Then a real hold scenario against a known IVR with a long queue.
|
||
|
||
**Safety rails while validating:**
|
||
- Drop `MAX_CONCURRENT_CALLS` to `1` for first calls; restore to 4 after.
|
||
- Keep `MAX_HOLD_TIME` low initially — 7200s is two hours of trunk time if
|
||
something wedges.
|
||
- Watch the provider's billing/CDR page live during the first calls.
|
||
- Have the provider portal open to kill calls out-of-band.
|
||
|
||
**Exit criteria:** `/health` genuinely `healthy`, one successful outbound to a
|
||
known number, one inbound handled, CDRs match expectations, no orphaned legs.
|
||
|
||
---
|
||
|
||
## Cross-cutting: `.env.example` drift ✅ FIXED (Phase 0)
|
||
|
||
`.env.example` had documented the removed `API_TOKEN` as the auth mechanism and
|
||
omitted every `CASDOOR_*` var, `OWNER_NAME`, `PUBLIC_BASE_URL`, and the whole
|
||
`TTS_*` block. Rewritten to match `config.py`: dropped `API_TOKEN`, added the
|
||
Casdoor/auth block (documenting both supported configurations and why SSO-off
|
||
off-loopback is refused), the full `TTS_*` and `RECEPTIONIST_*` blocks, and
|
||
reconciled `GATEWAY_SIP_PORT` to 5060 to match `.env`.
|
||
|
||
`TTS_BASE_URL` now defaults to `localhost:8001` in the template with a comment
|
||
warning that it must not equal the app's own `PORT` — this was blocker #3, and
|
||
the template previously would have reproduced it for any fresh checkout.
|
||
|
||
Validated by copying the template to a clean directory and parsing it through
|
||
`Settings()`: every sub-config loads, and `tts.base_url != localhost:{port}` is
|
||
asserted. `grep` confirms no `API_TOKEN` references remain in code, templates or
|
||
docs. The README config table was already current — no drift there.
|
||
|
||
---
|
||
|
||
## Summary of blockers found
|
||
|
||
| # | Blocker | Blocks | Severity |
|
||
|---|---|---|---|
|
||
| 1 | PJSUA2 not installed | all audio: TTS/STT/recording/classifier in-call | **hard gate** |
|
||
| 2 | REGISTER accepts any credentials, no digest auth | safe exposure of port 5060 | **security** |
|
||
| 3 | `TTS_BASE_URL` unset → defaults to app's own port | TTS entirely | config — ✅ fixed in template; **still set it in your real `.env`** |
|
||
| 4 | STT/LLM endpoints unreachable as tested | Phase 2 | environment |
|
||
| 5 | 6 failing tests (2 real, 4 `.env` bleed) | honest regression gate | ✅ **fixed** — 165 pass |
|
||
| 6 | `.env.example` documents removed `API_TOKEN` auth | fresh-environment test | ✅ **fixed** |
|
||
| 7 | Hold music classified as `LIVE_HUMAN` (found during Phase 0) | correct hold detection | ✅ **fixed** — real bug, see Phase 0 item 3 |
|
||
|
||
**Note on #3:** the fix landed in `.env.example` (the template). Your live
|
||
`.env` still has no `TTS_*` block at all, so TTS resolves to the
|
||
`localhost:8000` default and collides with the app. Set `TTS_BASE_URL` before
|
||
Phase 2.
|