hold-slayer/docs/deployment-validation-plan.md

Hold Slayer — Deployment & Validation Plan

Bring the gateway up in stages, proving each layer before the one above it can lie about working. The ordering principle: nothing touches the PSTN until everything that can be validated without it has been. A bug found on the trunk costs money and rings a stranger's phone; the same bug found internally costs nothing.

Phases gate on each other. Do not start a phase until its predecessor's exit criteria are all green.

---

Environment as found (2026-07-28)

Verified on caliban, not assumed:

Fact	State	Consequence
sippy 2.3.0	installed	SIP signaling is real
pjsua2	✅ built 2026-07-28 (pjproject 2.17)	real media pipeline; see pjsua2-build.md
USE_MOCK_SIP	true	engine is MockSIPEngine; /health can never be healthy
SIP_TRUNK_*	placeholders (sip.yourprovider.com)	no trunk configured
SIP_TRUNK_DID	+16472474242	real DID already allocated
GATEWAY_SIP_PORT	5060	.env.example says 5080 — reconcile
DATABASE_URL	portia.incus:5432/hold_slayer	remote Postgres, reachable? unverified
Casdoor	enabled, hold-slayer app, OWNER_NAME=r@helu.ca	SSO path is configured
SPEACHES_URL	pan.helu.ca:22070	not reachable from this host as tested
LLM_BASE_URL	nyx.helu.ca:29000	not reachable from this host as tested
TTS_*	unset → defaults to localhost:8000	collides with the app's own port — must be set
pytest	159 pass, 6 fail	4 are .env bleed, 2 are real (see Phase 0)

Two config landmines worth fixing before anything else: TTS_BASE_URL is unset and defaults to http://localhost:8000, which is Hold Slayer's own port — TTS calls would loop back into the gateway. And API_TOKEN still sits in .env though the code now uses Casdoor + PATs; it is dead weight that suggests the old auth path still exists.

---

Phase 0 — Baseline: make the test suite an honest gate ✅ COMPLETE

Done 2026-07-28. Suite is 165 passed / 0 failed from the repo root with the real .env present. Outcomes differed from the plan's guesses in one important way: the classifier failure was a real bug in classify_chunk, not a threshold or fixture problem — see item 3. One item is deferred with rationale (ruff, below).

You cannot use a suite with known failures as a regression gate; every later phase's "did I break something" check depends on a clean baseline.

Do:

1. Fix the .env bleed in tests/test_oauth_metadata.py. The 4 failures are entirely because pydantic reads the real .env and PUBLIC_BASE_URL=http://localhost:21081 overrides the test's expected http://test. Confirmed: the same tests pass when run from a directory without .env. Fix by monkeypatching get_settings().public_base_url = "" in the client fixture, so tests derive the base URL from request headers as intended.

2. Resolve tests/test_hold_slayer.py::TestMockSIPEngine::test_trunk_status. The test is wrong, not the code. It asserts the mock trunk reports registered is True after start(), but MockSIPEngine.get_trunk_status() deliberately returns registered: False with reason: "No SIP trunk configured (mock mode)" — which is exactly what the /health-tells-the-truth invariant requires. Update the test to assert False + the reason.

3. Triage test_audio_classifier.py::test_complex_tone_as_music. Outcome: the test was right and the code had a real bug. Measured feature values for the C-major chord: music_score = 0.850, comfortably above the 0.7 music_threshold — music was detected confidently. But step 5's if speech_score > music_score ran first and short-circuited, because speech_score = 1.000: the four speech bands are wide and overlapping (flatness 0.223 ∈ (0.1,0.5), centroid 852 ∈ (500,4000), ZCR 0.039 ∈ (0.02,0.2), RMS 0.183 ∈ (0.01,0.5)) and a sustained musical chord satisfies all four trivially. Verified structural, not a one-off — C major, A minor and a bright chord all classified live_human despite scoring 0.85/0.65 music.

   Fixed in services/audio_classifier.py by making the speech branch yield to a confident music score: if speech_score > music_score and music_score < self.settings.music_threshold: One line, both thresholds stay meaningful, scorers untouched. Chords now classify music (0.85) and all 18 classifier tests pass, so speech-like audio still classifies as speech.

   This was the plan's highest-value find: the bug is exactly the operational failure that rings your desk for a hold queue. Worth re-validating against real hold music in Phase 2 — the synthetic "bright chord" case scores 0.65, below threshold, so it still reads as speech.

Exit criteria: pytest tests/ -v green from the repo root with the real .env present ✅ (165 passed). ruff check . clean — deferred, see below.

Deferred: ruff check . reports 217 errors — all pre-existing. Verified identical (217) with my changes stashed, so nothing here was introduced by Phase 0, and the three files I touched add none. The backlog is stylistic and repo-wide: 157 × UP045 (Optional[X] → X | None), 18 × F401 unused imports, 14 × E501, spread across services/ (77), models/ (75) and core/ (49). Auto-fixing would rewrite annotations throughout sippy_engine.py and media_pipeline.py — the thread-boundary and media code that currently cannot be exercised (no PJSUA2, no trunk). That is a large unverifiable diff in the highest-risk files, unrelated to making the suite an honest gate. Recommend its own commit, ideally after Phase 1b restores the ability to actually run that code.

---

Phase 1a — Internal signaling with a registered softphone (no audio)

Prove the SIP stack, registration, dial plan, and call lifecycle with zero media and zero PSTN. This is the phase that de-risks the most for the least.

Config:

USE_MOCK_SIP=false          # real Sippy engine
SIP_TRUNK_HOST=             # blank — engine skips _register_trunk()
GATEWAY_SIP_HOST=0.0.0.0
GATEWAY_SIP_PORT=5060
GATEWAY_SIP_DOMAIN=voip.helu.ca

Leaving SIP_TRUNK_HOST blank is deliberate and load-bearing: SippyEngine.start() only calls _register_trunk() if self._trunk_host, so the gateway comes up listening for devices with no possible path to the PSTN. That is the strongest safety guarantee available in this phase — not policy, but absence of a route.

Setup:

1. Register a softphone (Linphone or Zoiper on a laptop; Groundwire on mobile) to <caliban-ip>:5060, any username, domain voip.helu.ca. Expect registration to succeed with any credentials — see the security finding below.
2. Confirm the device appears: GET /api/v1/devices and the list_devices MCP tool, plus the 📱 SIP REGISTER line in the log.
3. Create a Device row for it (type: sip_phone, sip_uri matching the registered contact) so call_device() can target it.

Validate:

• REGISTER lands, appears in _registered_devices, refreshes on re-REGISTER, and disappears on expires=0 (hang up / unregister in the softphone).
• Inbound INVITE from the softphone to the gateway surfaces an incoming_invite event and routes through the receptionist path.
• gateway.call_device() toward the softphone makes it ring — this exercises _call_sip_device, SDP generation, and the loop→Sippy funnel.
• Answer, then hang up from each end in turn; confirm terminated propagates and the leg is cleaned from _legs.
• DTMF from the softphone appears in the log (note: received DTMF currently has no consumer — it logs and stops).
• /health reports degraded with sip_trunk.registered: false. This is correct — do not chase a green light here.

Security finding to address in this phase: _handle_incoming_register() replies 200 OK to any REGISTER with no authentication challenge — no 401 with a nonce, no digest verification. Any host that can reach port 5060 can register as any AOR and become a transfer target. On a LAN-only bind for this phase that's tolerable; before Phase 3 exposes the gateway to a trunk it is not. Either add digest auth to the REGISTER path or firewall 5060 to known device IPs. Flagging rather than folding in — it's a real change to the SIP path and your call how to scope it.

Exit criteria: softphone registers, rings on call_device, both-direction teardown is clean, no leaked legs after 10 call cycles, suite still green.

---

Phase 1b — Gate: build PJSUA2 ✅ COMPLETE

Done 2026-07-28. Built out of order (ahead of Phase 1a) because nothing useful works without it. pjproject 2.17 — newer than expected, with Python 3.13 and modern-gcc support already upstream, so no patching was needed.

MediaPipeline.status() now reports pjsua2_available: True; the stub-mode warning is gone and the pipeline starts and stops cleanly at 16 kHz. Full procedure recorded in pjsua2-build.md; README note now points at it.

Notes worth carrying forward:

• No sudo was needed. Both missing tools (swig, patchelf) have PyPI wheels and installed into the venv, so nothing on the host changed outside ~/src and ~/.local.
• The RPATH step is the non-obvious part. The bindings compile and install cleanly and then fail at import with ImportError: libpjsua2.so.2: cannot open shared object file, because ~/.local/lib isn't on the loader path. Fixed with patchelf --set-rpath on both the extension and all 12 libpj*.so.2 libraries — patching only the extension just surfaces the transitive deps one layer down. Chosen over LD_LIBRARY_PATH because that would have to be set for uvicorn, systemd, cron and every subprocess, and it fails at call time rather than startup. Verified under env -i from /, so it depends on no inherited environment.
• Configure found OpenSSL, ALSA and Opus — the full codec/TLS surface.
• 165 tests still pass. All pjsua2 imports in the codebase are lazy (inside functions), so the suite still runs without touching real media.

Exit criteria: import pjsua2 succeeds in the venv ✅; MediaPipeline reports pjsua2_available: true ✅ (note the method is status(), not get_status() as this plan originally said).

Two caveats this build does not solve:

1. Not captured by pip install -e ".[dev]". It lives outside Python packaging metadata — a fresh venv, new host or rebuilt container needs it repeated. Host provisioning, not a dependency.
2. The Docker image still runs stub media — the Dockerfile deliberately skips this build. So the Phase 2 audio validation must run outside the container, or the Dockerfile needs a pjproject build stage. Worth deciding before Phase 2, since it determines where you test.

---

Phase 2 — Audio, TTS, STT, and the classifier on internal calls

Now the softphone is a full test instrument: you can speak into the gateway and hear it speak back, with no telephony charges.

Config to fix first:

TTS_BASE_URL=<real Rhema endpoint>   # NOT localhost:8000 — that's this app
TTS_MODEL=speaches-ai/Kokoro-82M-v1.0-ONNX
SPEACHES_URL=http://pan.helu.ca:22070

Validate leaf services standalone before wiring them into a call — a failure here is much easier to read outside the call path:

• TTSService.synthesize() returns PCM directly (a short script against the service). Confirm service.available flips to True and /health reports tts: ok.
• TranscriptionService.transcribe() against a known WAV returns expected text; /health reports stt: ok.
• Both are reachable from wherever the gateway actually runs — neither pan.helu.ca:22070 nor nyx.helu.ca:29000 answered from this host during assessment. Resolve that before blaming the call path.
• LLM: nyx.helu.ca:29000 with Qwen3.6-35B-A3B-UD-Q4_K_XL. Note the URL has no /v1 suffix while the default does — verify which the client expects.

Then in-call, softphone ↔ gateway only:

• Receptionist answers an inbound call from the softphone: you hear the TTS greeting, speak, and your speech is transcribed. This single test exercises TTS + STT + LLM + media in the real path.
• Recording writes a real file to recordings/ with actual audio in it (check the file plays, not just that it exists — the stub path created files too).
• Classifier: play hold music from a phone/laptop into the call and confirm MUSIC; speak and confirm LIVE_HUMAN. This is the ground truth that Phase 0's failing classifier test was gesturing at.
• Transcript persists to the DB on hangup and is readable via get_call_transcript.

Exit criteria: a full receptionist conversation over the softphone with audible TTS, accurate STT, correct classification, a playable recording, and a persisted transcript.

---

Phase 3 — MCP integration

Independent of audio, so it can run in parallel with 1b/2 if convenient.

• Mint a PAT via api/tokens.py, confirm hs_pat_… format.
• Connect an MCP client to https://<host>/mcp/ and confirm OAuth discovery (/.well-known/oauth-protected-resource) resolves against the real PUBLIC_BASE_URL.
• Read-only tools first: gateway_status, list_active_calls, list_devices, search_call_history, and the three resources.
• Verify the auth boundary negatively: a non-owner identity and a bogus PAT both get 403/401 on /mcp/, REST, and WS. Owner-only is an invariant; prove it rather than assuming it.
• make_call against the softphone (internal, no trunk) — confirms the MCP path reaches the gateway.
• Test the emergency guard through MCP specifically: make_call("911"), "9911", "112", "+1911", and with whitespace/dashes must every one raise ToolError before any SIP action. Do this while SIP_TRUNK_HOST is still blank, so a guard failure cannot become an actual emergency call. This is the single most important test in the plan — run it before Phase 4, never after.

Exit criteria: all read tools return sane data, make_call rings the softphone, every emergency variant is refused, non-owner is refused.

---

Phase 4 — SIP trunk (first PSTN contact)

Only now does real money and a real network get involved.

1. Set SIP_TRUNK_HOST/PORT/USERNAME/PASSWORD/TRANSPORT for the provider; keep SIP_TRUNK_DID=+16472474242.
2. Restart and watch registration: _register_trunk() posts trunk_registered, /health should flip to healthy — real engine + registered trunk + reachable DB. This is the first moment a green /health is meaningful.
3. Re-run the entire emergency-guard suite from Phase 3 now that a real route exists. The guard is what stands between an AI agent and a 911 dispatcher with no E911 location attached.
4. First outbound call: dial your own mobile, nothing else. Confirm ring, answer, two-way audio, clean teardown, DB persistence.
5. First inbound: call the DID from your mobile, receptionist answers, routing and transfer-to-softphone work.
6. Then a real hold scenario against a known IVR with a long queue.

Safety rails while validating:

• Drop MAX_CONCURRENT_CALLS to 1 for first calls; restore to 4 after.
• Keep MAX_HOLD_TIME low initially — 7200s is two hours of trunk time if something wedges.
• Watch the provider's billing/CDR page live during the first calls.
• Have the provider portal open to kill calls out-of-band.

Exit criteria: /health genuinely healthy, one successful outbound to a known number, one inbound handled, CDRs match expectations, no orphaned legs.

---

Cross-cutting: .env.example drift ✅ FIXED (Phase 0)

.env.example had documented the removed API_TOKEN as the auth mechanism and omitted every CASDOOR_* var, OWNER_NAME, PUBLIC_BASE_URL, and the whole TTS_* block. Rewritten to match config.py: dropped API_TOKEN, added the Casdoor/auth block (documenting both supported configurations and why SSO-off off-loopback is refused), the full TTS_* and RECEPTIONIST_* blocks, and reconciled GATEWAY_SIP_PORT to 5060 to match .env.

TTS_BASE_URL now defaults to localhost:8001 in the template with a comment warning that it must not equal the app's own PORT — this was blocker #3, and the template previously would have reproduced it for any fresh checkout.

Validated by copying the template to a clean directory and parsing it through Settings(): every sub-config loads, and tts.base_url != localhost:{port} is asserted. grep confirms no API_TOKEN references remain in code, templates or docs. The README config table was already current — no drift there.

---

Summary of blockers found

#	Blocker	Blocks	Severity
1	PJSUA2 not installed	all audio: TTS/STT/recording/classifier in-call	✅ fixed — pjproject 2.17 built, pjsua2_available: True
2	REGISTER accepts any credentials, no digest auth	safe exposure of port 5060	security
3	TTS_BASE_URL unset → defaults to app's own port	TTS entirely	config — ✅ fixed in template; still set it in your real .env
4	STT/LLM endpoints unreachable as tested	Phase 2	environment
5	6 failing tests (2 real, 4 .env bleed)	honest regression gate	✅ fixed — 165 pass
6	.env.example documents removed API_TOKEN auth	fresh-environment test	✅ fixed
7	Hold music classified as LIVE_HUMAN (found during Phase 0)	correct hold detection	✅ fixed — real bug, see Phase 0 item 3

Note on #3: the fix landed in .env.example (the template). Your live .env still has no TTS_* block at all, so TTS resolves to the localhost:8000 default and collides with the app. Set TTS_BASE_URL before Phase 2.