hold-slayer/docs/deployment-validation-plan.md

# Hold Slayer — Deployment & Validation Plan

Bring the gateway up in stages, proving each layer before the one above it can
lie about working. The ordering principle: **nothing touches the PSTN until
everything that can be validated without it has been.** A bug found on the trunk
costs money and rings a stranger's phone; the same bug found internally costs
nothing.

Phases gate on each other. Do not start a phase until its predecessor's exit
criteria are all green.

---

## Environment as found (2026-07-28)

Verified on `caliban`, not assumed:

| Fact | State | Consequence |
|---|---|---|
| `sippy` 2.3.0 | installed | SIP signaling is real |
| `pjsua2` | ✅ **built 2026-07-28** (pjproject 2.17) | real media pipeline; see [pjsua2-build.md](pjsua2-build.md) |
| `USE_MOCK_SIP` | `true` | engine is `MockSIPEngine`; `/health` can never be `healthy` |
| `SIP_TRUNK_*` | placeholders (`sip.yourprovider.com`) | no trunk configured |
| `SIP_TRUNK_DID` | `+16472474242` | real DID already allocated |
| `GATEWAY_SIP_PORT` | `5060` | `.env.example` says 5080 — reconcile |
| `DATABASE_URL` | `portia.incus:5432/hold_slayer` | remote Postgres, reachable? unverified |
| Casdoor | enabled, `hold-slayer` app, `OWNER_NAME=r@helu.ca` | SSO path is configured |
| `SPEACHES_URL` | `pan.helu.ca:22070` | not reachable from this host as tested |
| `LLM_BASE_URL` | `nyx.helu.ca:29000` | not reachable from this host as tested |
| `TTS_*` | unset → defaults to `localhost:8000` | **collides with the app's own port** — must be set |
| `pytest` | 159 pass, 6 fail | 4 are `.env` bleed, 2 are real (see Phase 0) |

Two config landmines worth fixing before anything else: `TTS_BASE_URL` is unset
and defaults to `http://localhost:8000`, which is Hold Slayer's own port — TTS
calls would loop back into the gateway. And `API_TOKEN` still sits in `.env`
though the code now uses Casdoor + PATs; it is dead weight that suggests the old
auth path still exists.

---

## Phase 0 — Baseline: make the test suite an honest gate ✅ COMPLETE

**Done 2026-07-28.** Suite is 165 passed / 0 failed from the repo root with the
real `.env` present. Outcomes differed from the plan's guesses in one important
way: the classifier failure was a **real bug in `classify_chunk`**, not a
threshold or fixture problem — see item 3. One item is deferred with rationale
(`ruff`, below).

You cannot use a suite with known failures as a regression gate; every later
phase's "did I break something" check depends on a clean baseline.

**Do:**
1. Fix the `.env` bleed in `tests/test_oauth_metadata.py`. The 4 failures are
   entirely because pydantic reads the real `.env` and `PUBLIC_BASE_URL=http://localhost:21081`
   overrides the test's expected `http://test`. Confirmed: the same tests pass
   when run from a directory without `.env`. Fix by monkeypatching
   `get_settings().public_base_url = ""` in the `client` fixture, so tests derive
   the base URL from request headers as intended.
2. Resolve `tests/test_hold_slayer.py::TestMockSIPEngine::test_trunk_status`.
   **The test is wrong, not the code.** It asserts the mock trunk reports
   `registered is True` after `start()`, but `MockSIPEngine.get_trunk_status()`
   deliberately returns `registered: False` with
   `reason: "No SIP trunk configured (mock mode)"` — which is exactly what the
   `/health`-tells-the-truth invariant requires. Update the test to assert
   `False` + the reason.
3. Triage `test_audio_classifier.py::test_complex_tone_as_music`. **Outcome: the
   test was right and the code had a real bug.** Measured feature values for the
   C-major chord: `music_score` = **0.850**, comfortably above the 0.7
   `music_threshold` — music was detected confidently. But step 5's
   `if speech_score > music_score` ran first and short-circuited, because
   `speech_score` = **1.000**: the four speech bands are wide and overlapping
   (flatness 0.223 ∈ (0.1,0.5), centroid 852 ∈ (500,4000), ZCR 0.039 ∈
   (0.02,0.2), RMS 0.183 ∈ (0.01,0.5)) and a sustained musical chord satisfies
   all four trivially. Verified structural, not a one-off — C major, A minor and
   a bright chord *all* classified `live_human` despite scoring 0.85/0.65 music.

   Fixed in `services/audio_classifier.py` by making the speech branch yield to a
   confident music score:
   `if speech_score > music_score and music_score < self.settings.music_threshold:`
   One line, both thresholds stay meaningful, scorers untouched. Chords now
   classify `music` (0.85) and all 18 classifier tests pass, so speech-like audio
   still classifies as speech.

   **This was the plan's highest-value find:** the bug is exactly the operational
   failure that rings your desk for a hold queue. Worth re-validating against
   *real* hold music in Phase 2 — the synthetic "bright chord" case scores 0.65,
   below threshold, so it still reads as speech.

**Exit criteria:** `pytest tests/ -v` green from the repo root with the real
`.env` present ✅ (165 passed). `ruff check .` clean — **deferred, see below**.

**Deferred: `ruff check .` reports 217 errors — all pre-existing.** Verified
identical (217) with my changes stashed, so nothing here was introduced by
Phase 0, and the three files I touched add none. The backlog is stylistic and
repo-wide: 157 × `UP045` (`Optional[X]` → `X | None`), 18 × `F401` unused
imports, 14 × `E501`, spread across `services/` (77), `models/` (75) and
`core/` (49). Auto-fixing would rewrite annotations throughout `sippy_engine.py`
and `media_pipeline.py` — the thread-boundary and media code that currently
**cannot be exercised** (no PJSUA2, no trunk). That is a large unverifiable diff
in the highest-risk files, unrelated to making the suite an honest gate. Recommend
its own commit, ideally after Phase 1b restores the ability to actually run that
code.

---

## Phase 1a — Internal signaling with a registered softphone (no audio)

Prove the SIP stack, registration, dial plan, and call lifecycle with zero
media and zero PSTN. This is the phase that de-risks the most for the least.

**Config:**
```
USE_MOCK_SIP=false          # real Sippy engine
SIP_TRUNK_HOST=             # blank — engine skips _register_trunk()
GATEWAY_SIP_HOST=0.0.0.0
GATEWAY_SIP_PORT=5060
GATEWAY_SIP_DOMAIN=voip.helu.ca
```
Leaving `SIP_TRUNK_HOST` blank is deliberate and load-bearing: `SippyEngine.start()`
only calls `_register_trunk()` `if self._trunk_host`, so the gateway comes up
listening for devices with **no possible path to the PSTN**. That is the
strongest safety guarantee available in this phase — not policy, but absence of
a route.

**Setup:**
1. Register a softphone (Linphone or Zoiper on a laptop; Groundwire on mobile)
   to `<caliban-ip>:5060`, any username, domain `voip.helu.ca`.
   **Expect registration to succeed with any credentials** — see the security
   finding below.
2. Confirm the device appears: `GET /api/v1/devices` and the `list_devices` MCP
   tool, plus the `📱 SIP REGISTER` line in the log.
3. Create a `Device` row for it (`type: sip_phone`, `sip_uri` matching the
   registered contact) so `call_device()` can target it.

**Validate:**
- REGISTER lands, appears in `_registered_devices`, refreshes on re-REGISTER,
  and disappears on `expires=0` (hang up / unregister in the softphone).
- Inbound INVITE from the softphone to the gateway surfaces an `incoming_invite`
  event and routes through the receptionist path.
- `gateway.call_device()` toward the softphone makes it **ring** — this
  exercises `_call_sip_device`, SDP generation, and the loop→Sippy funnel.
- Answer, then hang up from each end in turn; confirm `terminated` propagates
  and the leg is cleaned from `_legs`.
- DTMF from the softphone appears in the log (note: received DTMF currently has
  no consumer — it logs and stops).
- `/health` reports `degraded` with `sip_trunk.registered: false`. **This is
  correct** — do not chase a green light here.

**Security finding to address in this phase:** `_handle_incoming_register()`
replies `200 OK` to any REGISTER with no authentication challenge — no 401 with
a nonce, no digest verification. Any host that can reach port 5060 can register
as any AOR and become a transfer target. On a LAN-only bind for this phase
that's tolerable; **before Phase 3 exposes the gateway to a trunk it is not.**
Either add digest auth to the REGISTER path or firewall 5060 to known device
IPs. Flagging rather than folding in — it's a real change to the SIP path and
your call how to scope it.

**Exit criteria:** softphone registers, rings on `call_device`, both-direction
teardown is clean, no leaked legs after 10 call cycles, suite still green.

---

## Phase 1b — Gate: build PJSUA2 ✅ COMPLETE

**Done 2026-07-28.** Built out of order (ahead of Phase 1a) because nothing
useful works without it. pjproject **2.17** — newer than expected, with Python
3.13 and modern-gcc support already upstream, so no patching was needed.

`MediaPipeline.status()` now reports **`pjsua2_available: True`**; the stub-mode
warning is gone and the pipeline starts and stops cleanly at 16 kHz. Full
procedure recorded in **[pjsua2-build.md](pjsua2-build.md)**; README note now
points at it.

Notes worth carrying forward:

- **No `sudo` was needed.** Both missing tools (`swig`, `patchelf`) have PyPI
  wheels and installed into the venv, so nothing on the host changed outside
  `~/src` and `~/.local`.
- **The RPATH step is the non-obvious part.** The bindings compile and install
  cleanly and then fail at import with
  `ImportError: libpjsua2.so.2: cannot open shared object file`, because
  `~/.local/lib` isn't on the loader path. Fixed with `patchelf --set-rpath` on
  both the extension **and** all 12 `libpj*.so.2` libraries — patching only the
  extension just surfaces the transitive deps one layer down. Chosen over
  `LD_LIBRARY_PATH` because that would have to be set for uvicorn, systemd, cron
  and every subprocess, and it fails at call time rather than startup. Verified
  under `env -i` from `/`, so it depends on no inherited environment.
- **Configure found OpenSSL, ALSA and Opus** — the full codec/TLS surface.
- **165 tests still pass.** All `pjsua2` imports in the codebase are lazy
  (inside functions), so the suite still runs without touching real media.

**Exit criteria:** `import pjsua2` succeeds in the venv ✅; `MediaPipeline`
reports `pjsua2_available: true` ✅ (note the method is `status()`, not
`get_status()` as this plan originally said).

**Two caveats this build does not solve:**

1. **Not captured by `pip install -e ".[dev]"`.** It lives outside Python
   packaging metadata — a fresh venv, new host or rebuilt container needs it
   repeated. Host provisioning, not a dependency.
2. **The Docker image still runs stub media** — the Dockerfile deliberately
   skips this build. So the Phase 2 audio validation must run **outside the
   container**, or the Dockerfile needs a pjproject build stage. Worth deciding
   before Phase 2, since it determines where you test.

---

## Phase 2 — Audio, TTS, STT, and the classifier on internal calls

Now the softphone is a full test instrument: you can speak into the gateway and
hear it speak back, with no telephony charges.

**Config to fix first:**
```
TTS_BASE_URL=<real Rhema endpoint>   # NOT localhost:8000 — that's this app
TTS_MODEL=speaches-ai/Kokoro-82M-v1.0-ONNX
SPEACHES_URL=http://pan.helu.ca:22070
```

**Validate leaf services standalone before wiring them into a call** — a failure
here is much easier to read outside the call path:
- `TTSService.synthesize()` returns PCM directly (a short script against the
  service). Confirm `service.available` flips to `True` and `/health` reports
  `tts: ok`.
- `TranscriptionService.transcribe()` against a known WAV returns expected text;
  `/health` reports `stt: ok`.
- Both are reachable **from wherever the gateway actually runs** — neither
  `pan.helu.ca:22070` nor `nyx.helu.ca:29000` answered from this host during
  assessment. Resolve that before blaming the call path.
- LLM: `nyx.helu.ca:29000` with `Qwen3.6-35B-A3B-UD-Q4_K_XL`. Note the URL has
  no `/v1` suffix while the default does — verify which the client expects.

**Then in-call, softphone ↔ gateway only:**
- Receptionist answers an inbound call from the softphone: you hear the TTS
  greeting, speak, and your speech is transcribed. This single test exercises
  TTS + STT + LLM + media in the real path.
- Recording writes a real file to `recordings/` with actual audio in it (check
  the file plays, not just that it exists — the stub path created files too).
- Classifier: play hold music from a phone/laptop into the call and confirm
  `MUSIC`; speak and confirm `LIVE_HUMAN`. This is the ground truth that
  Phase 0's failing classifier test was gesturing at.
- Transcript persists to the DB on hangup and is readable via
  `get_call_transcript`.

**Exit criteria:** a full receptionist conversation over the softphone with
audible TTS, accurate STT, correct classification, a playable recording, and a
persisted transcript.

---

## Phase 3 — MCP integration

Independent of audio, so it can run in parallel with 1b/2 if convenient.

- Mint a PAT via `api/tokens.py`, confirm `hs_pat_…` format.
- Connect an MCP client to `https://<host>/mcp/` and confirm OAuth discovery
  (`/.well-known/oauth-protected-resource`) resolves against the real
  `PUBLIC_BASE_URL`.
- Read-only tools first: `gateway_status`, `list_active_calls`, `list_devices`,
  `search_call_history`, and the three resources.
- **Verify the auth boundary negatively**: a non-owner identity and a bogus PAT
  both get 403/401 on `/mcp/`, REST, and WS. Owner-only is an invariant; prove
  it rather than assuming it.
- `make_call` against the softphone (internal, no trunk) — confirms the MCP
  path reaches the gateway.
- **Test the emergency guard through MCP specifically**: `make_call("911")`,
  `"9911"`, `"112"`, `"+1911"`, and with whitespace/dashes must every one raise
  `ToolError` before any SIP action. Do this while `SIP_TRUNK_HOST` is still
  blank, so a guard failure cannot become an actual emergency call. **This is
  the single most important test in the plan** — run it before Phase 4, never
  after.

**Exit criteria:** all read tools return sane data, `make_call` rings the
softphone, every emergency variant is refused, non-owner is refused.

---

## Phase 4 — SIP trunk (first PSTN contact)

Only now does real money and a real network get involved.

1. Set `SIP_TRUNK_HOST/PORT/USERNAME/PASSWORD/TRANSPORT` for the provider; keep
   `SIP_TRUNK_DID=+16472474242`.
2. Restart and watch registration: `_register_trunk()` posts `trunk_registered`,
   `/health` should flip to **`healthy`** — real engine + registered trunk +
   reachable DB. This is the first moment a green `/health` is meaningful.
3. **Re-run the entire emergency-guard suite from Phase 3** now that a real
   route exists. The guard is what stands between an AI agent and a 911
   dispatcher with no E911 location attached.
4. First outbound call: dial **your own mobile**, nothing else. Confirm ring,
   answer, two-way audio, clean teardown, DB persistence.
5. First inbound: call the DID from your mobile, receptionist answers, routing
   and transfer-to-softphone work.
6. Then a real hold scenario against a known IVR with a long queue.

**Safety rails while validating:**
- Drop `MAX_CONCURRENT_CALLS` to `1` for first calls; restore to 4 after.
- Keep `MAX_HOLD_TIME` low initially — 7200s is two hours of trunk time if
  something wedges.
- Watch the provider's billing/CDR page live during the first calls.
- Have the provider portal open to kill calls out-of-band.

**Exit criteria:** `/health` genuinely `healthy`, one successful outbound to a
known number, one inbound handled, CDRs match expectations, no orphaned legs.

---

## Cross-cutting: `.env.example` drift ✅ FIXED (Phase 0)

`.env.example` had documented the removed `API_TOKEN` as the auth mechanism and
omitted every `CASDOOR_*` var, `OWNER_NAME`, `PUBLIC_BASE_URL`, and the whole
`TTS_*` block. Rewritten to match `config.py`: dropped `API_TOKEN`, added the
Casdoor/auth block (documenting both supported configurations and why SSO-off
off-loopback is refused), the full `TTS_*` and `RECEPTIONIST_*` blocks, and
reconciled `GATEWAY_SIP_PORT` to 5060 to match `.env`.

`TTS_BASE_URL` now defaults to `localhost:8001` in the template with a comment
warning that it must not equal the app's own `PORT` — this was blocker #3, and
the template previously would have reproduced it for any fresh checkout.

Validated by copying the template to a clean directory and parsing it through
`Settings()`: every sub-config loads, and `tts.base_url != localhost:{port}` is
asserted. `grep` confirms no `API_TOKEN` references remain in code, templates or
docs. The README config table was already current — no drift there.

---

## Summary of blockers found

| # | Blocker | Blocks | Severity |
|---|---|---|---|
| 1 | PJSUA2 not installed | all audio: TTS/STT/recording/classifier in-call | ✅ **fixed** — pjproject 2.17 built, `pjsua2_available: True` |
| 2 | REGISTER accepts any credentials, no digest auth | safe exposure of port 5060 | **security** |
| 3 | `TTS_BASE_URL` unset → defaults to app's own port | TTS entirely | config — ✅ fixed in template; **still set it in your real `.env`** |
| 4 | STT/LLM endpoints unreachable as tested | Phase 2 | environment |
| 5 | 6 failing tests (2 real, 4 `.env` bleed) | honest regression gate | ✅ **fixed** — 165 pass |
| 6 | `.env.example` documents removed `API_TOKEN` auth | fresh-environment test | ✅ **fixed** |
| 7 | Hold music classified as `LIVE_HUMAN` (found during Phase 0) | correct hold detection | ✅ **fixed** — real bug, see Phase 0 item 3 |

**Note on #3:** the fix landed in `.env.example` (the template). Your live
`.env` still has no `TTS_*` block at all, so TTS resolves to the
`localhost:8000` default and collides with the app. Set `TTS_BASE_URL` before
Phase 2.