hold-slayer/docs/asterisk-lab-design.md

# Asterisk Lab — design

A **fake PSTN** for Hold Slayer: an Asterisk instance in Virgo Dev that answers
calls, plays an IVR, holds you in a queue with music, and eventually connects a
"human". It gives the gateway something real to dial that is not the PSTN — no
charges, no strangers, no E911 exposure, and a *deterministic* script that makes
classifier regressions reproducible.

Status: **design, not built.** Nothing here has been deployed.

---

## Why Asterisk and not Kamailio

Kamailio is a SIP **proxy** — it routes signaling and does not answer calls or
handle media. The risks that remain unproven in Hold Slayer are mostly *media*
risks: `send_dtmf` has only ever run against `MockSIPEngine` (a no-op), the
audio classifier has never seen real RTP, and the PJSUA2 pipeline built in Phase
1b has never carried a packet. A proxy forwards the INVITE and finds nobody
home, so it exercises none of that.

Asterisk is a B2BUA: it answers, plays prompts, collects DTMF, and can hold a
call in a queue with music. That is precisely the hold-slayer scenario, so it is
the right primary target.

Kamailio still has a place — **later, and narrowly**. It models a real ITSP's
registration/digest-auth behaviour better than Asterisk does, so it is the right
tool for exercising `_register_trunk()` ([core/sippy_engine.py:495](../core/sippy_engine.py#L495))
in isolation. It is deliberately *not* in scope for this lab.

```
Phase 2/3 (this lab)          Phase 4a (optional)        Phase 4b
Asterisk IVR + media    →     Kamailio registration  →   real PSTN, one call
```

---

## The one thing that makes this work without code changes

`SIP_TRUNK_HOST` is just an address. `SippyEngine.make_call()` builds
`sip:{number}@{trunk_host}:{trunk_port}`
([core/sippy_engine.py:568](../core/sippy_engine.py#L568)) and registers against
whatever host it is given. Point it at Asterisk and Hold Slayer dials it exactly
as it would dial a real provider.

**There is no test-only branch, no mock, no `if lab:` anywhere.** The code path
under test is the production code path. That is the entire value of this
approach — a lab that requires special-casing the application proves less than
it costs.

It also means the safety story is structural: while `SIP_TRUNK_HOST` points at
Asterisk on the Dev LAN, there is **no route to the PSTN at all**. Not a policy
that could be misconfigured — an absence of route.

---

## Placement in Virgo

| Decision | Value | Why |
|---|---|---|
| Host | **nereid** (`10.0.1.214`) | Terraform describes it as "Experimental Apps (POC, testing new technologies)" — [terraform/incus/containers.tf](../../virgo/terraform/incus/containers.tf). An unauthenticated SIP endpoint is exactly experimental. |
| Deploy | Docker Compose via Ansible | Matches every other Virgo service. `nereid` already has `docker = true`. |
| Hostname | `asterisk.helu.ca` (internal) | LAN only. **No `*.d.helu.ca` HAProxy entry** — HAProxy is HTTP; SIP/RTP would not traverse it, and this must not be publicly reachable. |
| Database | none | Asterisk needs no DB, so the "no databases in Docker" rule is not engaged. |

Hold Slayer itself stays on **triton** (`10.0.1.213`), where it is already
deployed at port 21081. Splitting the two hosts is deliberate: SIP then crosses
a real network with real latency, jitter and MTU, rather than a loopback that
hides every transport problem.

### Ports

Project **210** is Hold Slayer's (existing: `hold_slayer_web_port: 21081`).
Verified free: only `21081` and `29181` are allocated in that space today.

| Var | Port | Purpose |
|---|---|---|
| `asterisk_sip_port` | **21061** | SIP signalling (UDP). `2-10-6-1`: project 210, service 6 (SIP), instance 1 |
| `asterisk_rtp_start` | **21100** | RTP media range start (UDP) |
| `asterisk_rtp_end` | **21149** | RTP range end — 50 ports ≈ 25 concurrent calls, well above `max_concurrent_calls: 4` |
| `asterisk_ari_port` | **21071** | ARI/HTTP management (service 7 = management) |
| `asterisk_syslog_port` | **51462** | Docker syslog, `514YZ` convention (daedalus uses 51461) |

Service digit `6` for SIP is a **new allocation** — the existing scheme
([docs/virgo.md](../../virgo/docs/virgo.md) §Port Numbering) defines 1/2/5/7/8/9
and has no telephony digit. Worth confirming before it becomes precedent.

> **Note:** 5060 is *not* used. The convention forbids random ports, and using
> the well-known SIP port invites scanner traffic. Nothing requires 5060 — both
> ends are configured.

---

## Call scenarios

Each maps to a Hold Slayer behaviour that is currently unproven. Extensions are
what Hold Slayer dials as `number`.

| Ext | Scenario | Proves |
|---|---|---|
| `1001` | **Immediate answer**, plays speech, hangs up after 30s | Baseline: INVITE→200→ACK→RTP→BYE, audio flows both ways, classifier reports `LIVE_HUMAN` |
| `1002` | **IVR menu** — "press 1 for accounts, 2 for cards", branches on DTMF | `send_dtmf` genuinely emits RFC 2833 and Asterisk receives it. This is the big one — currently a no-op in the mock |
| `1003` | **Hold music, then human** — 60s MoH, then answers | The whole hold-slayer loop: classify music → stay on hold → detect human → ring the owner |
| `1004` | **Long hold** — 10 min MoH | `MAX_HOLD_TIME` and the hold-check interval |
| `1005` | **Immediate busy** (`BUSY()`) | Failure path: call marked `FAILED`, no stuck leg |
| `1006` | **Ring, never answer** | Timeout path |
| `1007` | **Answer then hang up after 5s** | Remote-BYE handling, DB persistence on hangup |
| `1008` | **Silence after answer** | Classifier `SILENCE` vs the 30s no-audio case |

`1002` and `1003` are the two that matter most; the rest are cheap to add once
the dialplan exists.

### Classifier determinism

The reason for a scripted IVR rather than a real call: real hold music varies
per call, so a classifier regression on the PSTN is indistinguishable from
noise. Against a fixed prompt the answer is binary. Recommend a **fixed MoH
file committed to the repo** rather than Asterisk's stock music, so the
classifier's input is byte-identical on every run and across hosts.

This is what makes the Phase 0 music-vs-speech precedence fix
([services/audio_classifier.py](../services/audio_classifier.py)) testable
against real audio for the first time.

---

## Security — the part that needs a decision

Two unauthenticated SIP endpoints would exist on the Dev LAN.

**1. Asterisk.** Default `pjsip.conf` examples accept anonymous calls. This lab
must not: `allow_anonymous_inbound = no`, an explicit endpoint for Hold Slayer
with a password, and `permit=` limited to triton's address. Asterisk's default
config is a well-known toll-fraud target and must not be shipped as-is.

**2. Hold Slayer's own SIP listener — pre-existing, flagged earlier.**
`_handle_incoming_register()` replies `200 OK` to any REGISTER with **no digest
challenge**. On loopback that was tolerable. The moment the gateway binds a LAN
interface to talk to Asterisk, any host on the Dev LAN can register as a device
and receive transferred calls.

That is not caused by this lab, but this lab is what makes it reachable. Options,
in order of preference:

1. **Implement digest auth** on inbound REGISTER — the real fix
2. **Bind the SIP listener to a specific interface** and firewall 5060 to triton
   ↔ nereid only — mitigation, not a fix
3. Accept it explicitly, in writing, as Dev-only

I would not deploy this without at least (2), and (1) is required before
anything resembling production. **This needs your decision before build.**

Also note: `pjsua2` runs `--enable-shared` with TLS available, so SIP-TLS +
SRTP is possible later. Not proposed for the lab — plain UDP keeps `sngrep`
readable, which matters enormously when debugging signalling.

---

## Observability

Match the estate rather than inventing:

- **Logs** — default `json-file` driver, discovered by the host Alloy Docker
  socket source and labelled `job=asterisk`. **No syslog listener and no Loki
  URL env** — that would double-ship, the same note already carried in
  [hold-slayer's compose template](../../virgo/ansible/hold-slayer/docker-compose.yml.j2).
- **Health** — Asterisk has no HTTP health endpoint by default. Enable ARI on
  `21071` and probe `/ari/asterisk/info`, which is a genuine liveness signal
  (the SIP stack answers), unlike a bare TCP check.
- **`sngrep`** on nereid for live SIP ladder inspection. Not currently installed
  anywhere; it is the single most useful tool when signalling misbehaves.

---

## What this does *not* prove

Stated plainly so the lab is not over-trusted:

- **Not real PSTN audio.** No G.711 transcoding artefacts, no packet loss, no
  jitter, no carrier-side DTMF mangling. Asterisk is clean; the PSTN is not.
- **Not real IVR behaviour.** Our dialplan is what we imagine a bank sounds
  like. Real trees are longer, noisier, and interrupt.
- **Not trunk registration/auth** — that is Kamailio's job (Phase 4a), or the
  real trunk's.
- **Not carrier-specific quirks** — each ITSP has its own.

It proves the gateway's *own* logic end to end. That is the majority of the
risk, and it is the part that is currently entirely untested against real media.

---

## Blocking prerequisite — the container runs stub media

**The Hold Slayer Docker image deliberately does not build PJSUA2**
([docs/pjsua2-build.md](pjsua2-build.md)), so the deployed container on triton
runs the media pipeline in **stub mode**. Stub mode's audio calls *return
successfully while doing nothing*.

If the lab runs against the current image, every media test passes while proving
nothing. This is the single most dangerous failure mode in the plan, because it
looks like success.

Two options:

| Option | Effort | Trade-off |
|---|---|---|
| **A. Add a pjproject build stage to the Dockerfile** | Higher — multi-stage build, ~10 min build, larger image | The deployed artefact gains real media. Needed eventually regardless |
| **B. Run Hold Slayer from a venv on triton for the lab** | Lower — the Phase 1b build already exists on caliban | Tests the binaries actually built, but diverges from the deployed artefact |

**Recommendation: A.** B tests something that is not what ships, and the
Dockerfile needs this anyway before Hold Slayer can place a real call from a
container. Doing it now means the lab validates the real artefact. B is a
reasonable short-cut only if you want a fast first signal.

The existing deploy also has a **stale-config finding** — see below — that
touches the same file, so both are worth doing in one pass.

---

## Finding: the deployed compose template is stale

Independent of this lab, [the deployed template](../../virgo/ansible/hold-slayer/docker-compose.yml.j2)
sets `API_TOKEN`, which **no longer exists** — auth is now Casdoor SSO + PATs
via one resolver. The comment "the app's single static bearer across REST/WS/MCP
… required on 0.0.0.0" describes an auth model that was removed.

Live state confirms the service is up and `degraded`/`engine: mock` (correct and
honest). Given `_check_startup_config` refuses SSO-off on a non-loopback bind, it
is worth establishing how it is currently booting — most likely `CASDOOR_ENABLED`
defaults such that the unknown `API_TOKEN` is simply ignored.

Flagging, not fixing — it is outside this design, but it lives in the file the
lab will modify, and `hold_slayer_api_token` is still being pulled from the OCI
vault for a variable the app no longer reads.

---

## Build order

Each step is independently verifiable; none commits you to the next.

1. **Decide** the two open questions: media (A or B), and SIP-listener security
   (digest / firewall / accept)
2. **Dialplan + compose**, developed on caliban against a local Asterisk
   container — no Virgo changes yet, fastest iteration
3. **Prove `1001`** locally: Hold Slayer places a call, audio flows, classifier
   sees `LIVE_HUMAN`. This is the real Phase 2 gate
4. **Prove `1002`/`1003`** locally: DTMF lands, hold→human transition fires
5. **Promote to Virgo** — Ansible role on nereid, `SIP_TRUNK_HOST=nereid.helu.ca`
   on triton, re-run 1–8 across the LAN
6. **Only then** consider Kamailio (4a) or the PSTN (4b)

Steps 2–4 need no Virgo changes at all, which is worth exploiting: the dialplan
is where the fiddly work is, and iterating locally is far faster than through
Ansible.

---

## Files this would add

```
hold-slayer/
  tests/lab/
    dialplan/extensions.conf     # the 8 scenarios
    dialplan/pjsip.conf          # endpoint for Hold Slayer, anonymous denied
    sounds/hold-music.wav        # fixed MoH — deterministic classifier input
    docker-compose.lab.yml       # local Asterisk for steps 2–4
    README.md                    # how to run the lab locally

virgo/
  ansible/asterisk/
    deploy.yml                   # mirrors ansible/hold-slayer/deploy.yml
    docker-compose.yml.j2
    extensions.conf.j2
    pjsip.conf.j2
  ansible/inventory/host_vars/nereid.helu.ca.yml   # + asterisk_* vars, + service
```

Dialplan lives in **hold-slayer**, not virgo: it is test fixture data that
belongs with the code it tests, and step 2–4 iteration needs it locally.
Ansible templates it out to nereid for step 5.

---

## Open questions

1. **Media: A or B?** Determines whether step 5 tests the real artefact.
2. **SIP listener security** — digest auth, firewall, or documented acceptance?
   Blocking for step 5, not for steps 2–4.
3. **Is service digit `6` acceptable for SIP** in the 22XYZ scheme, or should
   telephony get a different digit? Sets estate precedent.
4. **`asterisk.helu.ca` DNS** — needs an entry, or is `nereid.helu.ca` on the
   allocated port sufficient? (Simpler, and one less thing to maintain.)