Files
hold-slayer/tests/lab
Robert Helewka 3150f78552 fix(lab): make Asterisk logs usable — drop ANSI codes and healthcheck noise
Logging was configured correctly and shipping to Loki, but the stream was
useless: measured at 100% healthcheck chatter, with every line wrapped in ANSI
escape codes. The real SIP events were there and completely buried.

Three causes, each masking the next:

- asterisk.conf was written in the first lab commit with `nocolor = yes` and
  never mounted, so the setting had no effect. Now mounted. It was also
  overriding [directories] and runuser/rungroup, which the image sets up
  correctly itself — removed, since overriding them risks breaking the
  container for no gain.

- The image's command is `-vvvdddf`: verbosity 3 and debug 3 forced on the
  command line, which overrides both asterisk.conf and logger.conf. Overridden
  in compose to drop -v and -d; warnings and errors still log, and verbosity
  is raisable at runtime when tracing a call.

- The actual source: the image's healthcheck makes ~7 separate `asterisk -rx`
  connections every 30s, and Asterisk logs a connect/disconnect pair for each.
  Replaced with a single check on a 60s interval, and the check now runs
  `pjsip show transports` rather than `core show version` — that fails when
  Asterisk is up but unconfigured, which is exactly the state that produced a
  "healthy" container with no SIP stack on first deploy.

logger.conf drops both `notice` and `verbose`, which is where those pairs
arrive.

Verified on galatea: noise down from ~48 to 8 lines per two minutes (-83%),
zero ANSI codes in Loki, 90% of the stream now signal, container still
healthy, transport and dialplan intact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 17:26:03 -04:00
..

Asterisk lab — a fake PSTN

An Asterisk instance that answers calls, plays an IVR, holds you with music, and eventually connects a "human". It gives the gateway something real to dial that is not the PSTN: no charges, no strangers, no E911 exposure, and a deterministic script that makes classifier regressions reproducible.

Design rationale and the Virgo deployment plan: docs/asterisk-lab-design.md.

This lab found five bugs in SippyEngine on its first call — the engine had never successfully placed one. Everything below runs against the real SippyEngine, never MockSIPEngine, which is the entire point.


Run it

cd tests/lab

# 1. Generate the audio fixtures (the image ships with NO sound files).
python sounds/generate.py

# 2. Render the local configs. They carry a host-specific IP and the lab
#    password, so they are gitignored — regenerate them per machine.
cd dialplan
LOCALIP=$(ip route get 1.1.1.1 | grep -oP '(?<=src\s)\d+(\.\d+){3}')
sed -e "s/{{ asterisk_sip_port }}/21061/" \
    -e "s/{{ asterisk_external_ip }}/$LOCALIP/" \
    -e "s#{{ asterisk_local_net }}#10.10.0.0/24#" \
    -e "s/{{ asterisk_match_host }}/127.0.0.1/" \
    -e "s/{{ asterisk_sip_username }}/holdslayer/" \
    -e "s/{{ asterisk_sip_password }}/labpassword/" \
    pjsip.conf > pjsip.local.conf
sed -e "s/{{ asterisk_rtp_start }}/21100/" \
    -e "s/{{ asterisk_rtp_end }}/21149/" \
    rtp.conf > rtp.local.conf
cd ..

# 3. Start it.
docker compose -f docker-compose.lab.yml up -d

Point Hold Slayer at it — no code changes, no test-only branch. make_call builds sip:{number}@{trunk_host}:{trunk_port}, so the lab is just an address:

USE_MOCK_SIP=false
SIP_TRUNK_HOST=127.0.0.1
SIP_TRUNK_PORT=21061
SIP_TRUNK_USERNAME=holdslayer
SIP_TRUNK_PASSWORD=labpassword
SIP_TRUNK_DID=+15550000000
GATEWAY_SIP_PORT=21062     # must differ from the Asterisk port

The repo's own .env sets USE_MOCK_SIP=true and a placeholder trunk host, and pydantic-settings lets .env win over the process environment. If the engine reports MockSIPEngine despite the above, that is why.

AIPSTNGateway(settings=...) also defaults to MockSIPEngine unless an engine is assigned — main.py's lifespan calls build_sip_engine() after construction. A harness that skips that step silently tests the mock.

The softphone (transfer target)

Asterisk is the registrar for devices, not Hold Slayer. The gateway reaches a desk phone by dialling extension 2001, which Asterisk routes to whatever has registered as softphone. This deliberately avoids Hold Slayer's own SIP listener, which answers 200 OK to any REGISTER with no digest challenge.

The pjsua CLI built alongside the Python bindings is the test device — same library stack as the gateway, no extra dependency. It needs an RPATH patch like the bindings did:

cp ~/src/pjproject/pjsip-apps/bin/pjsua-x86_64-pc-linux-gnu ~/.local/bin/pjsua
patchelf --set-rpath $HOME/.local/lib ~/.local/bin/pjsua

Register it (config file avoids shell-quoting pain):

cat > softphone.cfg <<'EOF'
--null-audio
--auto-answer=200
--max-calls=4
--local-port=21070
--id=sip:softphone@127.0.0.1
--registrar=sip:127.0.0.1:21061
--realm=asterisk
--username=softphone
--password=labphone
--log-level=3
EOF

# pjsua is an interactive console app: it exits ~8s after start if stdin is
# closed or /dev/null. Hold a fifo open on stdin — `script -qfc` and
# `setsid </dev/null` both look like they work (registration succeeds) and
# then the process dies, leaving a stale contact in Asterisk that routes
# INVITEs to a port nobody is listening on.
mkfifo sp.fifo
setsid sh -c 'exec 3<>sp.fifo; pjsua --config-file softphone.cfg <&3 >softphone.log 2>&1' &

Verify — check the port is actually bound, not just that Asterisk holds a contact, since a stale registration outlives the process:

ss -lnup | grep 21070                                     # must be listening
docker compose -f docker-compose.lab.yml exec asterisk \
    asterisk -rx "pjsip show contacts"                    # must show softphone

Then place a call to 2001. Both legs should show Up under one bridge id:

docker compose -f docker-compose.lab.yml exec asterisk \
    asterisk -rx "core show channels concise"

--realm=asterisk, not --realm='*' — the wildcard fails with PJSIP_EFAILEDCREDENTIAL against Asterisk's digest challenge.

Qualify is off for this AOR (qualify_frequency = 0): the pjsua console does not answer OPTIONS, so polling marks a working softphone Unavail and the dialplan refuses to ring it. The 2001 guard therefore tests PJSIP_AOR(softphone,contact) rather than DEVICE_STATE. A real hardphone answers OPTIONS and can have qualify re-enabled.

Useful commands

docker compose -f docker-compose.lab.yml exec asterisk asterisk -rvvv   # CLI
docker compose -f docker-compose.lab.yml logs -f asterisk               # logs
docker compose -f docker-compose.lab.yml exec asterisk \
    asterisk -rx "pjsip set logger on"                                  # SIP trace

Scenarios

Hold Slayer dials these as number.

Ext Scenario Proves
1001 Answers, speech, hangs up Baseline: INVITE→200→ACK→RTP→BYE, audio both ways
1002 IVR menu, branches on DTMF send_dtmf really emits RFC 2833 and Asterisk receives it
1003 Hold music, then a human The hold-slayer loop: music → wait → human → ring owner
1004 Long hold (~10 min) MAX_HOLD_TIME, HOLD_CHECK_INTERVAL
1005 Busy Failure path: call marked FAILED, no stuck leg
1006 Rings, never answers Timeout path
1007 Answers, hangs up after 5s Remote BYE, DB persistence on hangup
1008 Answers, then silence Classifier SILENCE vs. no-audio
1099 Echo test Debugging aid — confirm bidirectional RTP by ear

Audio fixtures

The Asterisk image ships no sound files, and the design calls for deterministic audio: real hold music varies per call, so a classifier regression on the PSTN is indistinguishable from noise. sounds/generate.py synthesises three fixtures from fixed seeds — byte-identical every run.

Verified against AudioClassifier (16 kHz):

Fixture Classifies as Confidence
lab-music.sln MUSIC 0.85
lab-speech.sln LIVE_HUMAN 0.75
lab-silence.sln SILENCE 1.00

Format is 8 kHz 16-bit mono signed-linear (.sln) — Asterisk's native telephony rate, played without transcoding.

The speech fixture's formants deliberately avoid the DTMF bands (rows 697941 Hz, columns 12091633 Hz). The first version landed on a valid DTMF pair and the whole utterance classified as a keypress.


Known limits

  • The classifier receives nothing on a live call. MediaPipeline.create_tap is a stub — it logs 🎤 Audio tap created and returns a tap that is never fed (core/media_pipeline.py, and the same at stream creation). RTP flows and Asterisk plays audio, but nothing reaches the classifier. The table above was measured by feeding the fixtures directly. This blocks the hold-slayer scenarios (1002/1003/1004).
  • Not real PSTN audio — no transcoding artefacts, packet loss, jitter, or carrier-side DTMF mangling. Asterisk is clean; the PSTN is not.
  • Not real IVR behaviour — this dialplan is what we imagine a bank sounds like. Real trees are longer, noisier, and interrupt.
  • Not trunk registration against a real ITSP_register_trunk() works against Asterisk, but carrier quirks are their own phase.

Security

pjsip.conf refuses anonymous inbound calls: every call must authenticate as the hold-slayer endpoint. Asterisk's stock examples allow anonymous calls and are a well-known toll-fraud target — there is no PSTN behind this instance, so an unauthorised call reaches only the dialplan, but the lock-down keeps this config safe to copy.

Endpoint matching is by source address (type=identify). Asterisk's default matches the From-header domain, which Hold Slayer populates from its SIP bind address — 0.0.0.0 on a wildcard bind, which never matches.

Separate, pre-existing: Hold Slayer's own SIP listener answers 200 OK to any REGISTER with no digest challenge. Fine on loopback; it must be resolved before the gateway binds a LAN interface, or any host on the network can register as a device and receive transferred calls.