Files
Robert Helewka 9dc766d631 fix(lab): silence Asterisk noise at the source, keep notice logging
Follow-up to 3150f78, which over-corrected. Dropping `notice` from logger.conf
made the log quiet by making the gateway undebuggable: Asterisk reports
rejected SIP requests at notice level via log_failed_request ("No matching
endpoint found", "Failed to authenticate"), and on a box whose entire job is
answering SIP those are the most useful lines it produces. A call was refused
and nothing said so.

`notice` is restored. The noise is dealt with where it originates instead:

- modules.conf, new. The image ships `autoload=yes` and loads every module it
  was built with, including chan_alsa on a container with no sound card:
  ~74 ALSA config errors per restart, plus module-load ERRORs for CDR/CEL
  backends, LDAP/ODBC realtime config, and format_ogg_vorbis — none of which
  can work here. Explicit noload for those; autoload stays on, because an
  allow-list would break quietly the first time a scenario needs a module
  nobody remembered to add.

- The healthcheck reduction from 3150f78 (one CLI connection on a 60s
  interval, rather than the image's ~7 every 30s) does the rest.

Verified on galatea: ALSA lines 74 → 0, all startup ERRORs gone (only one-off
benign WARNINGs remain), steady-state 4 lines per 2 minutes against ~840 per
30 minutes originally — and a refused call still logs 6 diagnostic lines.
Transport bound, both endpoints and dialplan loaded, container healthy.

`verbose` stays excluded: dialplan execution is worth having when tracing a
specific call, not worth shipping to Loki continuously. Raise it at runtime
with `asterisk -rx "core set verbose 3"`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 18:16:30 -04:00
..

Asterisk lab — a fake PSTN

An Asterisk instance that answers calls, plays an IVR, holds you with music, and eventually connects a "human". It gives the gateway something real to dial that is not the PSTN: no charges, no strangers, no E911 exposure, and a deterministic script that makes classifier regressions reproducible.

Design rationale and the Virgo deployment plan: docs/asterisk-lab-design.md.

This lab found five bugs in SippyEngine on its first call — the engine had never successfully placed one. Everything below runs against the real SippyEngine, never MockSIPEngine, which is the entire point.


Run it

cd tests/lab

# 1. Generate the audio fixtures (the image ships with NO sound files).
python sounds/generate.py

# 2. Render the local configs. They carry a host-specific IP and the lab
#    password, so they are gitignored — regenerate them per machine.
cd dialplan
LOCALIP=$(ip route get 1.1.1.1 | grep -oP '(?<=src\s)\d+(\.\d+){3}')
sed -e "s/{{ asterisk_sip_port }}/21061/" \
    -e "s/{{ asterisk_external_ip }}/$LOCALIP/" \
    -e "s#{{ asterisk_local_net }}#10.10.0.0/24#" \
    -e "s/{{ asterisk_match_host }}/127.0.0.1/" \
    -e "s/{{ asterisk_sip_username }}/holdslayer/" \
    -e "s/{{ asterisk_sip_password }}/labpassword/" \
    pjsip.conf > pjsip.local.conf
sed -e "s/{{ asterisk_rtp_start }}/21100/" \
    -e "s/{{ asterisk_rtp_end }}/21149/" \
    rtp.conf > rtp.local.conf
cd ..

# 3. Start it.
docker compose -f docker-compose.lab.yml up -d

Point Hold Slayer at it — no code changes, no test-only branch. make_call builds sip:{number}@{trunk_host}:{trunk_port}, so the lab is just an address:

USE_MOCK_SIP=false
SIP_TRUNK_HOST=127.0.0.1
SIP_TRUNK_PORT=21061
SIP_TRUNK_USERNAME=holdslayer
SIP_TRUNK_PASSWORD=labpassword
SIP_TRUNK_DID=+15550000000
GATEWAY_SIP_PORT=21062     # must differ from the Asterisk port

The repo's own .env sets USE_MOCK_SIP=true and a placeholder trunk host, and pydantic-settings lets .env win over the process environment. If the engine reports MockSIPEngine despite the above, that is why.

AIPSTNGateway(settings=...) also defaults to MockSIPEngine unless an engine is assigned — main.py's lifespan calls build_sip_engine() after construction. A harness that skips that step silently tests the mock.

The softphone (transfer target)

Asterisk is the registrar for devices, not Hold Slayer. The gateway reaches a desk phone by dialling extension 2001, which Asterisk routes to whatever has registered as softphone. This deliberately avoids Hold Slayer's own SIP listener, which answers 200 OK to any REGISTER with no digest challenge.

The pjsua CLI built alongside the Python bindings is the test device — same library stack as the gateway, no extra dependency. It needs an RPATH patch like the bindings did:

cp ~/src/pjproject/pjsip-apps/bin/pjsua-x86_64-pc-linux-gnu ~/.local/bin/pjsua
patchelf --set-rpath $HOME/.local/lib ~/.local/bin/pjsua

Register it (config file avoids shell-quoting pain):

cat > softphone.cfg <<'EOF'
--null-audio
--auto-answer=200
--max-calls=4
--local-port=21070
--id=sip:softphone@127.0.0.1
--registrar=sip:127.0.0.1:21061
--realm=asterisk
--username=softphone
--password=labphone
--log-level=3
EOF

# pjsua is an interactive console app: it exits ~8s after start if stdin is
# closed or /dev/null. Hold a fifo open on stdin — `script -qfc` and
# `setsid </dev/null` both look like they work (registration succeeds) and
# then the process dies, leaving a stale contact in Asterisk that routes
# INVITEs to a port nobody is listening on.
mkfifo sp.fifo
setsid sh -c 'exec 3<>sp.fifo; pjsua --config-file softphone.cfg <&3 >softphone.log 2>&1' &

Verify — check the port is actually bound, not just that Asterisk holds a contact, since a stale registration outlives the process:

ss -lnup | grep 21070                                     # must be listening
docker compose -f docker-compose.lab.yml exec asterisk \
    asterisk -rx "pjsip show contacts"                    # must show softphone

Then place a call to 2001. Both legs should show Up under one bridge id:

docker compose -f docker-compose.lab.yml exec asterisk \
    asterisk -rx "core show channels concise"

--realm=asterisk, not --realm='*' — the wildcard fails with PJSIP_EFAILEDCREDENTIAL against Asterisk's digest challenge.

Qualify is off for this AOR (qualify_frequency = 0): the pjsua console does not answer OPTIONS, so polling marks a working softphone Unavail and the dialplan refuses to ring it. The 2001 guard therefore tests PJSIP_AOR(softphone,contact) rather than DEVICE_STATE. A real hardphone answers OPTIONS and can have qualify re-enabled.

Useful commands

docker compose -f docker-compose.lab.yml exec asterisk asterisk -rvvv   # CLI
docker compose -f docker-compose.lab.yml logs -f asterisk               # logs
docker compose -f docker-compose.lab.yml exec asterisk \
    asterisk -rx "pjsip set logger on"                                  # SIP trace

Scenarios

Hold Slayer dials these as number.

Ext Scenario Proves
1001 Answers, speech, hangs up Baseline: INVITE→200→ACK→RTP→BYE, audio both ways
1002 IVR menu, branches on DTMF send_dtmf really emits RFC 2833 and Asterisk receives it
1003 Hold music, then a human The hold-slayer loop: music → wait → human → ring owner
1004 Long hold (~10 min) MAX_HOLD_TIME, HOLD_CHECK_INTERVAL
1005 Busy Failure path: call marked FAILED, no stuck leg
1006 Rings, never answers Timeout path
1007 Answers, hangs up after 5s Remote BYE, DB persistence on hangup
1008 Answers, then silence Classifier SILENCE vs. no-audio
1099 Echo test Debugging aid — confirm bidirectional RTP by ear

Audio fixtures

The Asterisk image ships no sound files, and the design calls for deterministic audio: real hold music varies per call, so a classifier regression on the PSTN is indistinguishable from noise. sounds/generate.py synthesises three fixtures from fixed seeds — byte-identical every run.

Verified against AudioClassifier (16 kHz):

Fixture Classifies as Confidence
lab-music.sln MUSIC 0.85
lab-speech.sln LIVE_HUMAN 0.75
lab-silence.sln SILENCE 1.00

Format is 8 kHz 16-bit mono signed-linear (.sln) — Asterisk's native telephony rate, played without transcoding.

The speech fixture's formants deliberately avoid the DTMF bands (rows 697941 Hz, columns 12091633 Hz). The first version landed on a valid DTMF pair and the whole utterance classified as a keypress.


Known limits

  • The classifier receives nothing on a live call. MediaPipeline.create_tap is a stub — it logs 🎤 Audio tap created and returns a tap that is never fed (core/media_pipeline.py, and the same at stream creation). RTP flows and Asterisk plays audio, but nothing reaches the classifier. The table above was measured by feeding the fixtures directly. This blocks the hold-slayer scenarios (1002/1003/1004).
  • Not real PSTN audio — no transcoding artefacts, packet loss, jitter, or carrier-side DTMF mangling. Asterisk is clean; the PSTN is not.
  • Not real IVR behaviour — this dialplan is what we imagine a bank sounds like. Real trees are longer, noisier, and interrupt.
  • Not trunk registration against a real ITSP_register_trunk() works against Asterisk, but carrier quirks are their own phase.

Security

pjsip.conf refuses anonymous inbound calls: every call must authenticate as the hold-slayer endpoint. Asterisk's stock examples allow anonymous calls and are a well-known toll-fraud target — there is no PSTN behind this instance, so an unauthorised call reaches only the dialplan, but the lock-down keeps this config safe to copy.

Endpoint matching is by source address (type=identify). Asterisk's default matches the From-header domain, which Hold Slayer populates from its SIP bind address — 0.0.0.0 on a wildcard bind, which never matches.

Separate, pre-existing: Hold Slayer's own SIP listener answers 200 OK to any REGISTER with no digest challenge. Fine on loopback; it must be resolved before the gateway binds a LAN interface, or any host on the network can register as a device and receive transferred calls.