Robert Helewka
92c45e9c4d
The speech fixture classified as LIVE_HUMAN on its first 3s window and drifted to MUSIC for every window after. I had validated only the first window and reported the fixture as verified, which overstated it: any lab result resting on that fixture — the hold-slayer scenarios above all — was proving less than it appeared to. The cause was one modelling error, not a tuning problem. `_detect_tonality` looks for an autocorrelation peak above 0.5 in the 50-1000 Hz lag range, and each syllable used a *constant* f0, which is perfectly periodic there. That scored is_tonal=True, handing the music score a free 0.3 that speech could not outrun — and the decision requires speech_score to strictly exceed music_score, so ties went to music. Real voices glide and jitter, so the periodicity never locks. The fundamental now follows a per-syllable pitch contour (rise or fall, plus ~2% cycle-to-cycle jitter), with the frequency integrated to phase rather than multiplied by t — `2*pi*f*t` is only a chirp when f is the instantaneous rate, which it is not once f0 itself moves. is_tonal is now False in every window. Two smaller fixes fell out of that: - Aspiration noise is high-passed rather than broadband. Flat noise puts energy in every Goertzel bin, so the strongest DTMF row and column both clear the detector's `total_power * 0.1` threshold and each syllable reads as a keypress. A first-difference filter leaves the 697-1633 Hz bands comparatively empty. The level is set for margin — spectral flatness lands at ~0.46, mid-way through the 0.1-0.5 band, not on an edge. - The music fixture gained two more harmonics and a recording-style noise floor. Windows straddling a chord change had a momentarily sparse spectrum and fell *below* the music score's 0.05 flatness floor, scoring as speech. All three fixtures now classify correctly in 100% of windows (music 27/27, speech 5/5, silence 2/2), and remain correct when the window is stepped by half a window — a fixture that only works on aligned boundaries would still be a trap in a live call, where the analysis window has no relationship to where the audio began. Confirmed on a real call through the lab: scenario 1003 now shows the whole hold-slayer arc, speech -> sustained music -> speech, matching the dialplan. tests/test_lab_fixtures.py guards this: it sweeps every window rather than sampling the first, which is exactly what the original validation missed, and checks the generator is byte-for-byte deterministic. It skips when the fixtures have not been generated, since they are gitignored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
110 lines
3.8 KiB
Python
110 lines
3.8 KiB
Python
"""
|
|
Lab audio fixtures — the classifier must agree with what each one claims to be.
|
|
|
|
These guard the *fixtures*, not the classifier. `tests/lab/sounds/generate.py`
|
|
synthesises music/speech/silence that the Asterisk lab plays down a real call;
|
|
if a fixture drifts into the wrong class, every lab result built on it is
|
|
quietly meaningless — a hold-music scenario that never classifies as music
|
|
proves nothing about the hold slayer.
|
|
|
|
The first version of these fixtures passed on the opening 3s window and drifted
|
|
to MUSIC after, which a single-window check would not have caught. Hence the
|
|
sweep across every window.
|
|
|
|
Skipped when the fixtures have not been generated: they are gitignored (~680K,
|
|
reproducible from a fixed seed), so a fresh checkout has none until
|
|
`python tests/lab/sounds/generate.py` runs.
|
|
"""
|
|
|
|
import subprocess
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
import numpy as np
|
|
import pytest
|
|
|
|
from config import Settings
|
|
from models.call import AudioClassification
|
|
from services.audio_classifier import SAMPLE_RATE, AudioClassifier
|
|
|
|
SOUNDS_DIR = Path(__file__).parent / "lab" / "sounds"
|
|
GENERATOR = SOUNDS_DIR / "generate.py"
|
|
|
|
# The lab writes 8 kHz .sln; the classifier works at 16 kHz.
|
|
LAB_RATE = 8000
|
|
WINDOW_SAMPLES = SAMPLE_RATE * 3 # classifier's 3s analysis window
|
|
|
|
FIXTURES = [
|
|
("lab-music.sln", AudioClassification.MUSIC),
|
|
("lab-speech.sln", AudioClassification.LIVE_HUMAN),
|
|
("lab-silence.sln", AudioClassification.SILENCE),
|
|
]
|
|
|
|
|
|
def _load_16k(path: Path) -> np.ndarray:
|
|
"""Load an 8 kHz .sln and upsample to the classifier's 16 kHz."""
|
|
return np.repeat(np.fromfile(path, dtype="<i2"), SAMPLE_RATE // LAB_RATE)
|
|
|
|
|
|
def _windows(samples: np.ndarray, step: int):
|
|
"""Yield successive analysis windows; at least one, even for short files."""
|
|
end = max(1, len(samples) - WINDOW_SAMPLES)
|
|
for offset in range(0, end, step):
|
|
yield samples[offset : offset + WINDOW_SAMPLES].astype("<i2").tobytes()
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def classifier():
|
|
return AudioClassifier(settings=Settings().classifier)
|
|
|
|
|
|
@pytest.mark.parametrize("filename,expected", FIXTURES)
|
|
def test_fixture_classifies_correctly_in_every_window(filename, expected, classifier):
|
|
"""Every window must classify correctly — not just the first.
|
|
|
|
Stepped at half the window length so windows overlap: a fixture that only
|
|
works on aligned boundaries would still be a trap in a live call, where
|
|
the window has no relationship to where the audio started.
|
|
"""
|
|
path = SOUNDS_DIR / filename
|
|
if not path.exists():
|
|
pytest.skip(f"{filename} not generated — run {GENERATOR}")
|
|
|
|
samples = _load_16k(path)
|
|
results = [
|
|
classifier.classify_chunk(w).audio_type
|
|
for w in _windows(samples, step=WINDOW_SAMPLES // 2)
|
|
]
|
|
|
|
wrong = [(i, r.value) for i, r in enumerate(results) if r is not expected]
|
|
assert not wrong, (
|
|
f"{filename} must classify as {expected.value} in all "
|
|
f"{len(results)} windows; wrong: {wrong}"
|
|
)
|
|
|
|
|
|
def test_generator_is_deterministic(tmp_path):
|
|
"""Same bytes on every run — the whole point of synthesising them.
|
|
|
|
Real hold music varies per call, so a classifier regression on the PSTN is
|
|
indistinguishable from noise. Fixed-seed audio makes the answer binary.
|
|
"""
|
|
if not GENERATOR.exists():
|
|
pytest.skip("generator not present")
|
|
|
|
def run(target: Path) -> dict[str, bytes]:
|
|
subprocess.run(
|
|
[sys.executable, str(GENERATOR), str(target)],
|
|
check=True,
|
|
capture_output=True,
|
|
)
|
|
return {p.name: p.read_bytes() for p in sorted(target.glob("*.sln"))}
|
|
|
|
first = run(tmp_path / "a")
|
|
second = run(tmp_path / "b")
|
|
|
|
assert first, "generator produced no .sln files"
|
|
assert first.keys() == second.keys()
|
|
for name in first:
|
|
assert first[name] == second[name], f"{name} differs between runs"
|