r/hold-slayer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

PJSUA2 media plane — audio finally reaches the classifier #8

Merged
r merged 5 commits from feat/pjsua2-media-tap into main 2026-07-29 21:34:10 +00:00
Conversation 1 Commits 5 Files Changed 14 +1147 -131
r commented 2026-07-29 17:41:35 +00:00
Owner
Copy Link

The gateway can hear. Real RTP audio now flows from a call into the classifier, verified end to end against the Asterisk lab.

Why a refactor was needed

The original design was Sippy signals, PJSUA2 carries media. It cannot work, for a reason that is not obvious until you try it:

PJSUA2 exposes no standalone RTP media object. Every AudioMedia subclass in the Python bindings is a file player, recorder, tone generator or capture port. RTP is reachable only through pj.Call.getAudioMedia(), on a dialog PJSUA2 itself owns. There is no "give me media for this remote host:port" API.

So MediaPipeline.add_remote_stream() was not unfinished work — it was a function that could not be written against this API. The consequence: audio never reached the classifier, and create_tap logged success while returning a tap nothing ever fed.

Owning the dialog is the price of owning the media, so PJSUAEngine places the call. Sippy keeps the SBC roles it is good at — device registration, routing, leg bridging — and SippyEngine remains fully functional for signalling; it simply cannot carry media, which its media branch now says plainly instead of calling a method that could never work.

Full rationale, including the two rejected alternatives, is in docs/architecture.md → "Media plane: why PJSUA2 places the call".

Safety invariants — untouched

PJSUAEngine is reachable only through gateway.make_call, which refuses emergency numbers and enforces the concurrency cap before any SIP action. No new dial path was introduced. The guard's position as the first check is unchanged.

Verified against the lab

Test Result
Speech (1001) live_human 0.75, stable
Hold music (1003) speech → music → speech, tracking the dialplan
DTMF (1002) Asterisk logged caller pressed 1 -> accounts and branched
Softphone (2001) two channels Up under one bridge id
RTP quality 1.2K packets, 0% loss, jitter 0.34 ms

DTMF reaching a real IVR closes out send_dtmf, which was a no-op under MockSIPEngine.

Opt-in

Selected with SIP_ENGINE=pjsua2. Default stays sippy while this proves out; the mock remains opt-in as before. PJSUAEngine implements the existing SIPEngine interface, so the gateway, call manager and hold-slayer service are unchanged.

Bugs found by running against real bindings

None of these would be caught by a unit test:

  • pj.Call and pj.Account finalised after libDestroy() abort the process on a native assertion, exactly as media ports do. Both are now dropped and collected before the endpoint is destroyed.
  • PJSUA2 keeps delivering callbacks during interpreter teardown, when module globals may already be cleared. Callbacks alias what they need locally and swallow everything — a raise there escapes into C++ and takes the worker thread with it.
  • hangup() only queues the BYE, so shutdown deleted the account with a call still active and left the far end on an unclosed dialog. stop() now waits for teardown.

Threading

Follows the established rule. PJSUA2 worker threads reach the loop only through _post_from_pj → run_coroutine_threadsafe; onFrameReceived touches nothing but AudioTap.feed (thread-safe via call_soon_threadsafe); any thread PJSUA2 did not create registers itself before touching a PJSUA2 object.

Lab fixture correction

The speech fixture classified correctly on its first 3 s window and drifted to music after — I had validated only the first window and reported it as verified, which overstated it.

One modelling error: _detect_tonality looks for autocorrelation periodicity in the 50–1000 Hz lag range, and each syllable used a constant f0, which is perfectly periodic there. That scored is_tonal=True, handing music a free 0.3 that speech could not outrun. The fundamental now follows a pitch contour with jitter, integrated to phase.

All three fixtures now classify correctly in 100% of windows (music 27/27, speech 5/5, silence 2/2), and stay correct when the window is stepped by half a window. tests/test_lab_fixtures.py guards this by sweeping every window — precisely what the original validation missed.

What this does not prove

Stated plainly so the branch is not over-trusted:

  • Trunk registration is untested. The lab has no registrar configured, so Account registration returned 404 Not Found. Expected, but unexercised.
  • Inbound calls are unexercised — onIncomingCall is written but never fired.
  • Device registration returns an empty list — get_registered_devices() is a stub.
  • Leg bridging is unexercised between two live calls.
  • Scenarios 1004–1008 (long hold, busy, no-answer, remote hangup, silence) have not been run.
  • Not real PSTN audio — no transcoding artefacts, packet loss, jitter, or carrier-side DTMF mangling.

This is a working media plane, not a finished engine.

Checks

  • 169 tests pass (up from 165)
  • ruff check clean on all new files; unchanged from baseline on pre-existing ones

🤖 Generated with Claude Code

The gateway can hear. Real RTP audio now flows from a call into the classifier, verified end to end against the Asterisk lab. ## Why a refactor was needed The original design was *Sippy signals, PJSUA2 carries media*. It cannot work, for a reason that is not obvious until you try it: **PJSUA2 exposes no standalone RTP media object.** Every `AudioMedia` subclass in the Python bindings is a file player, recorder, tone generator or capture port. RTP is reachable only through `pj.Call.getAudioMedia()`, on a dialog **PJSUA2 itself owns**. There is no "give me media for this remote host:port" API. So `MediaPipeline.add_remote_stream()` was not unfinished work — it was a function that could not be written against this API. The consequence: audio never reached the classifier, and `create_tap` logged success while returning a tap nothing ever fed. Owning the dialog is the price of owning the media, so `PJSUAEngine` places the call. Sippy keeps the SBC roles it is good at — device registration, routing, leg bridging — and `SippyEngine` remains fully functional for signalling; it simply cannot carry media, which its media branch now says plainly instead of calling a method that could never work. Full rationale, including the two rejected alternatives, is in `docs/architecture.md` → "Media plane: why PJSUA2 places the call". ## Safety invariants — untouched `PJSUAEngine` is reachable **only** through `gateway.make_call`, which refuses emergency numbers and enforces the concurrency cap before any SIP action. No new dial path was introduced. The guard's position as the first check is unchanged. ## Verified against the lab | Test | Result | |---|---| | Speech (1001) | `live_human` 0.75, stable | | Hold music (1003) | speech → **`music`** → speech, tracking the dialplan | | DTMF (1002) | Asterisk logged `caller pressed 1 -> accounts` and **branched** | | Softphone (2001) | two channels `Up` under one bridge id | | RTP quality | 1.2K packets, **0% loss**, jitter 0.34 ms | DTMF reaching a real IVR closes out `send_dtmf`, which was a no-op under `MockSIPEngine`. ## Opt-in Selected with `SIP_ENGINE=pjsua2`. Default stays `sippy` while this proves out; the mock remains opt-in as before. `PJSUAEngine` implements the existing `SIPEngine` interface, so the gateway, call manager and hold-slayer service are unchanged. ## Bugs found by running against real bindings None of these would be caught by a unit test: - `pj.Call` and `pj.Account` finalised after `libDestroy()` **abort the process** on a native assertion, exactly as media ports do. Both are now dropped and collected before the endpoint is destroyed. - PJSUA2 keeps delivering callbacks during interpreter teardown, when module globals may already be cleared. Callbacks alias what they need locally and swallow everything — a raise there escapes into C++ and takes the worker thread with it. - `hangup()` only queues the BYE, so shutdown deleted the account with a call still active and left the far end on an unclosed dialog. `stop()` now waits for teardown. ## Threading Follows the established rule. PJSUA2 worker threads reach the loop only through `_post_from_pj` → `run_coroutine_threadsafe`; `onFrameReceived` touches nothing but `AudioTap.feed` (thread-safe via `call_soon_threadsafe`); any thread PJSUA2 did not create registers itself before touching a PJSUA2 object. ## Lab fixture correction The speech fixture classified correctly on its first 3 s window and drifted to `music` after — I had validated only the first window and reported it as verified, which overstated it. One modelling error: `_detect_tonality` looks for autocorrelation periodicity in the 50–1000 Hz lag range, and each syllable used a *constant* `f0`, which is perfectly periodic there. That scored `is_tonal=True`, handing music a free 0.3 that speech could not outrun. The fundamental now follows a pitch contour with jitter, integrated to phase. All three fixtures now classify correctly in **100% of windows** (music 27/27, speech 5/5, silence 2/2), and stay correct when the window is stepped by half a window. `tests/test_lab_fixtures.py` guards this by sweeping every window — precisely what the original validation missed. ## What this does *not* prove Stated plainly so the branch is not over-trusted: - **Trunk registration is untested.** The lab has no registrar configured, so `Account` registration returned `404 Not Found`. Expected, but unexercised. - **Inbound calls are unexercised** — `onIncomingCall` is written but never fired. - **Device registration returns an empty list** — `get_registered_devices()` is a stub. - **Leg bridging is unexercised** between two live calls. - **Scenarios 1004–1008** (long hold, busy, no-answer, remote hangup, silence) have not been run. - **Not real PSTN audio** — no transcoding artefacts, packet loss, jitter, or carrier-side DTMF mangling. This is a working media plane, not a finished engine. ## Checks - **169 tests pass** (up from 165) - `ruff check` clean on all new files; unchanged from baseline on pre-existing ones 🤖 Generated with [Claude Code](https://claude.com/claude-code)
r added 4 commits 2026-07-29 17:41:36 +00:00
feat(media): implement PJSUA2 audio capture port; document the media-plane refactor 7979e70705 
Implements the tap half of the media path and records why the other half
requires moving call placement into PJSUA2.

MediaPipeline.create_tap was a stub: it logged "🎤 Audio tap created" and
returned a tap that nothing ever fed, so the classifier received no audio on
a live call. It now builds a real pj.AudioMediaPort subclass whose
onFrameReceived converts the SWIG ByteVector to PCM bytes and fans it out to
every tap on the stream.

One capture port per stream, shared by all taps: a second port on the same
stream would be mixed back into the conference bridge and the call would echo.

Thread safety is the constraint here. onFrameReceived runs on a PJSUA2 worker
thread — a third execution context beside the asyncio loop and the Sippy ED
thread — and touches nothing but AudioTap.feed, which hops to the owning loop
via call_soon_threadsafe. An exception escaping into PJSUA2's C++ callback
would tear down the worker thread and silently kill media for every call, so
the handler catches and logs once per port rather than on every 20ms frame.

Also fixes a hard crash found while testing this against real PJSUA2: a media
port finalised after Endpoint.libDestroy() calls pjmedia_conf_remove_port
against a freed conference bridge and aborts the process on a native
assertion. Ports are now released in remove_stream while the bridge still
exists, and stop() forces a collection before libDestroy — dropping the last
Python reference is not sufficient on its own.

Verified against the real bindings: frames fan out to multiple taps, cross the
thread boundary intact, and shutdown is clean.

add_remote_stream remains a stub, and deliberately so. PJSUA2 exposes no
standalone RTP media object — every AudioMedia subclass in the Python
bindings is a file player, recorder, tone generator or capture port, and RTP
is reachable only via pj.Call.getAudioMedia() on a dialog PJSUA2 itself owns.
A design where Sippy owns the dialog can never obtain media from PJSUA2, so
that function cannot be written against this API. docs/architecture.md now
explains this and records the resolution: PJSUA2 places the trunk call while
Sippy keeps the SBC roles (device registration, routing, leg bridging), with
the emergency guard and concurrency cap staying first in gateway.make_call
regardless of which library dials.

The architecture doc also had drift unrelated to media: it described the
thread boundary as asyncio.run_in_executor() when the real mechanism is
run_coroutine_threadsafe / ED2.callFromThread, claimed two execution contexts
where there are three, and cited MediaPipeline.add_stream() and
SippyEngine.bridge() — neither of which exists. Corrected, with the data flow
now showing the emergency guard and concurrency cap in their real positions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
feat(sip): add PJSUA2 engine — audio finally reaches the classifier 2a05be27bf 
The gateway can now hear. Verified end to end against the Asterisk lab:
speech classifies as live_human, hold music as music, the speech→music
transition tracks the dialplan, and DTMF reaches a real IVR (Asterisk logged
"caller pressed 1 -> accounts" and branched). RTP stats show 0% packet loss.

PJSUAEngine implements the existing SIPEngine interface, so the gateway,
call manager and hold-slayer service are unchanged. It is selected with
SIP_ENGINE=pjsua2; the default stays "sippy" while this is proven, and the
mock remains opt-in as before.

Why a new engine rather than fixing the old path: PJSUA2 exposes no
standalone RTP media object, so it will not surface media for a dialog it
does not own. Owning the dialog is the price of owning the media. Sippy keeps
the SBC roles it is good at — device registration, routing, leg bridging —
and SippyEngine remains fully functional for signalling; it simply cannot
carry media, which its media branch now says plainly instead of calling a
method that could never work.

The safety invariants are untouched. This engine is reachable only through
gateway.make_call, which refuses emergency numbers and enforces the
concurrency cap before any SIP action. No new dial path was introduced.

MediaPipeline.add_remote_stream(host, port) is replaced by
attach_call_media(stream_id, audio_media), called from onCallMediaState —
the one place PJSUA2 hands out RTP-backed media. Taps requested before media
comes up are attached when it does, so the classifier never misses the start
of a call.

Three crash/lifetime bugs found by running against the real bindings, none of
which any unit test would have caught:

- pj.Call and pj.Account objects finalised after libDestroy() abort the
  process on a native assertion, exactly as media ports do. Both are now
  dropped and collected before the pipeline destroys the endpoint.
- PJSUA2 keeps delivering callbacks during interpreter teardown, when module
  globals may already be cleared. The callbacks alias what they need locally
  and swallow everything: a raise there escapes into C++ and takes the worker
  thread with it.
- hangup() only queues the BYE, so shutdown deleted the account with a call
  still active and left the far end on an unclosed dialog. stop() now waits
  briefly for the teardown to complete.

Threading follows the established rule: PJSUA2 worker threads reach the loop
only through _post_from_pj → run_coroutine_threadsafe, and any thread PJSUA2
did not create registers itself before touching a PJSUA2 object.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix(lab): make the speech fixture actually classify as speech 92c45e9c4d 
The speech fixture classified as LIVE_HUMAN on its first 3s window and drifted
to MUSIC for every window after. I had validated only the first window and
reported the fixture as verified, which overstated it: any lab result resting
on that fixture — the hold-slayer scenarios above all — was proving less than
it appeared to.

The cause was one modelling error, not a tuning problem. `_detect_tonality`
looks for an autocorrelation peak above 0.5 in the 50-1000 Hz lag range, and
each syllable used a *constant* f0, which is perfectly periodic there. That
scored is_tonal=True, handing the music score a free 0.3 that speech could not
outrun — and the decision requires speech_score to strictly exceed
music_score, so ties went to music.

Real voices glide and jitter, so the periodicity never locks. The fundamental
now follows a per-syllable pitch contour (rise or fall, plus ~2% cycle-to-cycle
jitter), with the frequency integrated to phase rather than multiplied by t —
`2*pi*f*t` is only a chirp when f is the instantaneous rate, which it is not
once f0 itself moves. is_tonal is now False in every window.

Two smaller fixes fell out of that:

- Aspiration noise is high-passed rather than broadband. Flat noise puts
  energy in every Goertzel bin, so the strongest DTMF row and column both
  clear the detector's `total_power * 0.1` threshold and each syllable reads
  as a keypress. A first-difference filter leaves the 697-1633 Hz bands
  comparatively empty. The level is set for margin — spectral flatness lands
  at ~0.46, mid-way through the 0.1-0.5 band, not on an edge.

- The music fixture gained two more harmonics and a recording-style noise
  floor. Windows straddling a chord change had a momentarily sparse spectrum
  and fell *below* the music score's 0.05 flatness floor, scoring as speech.

All three fixtures now classify correctly in 100% of windows (music 27/27,
speech 5/5, silence 2/2), and remain correct when the window is stepped by
half a window — a fixture that only works on aligned boundaries would still
be a trap in a live call, where the analysis window has no relationship to
where the audio began.

Confirmed on a real call through the lab: scenario 1003 now shows the whole
hold-slayer arc, speech -> sustained music -> speech, matching the dialplan.

tests/test_lab_fixtures.py guards this: it sweeps every window rather than
sampling the first, which is exactly what the original validation missed, and
checks the generator is byte-for-byte deterministic. It skips when the
fixtures have not been generated, since they are gitignored.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
feat(lab): add softphone endpoint so device registration can be tested c516f659cc 
Asterisk is the registrar for devices, not Hold Slayer. A softphone REGISTERs
to the lab and the gateway transfers a live call to it by dialling extension
2001. This is deliberate: Hold Slayer's own SIP listener answers 200 OK to any
REGISTER with no digest challenge, so anything on the network could register
as a device and receive transferred calls. Keeping registration in Asterisk
means the lab does not exercise or depend on that path, and the device is
authenticated.

The pjsua CLI built alongside the Python bindings is the test device — same
library stack as the gateway, so no new dependency. Verified end to end: a
gateway call to 2001 produces two channels Up under one bridge id.

Three things that cost time and are now written down:

- `--realm=asterisk`, not `--realm='*'`: the wildcard fails against Asterisk's
  digest challenge with PJSIP_EFAILEDCREDENTIAL.

- pjsua is an interactive console app and exits ~8s after start if stdin is
  closed or /dev/null. `script -qfc` and `setsid </dev/null` both appear to
  work — registration succeeds — and then the process dies, leaving a stale
  contact in Asterisk that routes INVITEs to a port nobody is listening on.
  Hold a fifo open on stdin instead, and verify the port is actually bound
  rather than trusting `pjsip show contacts`.

- Qualify is off for this AOR: the pjsua console does not answer OPTIONS, so
  polling marks a working softphone Unavail and the dialplan refuses to ring
  it. The 2001 guard therefore tests PJSIP_AOR(softphone,contact) rather than
  DEVICE_STATE. A real hardphone answers OPTIONS and can have it re-enabled.

The identify block now matches source address *and port*. A host-only match
claims every packet from that address, so a co-located softphone's REGISTER
was attributed to the gateway endpoint and checked against the gateway's
password — surfacing as "Failed to authenticate" on a correct password.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
r added 1 commit 2026-07-29 21:30:03 +00:00
fix(lab): make Asterisk logs usable — drop ANSI codes and healthcheck noise 3150f78552 
Logging was configured correctly and shipping to Loki, but the stream was
useless: measured at 100% healthcheck chatter, with every line wrapped in ANSI
escape codes. The real SIP events were there and completely buried.

Three causes, each masking the next:

- asterisk.conf was written in the first lab commit with `nocolor = yes` and
  never mounted, so the setting had no effect. Now mounted. It was also
  overriding [directories] and runuser/rungroup, which the image sets up
  correctly itself — removed, since overriding them risks breaking the
  container for no gain.

- The image's command is `-vvvdddf`: verbosity 3 and debug 3 forced on the
  command line, which overrides both asterisk.conf and logger.conf. Overridden
  in compose to drop -v and -d; warnings and errors still log, and verbosity
  is raisable at runtime when tracing a call.

- The actual source: the image's healthcheck makes ~7 separate `asterisk -rx`
  connections every 30s, and Asterisk logs a connect/disconnect pair for each.
  Replaced with a single check on a 60s interval, and the check now runs
  `pjsip show transports` rather than `core show version` — that fails when
  Asterisk is up but unconfigured, which is exactly the state that produced a
  "healthy" container with no SIP stack on first deploy.

logger.conf drops both `notice` and `verbose`, which is where those pairs
arrive.

Verified on galatea: noise down from ~48 to 8 lines per two minutes (-83%),
zero ANSI codes in Loki, 90% of the stream now signal, container still
healthy, transport and dialplan intact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
r commented 2026-07-29 21:32:31 +00:00
Author
Owner
Copy Link

Added: 3150f78 — Asterisk logs made usable

Deployed the lab to Virgo Dev (galatea) and checked the logs through Grafana. Logging was configured correctly and shipping to Loki with the right labels — and the stream was still useless: 100% healthcheck chatter, every line wrapped in ANSI escape codes. The real SIP events were present and completely buried.

Three causes, each masking the next:

  • asterisk.conf was never mounted. It was written in the first lab commit carrying nocolor = yes and has been inert ever since. Now mounted. It also overrode [directories] and runuser/rungroup, which the image sets up correctly itself — removed, since overriding them risks breaking the container for no gain.

  • The image's command is -vvvdddf — verbosity 3 and debug 3 forced on the command line, which overrides both asterisk.conf and logger.conf. That is why editing logger.conf alone changed nothing. Overridden in compose to drop -v and -d.

  • The actual source: the image's healthcheck makes ~7 separate asterisk -rx connections every 30s, and Asterisk logs a connect/disconnect pair for each. Not verbosity at all. Replaced with a single check on a 60s interval.

That last check now runs pjsip show transports | grep <port> rather than core show version. It fails when Asterisk is up but unconfigured — precisely the state that produced a healthy container with no SIP stack on first deploy. The healthcheck now encodes the bug it previously missed.

Verified on galatea

Before After
Lines / 2 min ~48 8 (−83%)
ANSI codes in Loki every line 0
Signal share 0% ~90%

Container still healthy, transport bound on 21061, dialplan loaded. 169 tests pass.

Deployment note

The corresponding Ansible-side fixes are on virgo main (8995747, bd6d8da) — config file mode (the container's asterisk user is a different uid, so 640 made every config unreadable and Asterisk started with no configuration while reporting healthy), and the liveness check (wait_for cannot check a UDP port — it opens a TCP connection).

I applied these by hand on galatea to verify them, so a playbook re-run is worth doing to confirm Ansible asserts the same state.

Now-visible startup ERROR lines are benign module-load messages for CDR backends we do not use (cdr_pgsql, cdr_sqlite3_custom) — they were always there, just buried.

## Added: `3150f78` — Asterisk logs made usable Deployed the lab to Virgo Dev (galatea) and checked the logs through Grafana. Logging was configured correctly and shipping to Loki with the right labels — and the stream was still useless: **100% healthcheck chatter, every line wrapped in ANSI escape codes.** The real SIP events were present and completely buried. Three causes, each masking the next: - **`asterisk.conf` was never mounted.** It was written in the first lab commit carrying `nocolor = yes` and has been inert ever since. Now mounted. It also overrode `[directories]` and `runuser`/`rungroup`, which the image sets up correctly itself — removed, since overriding them risks breaking the container for no gain. - **The image's command is `-vvvdddf`** — verbosity 3 and debug 3 forced on the command line, which overrides both `asterisk.conf` and `logger.conf`. That is why editing `logger.conf` alone changed nothing. Overridden in compose to drop `-v` and `-d`. - **The actual source:** the image's healthcheck makes ~7 separate `asterisk -rx` connections every 30s, and Asterisk logs a connect/disconnect pair for each. Not verbosity at all. Replaced with a single check on a 60s interval. That last check now runs `pjsip show transports | grep <port>` rather than `core show version`. It **fails when Asterisk is up but unconfigured** — precisely the state that produced a `healthy` container with no SIP stack on first deploy. The healthcheck now encodes the bug it previously missed. ### Verified on galatea | | Before | After | |---|---|---| | Lines / 2 min | ~48 | **8** (−83%) | | ANSI codes in Loki | every line | **0** | | Signal share | 0% | **~90%** | Container still healthy, transport bound on 21061, dialplan loaded. 169 tests pass. ### Deployment note The corresponding Ansible-side fixes are on `virgo` main (`8995747`, `bd6d8da`) — config file mode (the container's `asterisk` user is a different uid, so `640` made every config unreadable and Asterisk started with *no* configuration while reporting healthy), and the liveness check (`wait_for` cannot check a UDP port — it opens a TCP connection). I applied these by hand on galatea to verify them, so a playbook re-run is worth doing to confirm Ansible asserts the same state. Now-visible startup `ERROR` lines are benign module-load messages for CDR backends we do not use (`cdr_pgsql`, `cdr_sqlite3_custom`) — they were always there, just buried.
r merged commit 98c80e3a56 into main 2026-07-29 21:34:10 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
r
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: r/hold-slayer#8
Delete Branch "feat/pjsua2-media-tap"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?