asterisk-voicemail/docs/OPERATIONS.md

Operations

Day-to-day running: backups, monitoring, common problems, upgrades.

---

Services

sudo systemctl status vm-api vm-portal     # API (:8098) + legacy portal (:8099)
sudo journalctl -u vm-api -n 50             # API logs (uvicorn access/startup)
sudo journalctl -u vm-portal -n 50          # legacy portal logs
sudo tail -f /var/log/asterisk/vm_mailcmd.log   # per-voicemail pipeline log

Health checks:

curl -s http://127.0.0.1:8098/api/healthz      # JSON API health
curl -s http://127.0.0.1:8099/healthz          # legacy portal health
curl -s --resolve vm.txt3.net:443:ORIGIN_IP https://vm.txt3.net/api/healthz

---

Logs to watch

Signal	Where	Meaning
transcribed …	vm_mailcmd.log	a voicemail was processed OK
stored message for mailbox …	vm_mailcmd.log	it also reached the MySQL store
no audio attachment; relaying original	vm_mailcmd.log	Asterisk sent a text-only notice (missed-call style) — expected
Telegram disabled / no route	vm_mailcmd.log	Telegram skipped (per config) — expected if unset
relaying original, mailcmd error	vm_mailcmd.log	pipeline threw; original email was preserved (fail-safe worked)
status=sent	/var/log/mail.log	the enriched email left Postfix
452 4.3.1 Insufficient system storage	/var/log/mail.log	/var is full — see below

---

Backups

Two things to back up — the code is reproducible, the data is not. The MySQL asterisk database now holds messages, contacts and CDR, so back up the DB (not a SQLite file):

# MySQL dump (messages + contacts + cdr) — run as root
sudo mysqldump --single-transaction asterisk \
     > /backup/vm-mysql-$(date +%F).sql

# audio blobs (still on disk)
sudo -u asterisk tar czf /backup/vm-audio-$(date +%F).tgz -C / var/lib/vm-transcribe/audio

# config (the live, secret-bearing files)
sudo tar czf /backup/vm-conf-$(date +%F).tgz \
     /opt/vm-transcribe/telegram.conf /opt/vm-transcribe/contacts.conf \
     /opt/vm-transcribe/db_secret /opt/vm-transcribe/api.env

The whisper model cache (/opt/vm-transcribe/models) is reproducible — no need to back it up, just re-run install.sh.

Restoring: stop the services, load the dump into MySQL, extract audio to /var/lib/vm-transcribe/audio, systemctl start vm-api vm-portal.

---

Disk

/var/lib/vm-transcribe/audio grows with every voicemail. At ~290 KB per WAV (8000 Hz, mono, 23 s) that is ~12 MB per 40 messages. The portal's max_inbox per-user auto-delete keeps individual mailboxes bounded, but the total store only shrinks when messages are deleted (and their audio blob is unreferenced by any other message).

A full /var is the single most common cause of "voicemail emails stopped arriving" — Postfix rejects everything with 452 4.3.1 Insufficient system storage. Check df -h /var first when the pipeline looks dead.

---

Upgrading

git -C /home/jp/Work/asterisk-voicemail pull
cd /home/jp/Work/asterisk-voicemail

# backend
sudo cp src/*.py /opt/vm-transcribe/
sudo /opt/vm-transcribe/venv/bin/pip install -U faster-whisper   # occasionally
sudo systemctl restart vm-api vm-portal

# frontend (build in the UI repo, rsync the dist)
cd /home/jp/Work/voicemail-ui && npm install && npm run build
sudo -A rsync -a --exclude node_modules --exclude dist \
     /home/jp/Work/voicemail-ui/ /home/txt3/domains/vm.txt3.net/public_html/

install.sh is idempotent-ish for the first install but is not a general upgrade tool — it backs up voicemail.conf each run, so avoid re-running it blindly. For upgrades, copy src/*.py as above.

Note: the deploy paths in AGENTS.md use /home/jp/asterisk-vm and /home/jp/Work/voicemail-ui respectively — keep the two in sync.

---

Common problems

No transcription email arrived.

1. df -h /var — full disk blocks Postfix.
2. sudo grep '^mailcmd' /etc/asterisk/voicemail.conf — is it our script?
3. sudo tail /var/log/asterisk/vm_mailcmd.log — look for relaying original, mailcmd error.
4. sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 -c 'import faster_whisper' — is the venv intact?

Portal returns 502 / can't connect. The API backend is down or not on loopback: sudo systemctl status vm-api; ss -ltnp | grep 8098. Also confirm Apache has proxy/proxy_http enabled and that /api/ is proxied to 127.0.0.1:8098. (Legacy /audio/ 502 → check vm-portal on :8099.)

Login fails for a real mailbox. The PIN in voicemail.conf is what the API checks — not anything in the DB. If you changed a voicemail PIN, that change is picked up immediately (the file is reparsed each login). If the mailbox line is commented out or in a context the parser doesn't reach, login fails. Test the parser directly:

sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
     -c "import sys; sys.path.insert(0,'/opt/vm-transcribe'); import vm_auth; print(vm_auth.check_login('7940','5159'))"

MySQL connection errors in the API. Check /opt/vm-transcribe/db_secret (or api.env) and that the asterisk DB user exists with access to the asterisk database. The API logs a clear vm_store unavailable message if MySQL is unreachable.

Certbot renewal. Renewal runs from /etc/cron.d/certbot; certbot.timer is masked (normal on Debian's package). Test non-destructively in the background (it can be slow):

sudo certbot renew --cert-name vm.txt3.net --dry-run &

I locked myself out testing lockout. Restart the service: sudo systemctl restart vm-api. In-memory counters clear.

Backfill shows many no_speech. Expected. 44-byte WAVs (duration=0) are hung-up calls with no audio. Not a bug.

---

Re-running the backfill

Idempotent — safe any time:

sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_import.py

Useful after importing a large batch of new spool messages, or after a fresh install on a box with existing voicemails. To also load CDR history from Master.csv:

sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_backfill_cdr.py