asterisk-voicemail/docs/OPERATIONS.md

# Operations

Day-to-day running: backups, monitoring, common problems, upgrades.

---

## Services

```bash
sudo systemctl status vm-api vm-portal     # API (:8098) + legacy portal (:8099)
sudo journalctl -u vm-api -n 50             # API logs (uvicorn access/startup)
sudo journalctl -u vm-portal -n 50          # legacy portal logs
sudo tail -f /var/log/asterisk/vm_mailcmd.log   # per-voicemail pipeline log
```

Health checks:

```bash
curl -s http://127.0.0.1:8098/api/healthz      # JSON API health
curl -s http://127.0.0.1:8099/healthz          # legacy portal health
curl -s --resolve vm.txt3.net:443:ORIGIN_IP https://vm.txt3.net/api/healthz
```

---

## Logs to watch

| Signal | Where | Meaning |
|---|---|---|
| `transcribed …` | vm_mailcmd.log | a voicemail was processed OK |
| `stored message for mailbox …` | vm_mailcmd.log | it also reached the MySQL store |
| `no audio attachment; relaying original` | vm_mailcmd.log | Asterisk sent a text-only notice (missed-call style) — expected |
| `Telegram disabled` / `no route` | vm_mailcmd.log | Telegram skipped (per config) — expected if unset |
| `relaying original, mailcmd error` | vm_mailcmd.log | **pipeline threw**; original email was preserved (fail-safe worked) |
| `status=sent` | /var/log/mail.log | the enriched email left Postfix |
| `452 4.3.1 Insufficient system storage` | /var/log/mail.log | **/var is full** — see below |

---

## Backups

Two things to back up — the code is reproducible, the *data* is not. The MySQL
`asterisk` database now holds `messages`, `contacts` and CDR, so back up the DB
(not a SQLite file):

```bash
# MySQL dump (messages + contacts + cdr) — run as root
sudo mysqldump --single-transaction asterisk \
     > /backup/vm-mysql-$(date +%F).sql

# audio blobs (still on disk)
sudo -u asterisk tar czf /backup/vm-audio-$(date +%F).tgz -C / var/lib/vm-transcribe/audio

# config (the live, secret-bearing files)
sudo tar czf /backup/vm-conf-$(date +%F).tgz \
     /opt/vm-transcribe/telegram.conf /opt/vm-transcribe/contacts.conf \
     /opt/vm-transcribe/db_secret /opt/vm-transcribe/api.env
```

The whisper model cache (`/opt/vm-transcribe/models`) is reproducible — no need
to back it up, just re-run `install.sh`.

Restoring: stop the services, load the dump into MySQL, extract audio to
`/var/lib/vm-transcribe/audio`, `systemctl start vm-api vm-portal`.

---

## Disk

`/var/lib/vm-transcribe/audio` grows with every voicemail. At ~290 KB per
WAV (8000 Hz, mono, 23 s) that is ~12 MB per 40 messages. The portal's
`max_inbox` per-user auto-delete keeps individual mailboxes bounded, but the
*total* store only shrinks when messages are deleted (and their audio blob is
unreferenced by any other message).

A full `/var` is the single most common cause of "voicemail emails stopped
arriving" — Postfix rejects everything with `452 4.3.1 Insufficient system
storage`. Check `df -h /var` first when the pipeline looks dead.

---

## Upgrading

```bash
git -C /home/jp/Work/asterisk-voicemail pull
cd /home/jp/Work/asterisk-voicemail

# backend
sudo cp src/*.py /opt/vm-transcribe/
sudo /opt/vm-transcribe/venv/bin/pip install -U faster-whisper   # occasionally
sudo systemctl restart vm-api vm-portal

# frontend (build in the UI repo, rsync the dist)
cd /home/jp/Work/voicemail-ui && npm install && npm run build
sudo -A rsync -a --exclude node_modules --exclude dist \
     /home/jp/Work/voicemail-ui/ /home/txt3/domains/vm.txt3.net/public_html/
```

`install.sh` is idempotent-ish for the first install but is **not** a general
upgrade tool — it backs up `voicemail.conf` each run, so avoid re-running it
blindly. For upgrades, copy `src/*.py` as above.

> Note: the deploy paths in AGENTS.md use `/home/jp/asterisk-vm` and
> `/home/jp/Work/voicemail-ui` respectively — keep the two in sync.

---

## Common problems

**No transcription email arrived.**
1. `df -h /var` — full disk blocks Postfix.
2. `sudo grep '^mailcmd' /etc/asterisk/voicemail.conf` — is it our script?
3. `sudo tail /var/log/asterisk/vm_mailcmd.log` — look for `relaying original, mailcmd error`.
4. `sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 -c 'import faster_whisper'` — is the venv intact?

**Portal returns 502 / can't connect.**
The API backend is down or not on loopback: `sudo systemctl status vm-api`;
`ss -ltnp | grep 8098`. Also confirm Apache has `proxy`/`proxy_http` enabled and
that `/api/` is proxied to `127.0.0.1:8098`. (Legacy `/audio/` 502 → check
`vm-portal` on :8099.)

**Login fails for a real mailbox.**
The PIN in `voicemail.conf` is what the API checks — not anything in the DB. If
you changed a voicemail PIN, that change is picked up immediately (the file is
reparsed each login). If the mailbox line is commented out or in a context the
parser doesn't reach, login fails. Test the parser directly:

```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
     -c "import sys; sys.path.insert(0,'/opt/vm-transcribe'); import vm_auth; print(vm_auth.check_login('7940','5159'))"
```

**MySQL connection errors in the API.**
Check `/opt/vm-transcribe/db_secret` (or `api.env`) and that the `asterisk` DB
user exists with access to the `asterisk` database. The API logs a clear
`vm_store unavailable` message if MySQL is unreachable.

**Certbot renewal.**
Renewal runs from `/etc/cron.d/certbot`; `certbot.timer` is masked (normal on
Debian's package). Test non-destructively in the background (it can be slow):

```bash
sudo certbot renew --cert-name vm.txt3.net --dry-run &
```

**I locked myself out testing lockout.**
Restart the service: `sudo systemctl restart vm-api`. In-memory counters clear.

**Backfill shows many `no_speech`.**
Expected. 44-byte WAVs (`duration=0`) are hung-up calls with no audio. Not a bug.

---

## Re-running the backfill

Idempotent — safe any time:

```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_import.py
```

Useful after importing a large batch of new spool messages, or after a fresh
install on a box with existing voicemails. To also load CDR history from
`Master.csv`:

```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_backfill_cdr.py
```