# Operations

Day-to-day running: backups, monitoring, common problems, upgrades.

---

## Service status

```bash
sudo systemctl status vm-portal            # the portal
sudo journalctl -u vm-portal -n 50         # portal logs (uvicorn access/startup)
sudo tail -f /var/log/asterisk/vm_mailcmd.log   # per-voicemail pipeline log
```

Health of the portal (works locally or via the real hostname):

```bash
curl -s http://127.0.0.1:8099/healthz     # {"ok":true,"messages":N}
curl -s --resolve vm.txt3.net:443:ORIGIN_IP https://vm.txt3.net/healthz
```

---

## Logs to watch

| Signal | Where | Meaning |
|---|---|---|
| `transcribed …` | vm_mailcmd.log | a voicemail was processed OK |
| `stored message for mailbox …` | vm_mailcmd.log | it also reached the portal DB |
| `no audio attachment; relaying original` | vm_mailcmd.log | Asterisk sent a text-only notice (missed-call style) — expected |
| `Telegram disabled` / `no route` | vm_mailcmd.log | Telegram skipped (per config) — expected if unset |
| `relaying original, mailcmd error` | vm_mailcmd.log | **pipeline threw**; original email was preserved (fail-safe worked) |
| `status=sent` | /var/log/mail.log | the enriched email left Postfix |
| `452 4.3.1 Insufficient system storage` | /var/log/mail.log | **/var is full** — see below |

---

## Backups

Two things to back up — the code is reproducible, the *data* is not:

```bash
# database + audio (once or twice a day is plenty)
sudo -u asterisk tar czf /backup/vm-$(date +%F).tgz \
     -C / var/lib/vm-transcribe/voicemail.db var/lib/vm-transcribe/audio

# config (the live, secret-bearing files)
sudo tar czf /backup/vm-conf-$(date +%F).tgz \
     /opt/vm-transcribe/telegram.conf /opt/vm-transcribe/contacts.conf
```

The whisper model cache (`/opt/vm-transcribe/models`) is reproducible — no need
to back it up, just re-run `install.sh`.

Restoring: stop the service, extract, `systemctl start vm-portal`. The DB schema
is created on first connect, but **preserve the existing file** to keep history.

---

## Disk

`/var/lib/vm-transcribe/audio` grows with every voicemail. At ~290 KB per
WAV (8000 Hz, mono, 23 s) that is ~12 MB per 40 messages. The portal's
`max_inbox` per-user auto-delete keeps individual mailboxes bounded, but the
*total* store only shrinks when messages are deleted (and their audio blob is
unreferenced by any other message).

A full `/var` is the single most common cause of "voicemail emails stopped
arriving" — Postfix rejects everything with `452 4.3.1 Insufficient system
storage`. Check `df -h /var` first when the pipeline looks dead.

---

## Upgrading

```bash
git -C /home/jp/Work/asterisk-voicemail pull
cd /home/jp/Work/asterisk-voicemail
sudo cp src/*.py /opt/vm-transcribe/
sudo /opt/vm-transcribe/venv/bin/pip install -U faster-whisper   # occasionally
sudo systemctl restart vm-portal
```

`install.sh` is idempotent-ish for the first install but is **not** a general
upgrade tool — it backs up `voicemail.conf` each run, so avoid re-running it
blindly. For upgrades, copy `src/*.py` as above.

---

## Common problems

**No transcription email arrived.**
1. `df -h /var` — full disk blocks Postfix.
2. `sudo grep '^mailcmd' /etc/asterisk/voicemail.conf` — is it our script?
3. `sudo tail /var/log/asterisk/vm_mailcmd.log` — look for `relaying original, mailcmd error`.
4. `sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 -c 'import faster_whisper'` — is the venv intact?

**Portal returns 502 / can't connect.**
The uvicorn backend is down or not on loopback: `sudo systemctl status vm-portal`;
`ss -ltnp | grep 8099`. Also confirm Apache has `proxy`/`proxy_http` enabled.

**Login fails for a real mailbox.**
The PIN in `voicemail.conf` is what the portal checks — not anything in the
portal. If you changed a voicemail PIN, that change is picked up immediately
(the file is reparsed each login). If the mailbox line is commented out or in a
context the parser doesn't reach, login fails. Test the parser directly:

```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
     -c "import sys; sys.path.insert(0,'/opt/vm-transcribe'); import vm_auth; print(vm_auth.check_login('7940','5159'))"
```

**Certbot renewal.**
Renewal runs from `/etc/cron.d/certbot`; `certbot.timer` is masked (normal on
Debian's package). Test non-destructively in the background (it can be slow):

```bash
sudo certbot renew --cert-name vm.txt3.net --dry-run &
```

**I locked myself out testing lockout.**
Restart the service: `sudo systemctl restart vm-portal`. In-memory counters clear.

**Backfill shows many `no_speech`.**
Expected. 44-byte WAVs (`duration=0`) are hung-up calls with no audio. Not a bug.

---

## Re-running the backfill

Idempotent — safe any time:

```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_import.py
```

Useful after importing a large batch of new spool messages, or after a fresh
install on a box with existing voicemails.