jp/asterisk-voicemail
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial build: Asterisk voicemail transcription + portal

Browse Source 
- mailcmd replacement (vm_mailcmd.py): faster-whisper transcription (CPU int8),
  extractive summary + intent tags + spoken-digit number extraction,
  multipart/alternative HTML email, fail-safe relay of original message
- Telegram DM delivery (vm_telegram.py) with per-mailbox routing
- Caller-ID -> name (vm_contacts.py): file / google / carddav backends
- SQLite store (vm_store.py) with content-addressed audio
- FastAPI portal (vm_web.py): PIN login, list/play/delete, per-user settings,
  zero JS, loopback-only behind Apache TLS
- Backfill importer (vm_import.py) for existing spool recordings
- systemd unit, Apache vhost + certbot TLS, install.sh
- Docs: INSTALL, CONFIGURATION, ARCHITECTURE, OPERATIONS, SECURITY, TESTING

Verified end-to-end on mail.txt3.net: 157 historical messages backfilled,
live voicemail -> transcribed -> stored -> visible at https://vm.txt3.net.
This commit is contained in:
jp
2026-08-13 09:31:41 +01:00
commit 857284abbf
28 changed files with 3793 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

137
docs/ARCHITECTURE.md Normal file
View File

@ -0,0 +1,137 @@
# Architecture
How the pieces fit and *why* they are shaped the way they are. Read this
before changing anything in `vm_mailcmd.py` or `vm_store.py`.
---
## Flow
```
Asterisk ──(RFC822 on stdin)──▶ vm_mailcmd.py
│
┌──────────┬───────────────┼───────────────┬───────────────┐
▼ ▼ ▼ ▼ ▼
transcribe summarise resolve store telegram
(whisper) +intents contact (sqlite) (DM)
│ │ │ │ │
└────▶ build multipart email ─▶ Postfix :25
(plain + HTML + audio attached)
│
└──▶ vm_store.add_message() (best-effort)
└──▶ vm_telegram.send() (best-effort)
```
**Email is sent before anything else.** The portal and Telegram are conveniences;
the email is the contract with the user. If the database is down or Telegram's
API throws, the mail has already gone. Each later stage is in its own `try/except`
that only logs. The reverse — a transcription error blocking the email — never
happens, because the email builder is given whatever text we have, even `""`.
At the very top, the whole `main()` body is wrapped so that *any* unexpected
exception relays the **original** Asterisk message unchanged. Nothing we add can
silently swallow a voicemail.
---
## Why faster-whisper, CPU, int8, base.en
- **faster-whisper** (CTranslate2) is dramatically faster than OpenAI's
`whisper` on CPU — ~8 s for 23 s of 8 kHz audio here, versus minutes.
- **CPU/int8** because this host has no GPU and the voicemail rate is low. A
GPU would be idle 99.9 % of the time.
- **base.en** is plenty for clear phone audio; `small.en` only if you see
accuracy problems with names/numbers. Set `VM_WHISPER_MODEL`.
The model is loaded once per process. Asterisk fires a fresh `mailcmd` per
voicemail, so there is a ~1 s model-load cost *per message* — acceptable at PBX
volumes, and the model is pre-cached at install so the first call doesn't also
pay a download.
---
## Why the summary is extractive, not an LLM
The user explicitly declined both OpenAI and Ollama. `summarise()` is:
1. split into sentences,
2. score each by word frequency (stop-words removed) plus a boost for
position (first/last sentence) and for sentences containing digits,
3. pick the top sentences, ordered as spoken, capped at ~3 and a length limit,
4. wrap with regex intent tags (`Call back requested`, `Urgent`,
`Payment / invoice`, …) and number extraction.
It reads well on phone voicemail and needs no network. The single function
`summarise()` is a deliberate **swap-in point** — replace it with a local model
or an API call without touching the rest of the pipeline.
---
## Why audio is content-addressed
Asterisk **renumbers** `msgNNNN` files when a message is deleted from a folder.
A stored path like `…/INBOX/msg0042.wav` therefore points at a *different*
recording a week later. So:
- On store, the audio is copied to
`audio/<sha256[:2]>/<sha256>.<ext>` (blake2/sha256 in `vm_store`).
- The spool path is kept only as a *delete hint*.
- On delete, the blob is unlinked only if no other row references that hash.
This decouples the portal's playback from Asterisk's ever-shifting filenames.
---
## Why the portal is its own service behind Apache
- A long-running `uvicorn` is the natural fit; Asterisk's `mailcmd` model
(one short-lived process per call) is wrong for a web app.
- It binds **127.0.0.1:8099 only** (`IPAddressAllow=localhost` in the unit), so
the only way in is the Apache reverse proxy, which terminates TLS and enforces
headers.
- Running as the `asterisk` user means it can read `voicemail.conf` (for PIN
auth) and the spool (for future direct-file features) without widening root.
---
## Auth model
Users log in with their **mailbox number + existing voicemail PIN** parsed from
`/etc/asterisk/voicemail.conf` (`vm_auth.py`). No password database to keep in
sync, no new credentials to provision. Sessions are stored in the DB (random
token, `Secure` cookie, `HttpOnly`, `SameSite=lax`, 12 h) so logout and expiry
are enforced server-side, not just by deleting a cookie.
Every data query is filtered by the session's mailbox. Cross-mailbox access must
404 — there is a test for this in [TESTING.md](TESTING.md).
---
## Lockout
Voicemail PINs are short (often 4 digits) and the portal is internet-facing, so
failed logins are throttled in-process: 5 per `(mailbox, IP)` then 15 minutes.
State is in memory by design (single worker). A restart clears it — fine, because
a restart is an admin action, not an attacker one.
---
## Telegram routing
`telegram.conf` is re-read on every voicemail (no reload). Resolution order:
`[mailbox:N]` `chat_id` → `default_chat_id` → nothing. A chat id may fan out to
several recipients (comma list) or be a group (negative). Per-mailbox sections
can override any global (`send_audio`, `send_transcript`). Delivery degrades
`sendVoice → sendDocument → sendMessage` so a summary always gets through.
---
## Privacy posture
All audio and transcripts stay on the PBX. The only egress is:
- **Postfix → your MX** for email (the existing path),
- **Telegram API** (only if you enable Telegram),
- **Google People API** (only if you enable the `google` contacts backend).
Nothing is sent to OpenAI, Anthropic, or any transcription vendor.

159
docs/CONFIGURATION.md Normal file
View File

@ -0,0 +1,159 @@
# Configuration
All config lives in `/opt/vm-transcribe/`. Templates are committed as
`*.example`; the live files are created by `install.sh` or by you. Secret-bearing
files (`telegram.conf`, `contacts.conf`) are mode `640 root:asterisk` and git-
ignored.
Most settings can also be set via environment variables, which is how the systemd
unit wires production values. Env vars override file values when both are present
unless noted.
---
## Environment variables (shared)
| Variable | Default | Used by | Meaning |
|---|---|---|---|
| `VM_DB` | `/var/lib/vm-transcribe/voicemail.db` | store, web, import | SQLite path. Created with WAL mode if missing. |
| `VM_AUDIO_DIR` | `/var/lib/vm-transcribe/audio` | store, web, import | Where copied recordings live (`<sha[:2]>/<sha>.wav`). |
| `VM_LOG` | `/var/log/asterisk/vm_mailcmd.log` | mailcmd | Log path. |
| `VM_MODEL_CACHE` | `/opt/vm-transcribe/models` | mailcmd, import | Whisper model download/load dir. |
| `VM_WHISPER_MODEL` | `base.en` | mailcmd, import | Model name. `tiny`/`base`/`small` trade speed for accuracy. |
| `VM_SPOOL` | `/var/spool/asterisk/voicemail` | import | Spool root to scan. |
| `VM_ASTERISK_CONF` | `/etc/asterisk/voicemail.conf` | web, auth | For PIN lookup. |
---
## voicemail.conf (Asterisk)
One line is all the pipeline needs:
```ini
mailcmd=/opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_mailcmd.py
```
Place it in the `[general]` section (or any context). After editing:
```bash
sudo asterisk -rx 'voicemail reload'
```
`install.sh` backs up the file and adds this line automatically.
---
## telegram.conf
```ini
[telegram]
enabled = yes
token = 123456789:AAH... ; from @BotFather
default_chat_id = ; fallback for unmapped mailboxes (blank = none)
send_audio = yes ; voice note with caption
send_transcript = yes ; full transcript as a follow-up message
timeout = 20 ; seconds per send attempt
[mailbox:1001]
chat_id = 123456789 ; single recipient
[mailbox:1002]
chat_id = 5551, 5552 ; comma list fans out to several people
send_transcript = no ; any global key can be overridden per mailbox
[mailbox:1003]
chat_id = -1001234567890 ; negative = group chat
send_audio = no ; text-only DM
```
**Routing rules** (evaluated per voicemail):
1. Match the `[mailbox:N]` section for the voicemail's mailbox number.
2. `chat_id` may be a single id, a comma-separated list (fan-out), or a group id
(negative).
3. If no mailbox section matches, fall back to `default_chat_id`.
4. If `enabled = no` or no route resolves, nothing is sent — email still goes out.
5. The file is re-read on every voicemail; no reload needed after editing.
**Delivery fallback chain** (per chat id, tried in order): `sendVoice` →
`sendDocument` → `sendMessage` (text-only). So a Telegram outage or a bad audio
file still delivers the summary.
> Contact resolution note: a user must message the bot once before it can DM
> them. Telegram does not permit bots to initiate conversations.
---
## contacts.conf
```ini
[contacts]
backends = file ; space/comma separated: file google carddav
cache_ttl = 86400 ; seconds; cache hits AND misses
cache_file = /var/lib/vm-transcribe/contacts-cache.json
# --- 'file' backend (recommended): a vCard or CSV export -------------
[file]
path = /var/lib/vm-transcribe/contacts.vcf
# CSV (Google export "All contact data") is auto-detected:
# path = /var/lib/vm-transcribe/contacts.csv
# --- 'google' backend: People API via OAuth ---------------------------
[google]
token_file = /var/lib/vm-transcribe/google-token.json
# Obtain a token with the People API (contacts.readonly) scope. See
# docs/SECURITY.md. No app password here — OAuth only.
# --- 'carddav' backend: app password, Nextcloud/Fastmail/iCloud --------
[carddav]
# Google rejects app passwords (basic auth disabled 2024-09-30); a Google
# URL here is refused with a log warning and the backend is skipped.
url = https://contacts.fastmail.com/dav/addressbooks/user/xxx/Default
username = jp@txt3.com
app_password = ; app-specific password, not your login password
```
**Matching** is on the **last 9 digits** of the caller ID, so `+447700900123`,
`07700900123` and `447700900123` all collapse to the same contact. Names resolve
best-effort; on any backend error the caller ID is used as-is.
---
## Portal settings (per mailbox, in the SQLite store)
These are edited from the portal's **Settings** page by each user; they are not
config files. Every value defaults to `yes`.
| Setting | Effect |
|---|---|
| `summarise` | Run the summariser (if off, the raw transcript is shown). |
| `telegram` | Forward new voicemails to the mailbox's Telegram route (requires telegram.conf). |
| `email_notify` | Send the enriched email (if off, the message is stored but not emailed). |
| `highlight` | Colour the caller chip when a name is resolved from contacts. |
| `max_inbox` | Keep at most N messages; oldest beyond N are auto-deleted (0 = unlimited). |
---
## vm-portal.service
Production values are passed as environment in the unit file:
```ini
Environment=VM_DB=/var/lib/vm-transcribe/voicemail.db
Environment=VM_AUDIO_DIR=/var/lib/vm-transcribe/audio
Environment=VM_ASTERISK_CONF=/etc/asterisk/voicemail.conf
Environment=VM_SESSION_HOURS=12
```
Login hardening (in-process):
| Var | Default | Meaning |
|---|---|---|
| `VM_MAX_FAILS` | `5` | Failed attempts per (mailbox, IP) before locking. |
| `VM_LOCK_MINUTES` | `15` | Lock duration. |
| `VM_SESSION_HOURS` | `12` | Session lifetime. |
| `VM_INSECURE_COOKIE` | unset | Set to `1` only for plain-HTTP testing; normally cookies are `Secure`. |
| `VM_BASE_PATH` | `""` | Set to e.g. `/voicemail` if the app is served under a sub-path. |
> A service restart clears the in-memory lockout counters. Testing lockout will
> lock you out of your own verification for `VM_LOCK_MINUTES`.

244
docs/INSTALL.md Normal file
View File

@ -0,0 +1,244 @@
# Installation
Written against Debian 12 + Asterisk 20 + Apache 2.4 + Postfix, running under
Virtualmin. Adjust paths if your layout differs.
Everything installs to `/opt/vm-transcribe` with data in
`/var/lib/vm-transcribe`. Nothing is installed into system Python.
---
## 0. Prerequisites
```bash
# Asterisk voicemail must use file-based storage (the default), not ODBC/IMAP
grep -E '^(odbcstorage|imapserver)' /etc/asterisk/voicemail.conf # expect nothing
# tools
sudo apt install ffmpeg sox python3-venv # sox optional (gsm)
which certbot # for the portal's TLS
# an MTA on localhost:25
sudo ss -ltnp | grep ':25 '
```
**Check disk space before you start.** A full `/var` makes Postfix reject all
mail with `452 4.3.1 Insufficient system storage`, which looks like a bug in
this pipeline but is not:
```bash
df -h /var
```
---
## 1. Core pipeline
```bash
sudo scripts/install.sh
```
That script:
1. creates `/opt/vm-transcribe` and `/opt/vm-transcribe/models` owned by `asterisk`
2. builds a venv and installs `faster-whisper`
3. installs the Python modules
4. seeds `telegram.conf` / `contacts.conf` if absent (never overwrites)
5. creates `/var/log/asterisk/vm_mailcmd.log`
6. **pre-downloads the whisper model as the `asterisk` user** — do not skip
this, or the first real voicemail pays a ~150 MB download while the caller
waits for their notification
7. backs up `voicemail.conf` and rewrites `mailcmd=`
8. reloads Asterisk
Verify the wiring:
```bash
grep '^mailcmd' /etc/asterisk/voicemail.conf
# mailcmd=/opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_mailcmd.py
```
### Test it before trusting it
Build a fake notification and pipe it through **as the `asterisk` user** —
testing as yourself hides permission problems:
```bash
cd /path/to/repo
python3 tests/make_test_mail.py /path/to/some.wav you@example.com > /tmp/t.eml
sudo chmod 644 /tmp/t.eml
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
/opt/vm-transcribe/vm_mailcmd.py < /tmp/t.eml
sudo tail -5 /var/log/asterisk/vm_mailcmd.log
```
You should see `transcribed … chars` then `sent enriched notification to …`.
Confirm the mail actually left:
```bash
sudo grep "to=<you@example.com>" /var/log/mail.log | tail -1 # expect status=sent
```
Also verify the fail-safe — a message with **no** audio must relay unchanged
rather than erroring:
```bash
printf 'From: a@b\nTo: you@example.com\nSubject: no audio\n\nbody\n' > /tmp/n.eml
sudo chmod 644 /tmp/n.eml
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
/opt/vm-transcribe/vm_mailcmd.py < /tmp/n.eml
# log: "no audio attachment; relaying original"
```
---
## 2. Telegram (optional)
1. Create a bot: message [@BotFather](https://t.me/BotFather) → `/newbot` → copy the token.
2. Put it in `/opt/vm-transcribe/telegram.conf` (`sudo`, mode 640 root:asterisk).
3. **Each recipient must message the bot once** (`/start`) — Telegram forbids
bots from initiating conversations.
4. Discover chat IDs and test:
```bash
sudo /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py ids
sudo /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py test 1001
```
The `test` subcommand sends a real DM with a playable voice note and exits
non-zero on failure. See [CONFIGURATION.md](CONFIGURATION.md#telegramconf) for
per-mailbox routing.
---
## 3. Contact lookup (optional)
Simplest and most reliable — a local export, no tokens, no rate limits:
1. contacts.google.com → **Export** → **vCard**
2. `sudo install -o asterisk -g asterisk -m 640 contacts.vcf /var/lib/vm-transcribe/contacts.vcf`
3. Confirm `contacts.conf` has `backends = file` and the matching `path`.
```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
/opt/vm-transcribe/vm_contacts.py '<07700900123>'
```
> **Google app passwords do not work for contacts.** Google disabled basic auth
> for CardDAV/CalDAV/IMAP/SMTP/POP on 2024-09-30. Use the `file` or `google`
> (OAuth) backend. The `carddav` backend is for Nextcloud/Fastmail/iCloud.
---
## 4. Portal service
```bash
sudo install -d -o asterisk -g asterisk -m 750 \
/var/lib/vm-transcribe /var/lib/vm-transcribe/audio
sudo /opt/vm-transcribe/venv/bin/pip install fastapi 'uvicorn[standard]' python-multipart
sudo install -o root -g root -m 644 systemd/vm-portal.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now vm-portal
sudo systemctl is-active vm-portal
curl -s http://127.0.0.1:8099/healthz # {"ok":true,"messages":N}
```
The service runs as `asterisk` (so it can read `voicemail.conf` and the spool)
and binds **loopback only** — `IPAddressAllow=localhost` means Apache is the
only possible client.
---
## 5. Apache vhost + TLS
**Order matters.** Installing a vhost that references a cert which does not yet
exist breaks `apache2ctl configtest`, which blocks reloads for *every* site on
the box. So: DNS → cert → full vhost.
### 5a. DNS
```bash
dig +short vm.example.com A # must return your ORIGIN ip
```
Behind Cloudflare, set the record to **DNS only (grey cloud)** first, or the
HTTP-01 challenge is intercepted by the edge.
### 5b. Minimal :80 vhost, so certbot can answer the challenge
```bash
sudo install -d -o www-data -g www-data -m 755 /var/www/vm.example.com/public_html
sudo cp apache/vm.txt3.net-step1.conf /etc/apache2/sites-available/vm.example.com.conf
# edit ServerName / paths / IPs to match your host
sudo a2ensite vm.example.com
sudo apache2ctl configtest && sudo systemctl reload apache2
```
### 5c. Issue the certificate
```bash
sudo certbot certonly --webroot -w /var/www/vm.example.com/public_html \
-d vm.example.com --cert-name vm.example.com
```
### 5d. Full vhost — HTTPS + reverse proxy
```bash
sudo a2enmod proxy proxy_http headers rewrite ssl
sudo cp apache/vm.txt3.net.conf /etc/apache2/sites-available/vm.example.com.conf
# edit ServerName, cert paths, IPs
sudo install -d /var/www/vm.example.com/public_html/.well-known/acme-challenge
sudo apache2ctl configtest && sudo systemctl reload apache2
```
### 5e. Verify through the real hostname
Not localhost — the whole point is to exercise Apache, TLS and the proxy:
```bash
R="--resolve vm.example.com:443:YOUR.ORIGIN.IP"
curl -s $R https://vm.example.com/healthz
curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.example.com/login
curl -s $R -D- -o /dev/null https://vm.example.com/login | grep -i 'strict-transport\|content-security'
```
Behind a CDN, `curl https://host/` tests the *CDN*, not your origin. Always
`--resolve` to the origin IP when verifying a change.
---
## 6. Backfill existing voicemails
```bash
V=/opt/vm-transcribe/venv/bin/python3
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --dry-run # preview
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --limit 5 # trial
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py # all
```
Idempotent — safe to re-run; it resumes rather than duplicating. Expect a
`no_speech` count: 44-byte WAVs are hung-up calls with no audio, not failures.
Reckon on ~1.5 s per message plus transcription time; 157 messages took 3m40s
on 4 cores.
---
## 7. Log in
Browse to `https://vm.example.com/` and log in with a **mailbox number and its
existing voicemail PIN** from `voicemail.conf`. No new passwords are created.
---
## Uninstall
```bash
sudo systemctl disable --now vm-portal
sudo rm /etc/systemd/system/vm-portal.service && sudo systemctl daemon-reload
sudo a2dissite vm.example.com && sudo systemctl reload apache2
# restore the original mailcmd
sudo cp /etc/asterisk/voicemail.conf.bak-<timestamp> /etc/asterisk/voicemail.conf
sudo asterisk -rx 'voicemail reload'
sudo rm -rf /opt/vm-transcribe # keep /var/lib/vm-transcribe for the data
```

138
docs/OPERATIONS.md Normal file
View File

@ -0,0 +1,138 @@
# Operations
Day-to-day running: backups, monitoring, common problems, upgrades.
---
## Service status
```bash
sudo systemctl status vm-portal # the portal
sudo journalctl -u vm-portal -n 50 # portal logs (uvicorn access/startup)
sudo tail -f /var/log/asterisk/vm_mailcmd.log # per-voicemail pipeline log
```
Health of the portal (works locally or via the real hostname):
```bash
curl -s http://127.0.0.1:8099/healthz # {"ok":true,"messages":N}
curl -s --resolve vm.txt3.net:443:ORIGIN_IP https://vm.txt3.net/healthz
```
---
## Logs to watch
| Signal | Where | Meaning |
|---|---|---|
| `transcribed …` | vm_mailcmd.log | a voicemail was processed OK |
| `stored message for mailbox …` | vm_mailcmd.log | it also reached the portal DB |
| `no audio attachment; relaying original` | vm_mailcmd.log | Asterisk sent a text-only notice (missed-call style) — expected |
| `Telegram disabled` / `no route` | vm_mailcmd.log | Telegram skipped (per config) — expected if unset |
| `relaying original, mailcmd error` | vm_mailcmd.log | **pipeline threw**; original email was preserved (fail-safe worked) |
| `status=sent` | /var/log/mail.log | the enriched email left Postfix |
| `452 4.3.1 Insufficient system storage` | /var/log/mail.log | **/var is full** — see below |
---
## Backups
Two things to back up — the code is reproducible, the *data* is not:
```bash
# database + audio (once or twice a day is plenty)
sudo -u asterisk tar czf /backup/vm-$(date +%F).tgz \
-C / var/lib/vm-transcribe/voicemail.db var/lib/vm-transcribe/audio
# config (the live, secret-bearing files)
sudo tar czf /backup/vm-conf-$(date +%F).tgz \
/opt/vm-transcribe/telegram.conf /opt/vm-transcribe/contacts.conf
```
The whisper model cache (`/opt/vm-transcribe/models`) is reproducible — no need
to back it up, just re-run `install.sh`.
Restoring: stop the service, extract, `systemctl start vm-portal`. The DB schema
is created on first connect, but **preserve the existing file** to keep history.
---
## Disk
`/var/lib/vm-transcribe/audio` grows with every voicemail. At ~290 KB per
WAV (8000 Hz, mono, 23 s) that is ~12 MB per 40 messages. The portal's
`max_inbox` per-user auto-delete keeps individual mailboxes bounded, but the
*total* store only shrinks when messages are deleted (and their audio blob is
unreferenced by any other message).
A full `/var` is the single most common cause of "voicemail emails stopped
arriving" — Postfix rejects everything with `452 4.3.1 Insufficient system
storage`. Check `df -h /var` first when the pipeline looks dead.
---
## Upgrading
```bash
git -C /home/jp/Work/asterisk-voicemail pull
cd /home/jp/Work/asterisk-voicemail
sudo cp src/*.py /opt/vm-transcribe/
sudo /opt/vm-transcribe/venv/bin/pip install -U faster-whisper # occasionally
sudo systemctl restart vm-portal
```
`install.sh` is idempotent-ish for the first install but is **not** a general
upgrade tool — it backs up `voicemail.conf` each run, so avoid re-running it
blindly. For upgrades, copy `src/*.py` as above.
---
## Common problems
**No transcription email arrived.**
1. `df -h /var` — full disk blocks Postfix.
2. `sudo grep '^mailcmd' /etc/asterisk/voicemail.conf` — is it our script?
3. `sudo tail /var/log/asterisk/vm_mailcmd.log` — look for `relaying original, mailcmd error`.
4. `sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 -c 'import faster_whisper'` — is the venv intact?
**Portal returns 502 / can't connect.**
The uvicorn backend is down or not on loopback: `sudo systemctl status vm-portal`;
`ss -ltnp | grep 8099`. Also confirm Apache has `proxy`/`proxy_http` enabled.
**Login fails for a real mailbox.**
The PIN in `voicemail.conf` is what the portal checks — not anything in the
portal. If you changed a voicemail PIN, that change is picked up immediately
(the file is reparsed each login). If the mailbox line is commented out or in a
context the parser doesn't reach, login fails. Test the parser directly:
```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
-c "import sys; sys.path.insert(0,'/opt/vm-transcribe'); import vm_auth; print(vm_auth.check_login('7940','5159'))"
```
**Certbot renewal.**
Renewal runs from `/etc/cron.d/certbot`; `certbot.timer` is masked (normal on
Debian's package). Test non-destructively in the background (it can be slow):
```bash
sudo certbot renew --cert-name vm.txt3.net --dry-run &
```
**I locked myself out testing lockout.**
Restart the service: `sudo systemctl restart vm-portal`. In-memory counters clear.
**Backfill shows many `no_speech`.**
Expected. 44-byte WAVs (`duration=0`) are hung-up calls with no audio. Not a bug.
---
## Re-running the backfill
Idempotent — safe any time:
```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_import.py
```
Useful after importing a large batch of new spool messages, or after a fresh
install on a box with existing voicemails.

117
docs/SECURITY.md Normal file
View File

@ -0,0 +1,117 @@
# Security
Threat model and the hardening already in place. Read before exposing the portal
on a hostile network.
---
## What we are protecting
Voicemail is sensitive: it contains callers' names, numbers, and the content of
their messages — often personal or commercial. The portal also proves who
*isn't* a mailbox owner (knowing a PIN is the only gate). Breaches of concern:
1. An attacker reads someone else's voicemail (broken authz or IDOR).
2. An attacker brute-forces a 4-digit PIN at scale.
3. A stored credential (Telegram token, contacts app password) leaks.
4. XSS / injection poisons the HTML shown to a user.
5. The recording audio leaks off-box.
---
## What is already done
**Authentication — mailbox PIN from voicemail.conf.**
No new password store to breach; the portal trusts Asterisk's PINs. Sessions are
DB-backed (random token, `Secure`+`HttpOnly`+`SameSite=lax`, 12 h). Logout and
expiry are enforced server-side, not merely by dropping the cookie.
**Authorization — every query is mailbox-scoped.**
`vm_store` and `vm_web` filter every read and delete by the session's mailbox.
Cross-mailbox reads/deletes return **404**. There is a test that asserts this
([TESTING.md](TESTING.md)). A mailbox owner can only ever see their own messages.
**Brute-force lockout.**
5 failed attempts per `(mailbox, IP)` → 15-minute lockout. The correct PIN is
also refused while locked. State is in-memory (single worker); a restart clears
it.
**Transport — TLS terminated at Apache, with headers.**
`Strict-Transport-Security`, `X-Frame-Options: DENY`, `X-Content-Type-Options:
nosniff`, `Referrer-Policy`, and a tight CSP:
```
default-src 'self'; style-src 'self' 'unsafe-inline';
media-src 'self'; img-src 'self' data:;
script-src 'none'; frame-ancestors 'none';
base-uri 'none'; form-action 'self'
```
**No JavaScript in the portal at all**, which is what lets `script-src 'none'`
be genuinely enforceable — there is nothing to inject.
**Network exposure is minimal.**
The uvicorn backend binds **loopback only** (`IPAddressAllow=localhost` in the
unit). The only ingress is Apache. The systemd unit also sets
`NoNewPrivileges`, `ProtectSystem=full`, `ProtectHome`, `PrivateTmp`,
`ProtectKernelTunables`, `ProtectControlGroups`, `RestrictSUIDSGID`, and writable
paths are limited to `/var/lib/vm-transcribe` and `/var/log/asterisk`.
**Secrets are separated and git-ignored.**
`telegram.conf`, `contacts.conf`, `*.db`, `audio/` are in `.gitignore`. The
repo ships `*.example` templates only. Both secret files are mode `640 root:asterisk`.
**Fail-safe preserves mail.**
A pipeline exception relays the original Asterisk message unchanged — we never
lose a notification to a bug we introduced.
---
## Residual risks and how to close them
**(R1) 4-digit PINs.** Lockout helps but a distributed attack from many IPs
defeats per-IP throttling. If the portal is on a hostile network, put it behind
your VPN (there is already a `t01.vpn.conf` on this host) or fail2ban the Apache
access log. Recommended.
**(R2) No TLS on the loopback hop.** Apache→uvicorn is plain HTTP on localhost.
Acceptable (same host, no network path), but if you ever run uvicorn on another
host, use a unix socket or mTLS.
**(R3) Session token in SQLite.** If the DB file is stolen, sessions are
replayable until they expire. DB is `640 asterisk:asterisk` and not web-served.
For higher assurance, store sessions in a server-side cache with shorter TTLs.
**(R4) Contact resolver tokens.** A Google OAuth token or a CardDAV app password
grants read access to your address book. Scope the Google token to
`contacts.readonly`, use a dedicated app password (never your login password),
and keep these files `640`. Prefer the `file` backend (a periodic vCard export) —
no live token at all.
**(R5) Telegram bot token.** Whoever holds it can post as your bot. Keep
`telegram.conf` `640 root:asterisk`; rotate via @BotFather if leaked.
**(R6) Content-addressed audio filenames.** The sha256 of the audio is in the URL
(`/audio/<sha>`). An attacker who guesses a sha could fetch that recording
without a session. The portal checks the session's mailbox *owns* that message
before serving, so this is not directly exploitable — but consider a random
per-message token instead of the content hash if you want defence in depth.
**(R7) CDN / edge.** If `vm.txt3.net` is orange-clouded at Cloudflare, recording
audio streams through the edge. Grey-cloud it (DNS only) to keep voice data off
the CDN. Your call.
---
## Obtaining a Google token (if you use the `google` backend)
1. Google Cloud console → OAuth consent screen (External) → add your account as
a test user.
2. Credentials → OAuth client ID → **Desktop app**.
3. Scope `https://www.googleapis.com/auth/contacts.readonly`.
4. Authorize once (the token is written to `google-token.json`, mode `640`).
The People API is used, not CardDAV, because basic auth / app passwords were
disabled by Google on **2024-09-30**.
There is no support for an app-password read of Google Contacts — it does not
work. Use the `file` or `google` backend.

159
docs/TESTING.md Normal file
View File

@ -0,0 +1,159 @@
# Testing
How to verify each part of the system. Do the mailcmd tests **as the `asterisk`
user**, not as yourself — that's where permission bugs hide.
---
## 0. Pre-checks
```bash
df -h /var # must not be full (Postfix 452)
grep '^mailcmd' /etc/asterisk/voicemail.conf
sudo systemctl is-active vm-portal
curl -s http://127.0.0.1:8099/healthz
```
---
## 1. Unit tests (no sudo, no asterisk)
```bash
cd /home/jp/Work/asterisk-voicemail
./venv/bin/python tests/test_telegram.py # routing, caption clipping, opus transcode
./venv/bin/python tests/test_contacts.py # vCard/CSV parse, digit-normalised match
```
`test_telegram.py` asserts: per-mailbox → one chat; comma list → N chats;
group id negative; unmapped → default; `enabled=no` → no route; caption clipped
to 1017 chars on a word boundary when over 1024; a 374 KB wav → 64 KB `OggS`
opus.
`test_contacts.py` asserts: `+447700900123`, `07700900123`, `447700900123` all
match one contact (last-9-digit key); multi-TEL cards; Google CSV `:::` split.
---
## 2. Build a fake voicemail and run the real pipeline
```bash
# needs ffmpeg; uses test_vm.wav shipped in the repo (or any wav)
python3 tests/make_test_mail.py test_vm.wav you@example.com > /tmp/t.eml
sudo chmod 644 /tmp/t.eml
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
/opt/vm-transcribe/vm_mailcmd.py < /tmp/t.eml
sudo tail -5 /var/log/asterisk/vm_mailcmd.log
```
Expect `transcribed … chars` then `sent enriched notification to you@example.com`.
Then confirm real delivery:
```bash
sudo grep "to=<you@example.com>" /var/log/mail.log | tail -1 # status=sent
```
### Fail-safe: a message with no audio
```bash
printf 'From: a@b\nTo: you@example.com\nSubject: no audio\n\nbody\n' > /tmp/n.eml
sudo chmod 644 /tmp/n.eml
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
/opt/vm-transcribe/vm_mailcmd.py < /tmp/n.eml
# log must show: "no audio attachment; relaying original"
```
### Fail-safe: a thrown error relays the original
Temporarily point `VM_DB` at an unreadable path; the script should fall back to
relaying the original Asterisk message and log `relaying original, mailcmd error`.
---
## 3. Telegram (needs a real token)
```bash
sudo /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py ids
# each recipient must have messaged the bot once (/start) to appear
sudo /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py test 1001
# sends a real voice note + summary; exits non-zero on any failure
```
Offline, the routing logic is covered by `tests/test_telegram.py` so you don't
need a token to verify the config parser.
---
## 4. Contacts
```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
/opt/vm-transcribe/vm_contacts.py '<07700900123>'
# resolves against the configured backend; prints name or the raw caller id
```
---
## 5. Portal — auth, authz, playback, delete
Run the app locally (or against the live service) and exercise it with curl.
Use `VM_INSECURE_COOKIE=1` only for plain-HTTP local tests so the `Secure`
cookie can be set.
```bash
# login ok
curl -s -c /tmp/j -o /dev/null -w '%{redirect_url}\n' \
-d 'mailbox=7940&pin=5159' http://127.0.0.1:8099/login
# wrong pin rejected
curl -s -o /dev/null -w '%{http_code}\n' -d 'mailbox=7940&pin=1111' http://127.0.0.1:8099/login
# authenticated list
curl -s -b /tmp/j http://127.0.0.1:8099/ | grep -c 'class="card'
# audio streams
curl -s -b /tmp/j -o /tmp/x.wav http://127.0.0.1:8099/audio/1
file /tmp/x.wav # RIFF WAVE, 8000 Hz mono
# delete
curl -s -b /tmp/j -X POST -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8099/delete/1
```
### Authorization (IDOR) — must 404
A session for mailbox A must never reach mailbox B's data. With a cookie for
mailbox 7940, hitting `/audio/<id-owned-by-1001>` and `/delete/<id-owned-by-1001>`
must both return **404/303**, never serve or delete the other mailbox's message.
Add this assertion whenever you change `vm_store` or `vm_web`.
---
## 6. Portal over the real URL (TLS + proxy)
```bash
R="--resolve vm.txt3.net:443:ORIGIN.IP"
curl -s $R https://vm.txt3.net/healthz
curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.txt3.net/login
curl -s $R -D- -o /dev/null https://vm.txt3.net/login | grep -iE 'strict-transport|content-security|x-frame'
# http -> https redirect
curl -s $R -o /dev/null -w '%{redirect_url}\n' http://vm.txt3.net/
```
Behind a CDN, `--resolve` to the **origin** IP; otherwise you are testing the CDN,
not your server.
---
## 7. Backfill
```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
/opt/vm-transcribe/vm_import.py --dry-run --limit 5
# preview: lists 5 messages with caller + date, no writes
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
/opt/vm-transcribe/vm_import.py --limit 3
# real: 3 imported, others skipped; re-running is a no-op (idempotent)
```
Verify they appear in the portal and are playable (§5). Expect a `no_speech`
count — 44-byte WAVs are hung-up calls, not failures.