jp/asterisk-voicemail
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update all documentation to current MySQL + React SPA + vm_api.py architecture

Browse Source 
README, docs/{ARCHITECTURE,CONFIGURATION,INSTALL,OPERATIONS,SECURITY,TESTING}
and CHANGELOG were stale: they described the original SQLite + CardDAV +
zero-JS vm_web portal. Brought them in line with the actual code:
- MySQL (PyMySQL) as the default store; SQLite fallback only
- vm_contacts resolves from the MySQL contacts table (file/google/carddav removed)
- React SPA frontend + vm_api.py JSON API (:8098); vm_web.py now serves /audio/
- Apache vhost proxies /api/ -> :8098, /audio/ -> :8099, serves the SPA
- CSP relaxed to script-src 'self' 'unsafe-inline' (UI is now JavaScript)
- Added CHANGELOG 1.2.0 entry for the React/vm_api frontend work
This commit is contained in:
jp
2026-08-13 20:20:39 +01:00
parent 3155020f67
commit b8c60605fd
8 changed files with 553 additions and 290 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
CHANGELOG.md
View File

@ -1,5 +1,27 @@
# Changelog # Changelog
## 1.2.0 — 2026-08-13
- **React SPA frontend + JSON API.** Added `vm_api.py` (FastAPI JSON API on
:8098) and a React frontend (`/home/jp/Work/voicemail-ui`, served as static
files with an SPA fallback by Apache). The portal UI is now a JavaScript SPA
(react-router-dom + TanStack Query + Vite), not the zero-JS server-rendered
app. `vm_web.py` (:8099) remains to serve `/audio/` playback and `/healthz`.
- **Apache vhost updated** to proxy `/api/` → :8098, `/audio/` → :8099, serve
the React `dist`, and SPA-fallback to `index.html`. CSP relaxed to
`script-src 'self' 'unsafe-inline'` (was `script-src 'none'`) because the UI
is now JavaScript.
- **Per-message / per-contact dropdown menus** in the UI (⋮): **Call back**
opens a `tel:` link from the sanitised caller digits; **Lookup number** opens
a who-called.co.uk reverse-lookup in a new tab, using the **0-prefixed** UK
number (swap `+44` → `0` before lookup, search by sanitised digits only).
Applied to both the Messages and Contacts pages.
- **Contacts backfill on write.** Creating/updating a contact via the API now
updates matching `messages.contact_name`/`contact_email` (spaces/quotes
stripped before the LIKE; matches both E.164 and 0-prefix variants).
- Documentation (`README.md`, `docs/*`) brought in line with the MySQL +
React + `vm_api.py` architecture.
## 1.1.0 — 2026-08-13 ## 1.1.0 — 2026-08-13
- **Storage migrated to MySQL**: `vm_store` now defaults to MySQL (PyMySQL) - **Storage migrated to MySQL**: `vm_store` now defaults to MySQL (PyMySQL)
@ -17,6 +39,9 @@
cannot shadow real names. cannot shadow real names.
- **Migrations**: `vm_migrate_sqlite_to_mysql.py` and `vm_backfill_cdr.py` - **Migrations**: `vm_migrate_sqlite_to_mysql.py` and `vm_backfill_cdr.py`
added under `src/` for first-run and CDR backfill. added under `src/` for first-run and CDR backfill.
- **Contacts backend consolidated to MySQL only.** The `file` / `google` /
`carddav` resolver backends were removed; `vm_contacts` now resolves
exclusively from the MySQL `contacts` table.
## 1.0.0 — 2026-08-13 ## 1.0.0 — 2026-08-13
@ -31,17 +56,12 @@ Initial build on mail.txt3.net (Debian 12, Asterisk 20, Apache 2.4, Postfix).
any error (fail-safe). any error (fail-safe).
- **Telegram** (optional): per-mailbox routed voice-note DM with summary caption - **Telegram** (optional): per-mailbox routed voice-note DM with summary caption
and transcript follow-up; degrades sendVoice → sendDocument → sendMessage. and transcript follow-up; degrades sendVoice → sendDocument → sendMessage.
- **Contacts** (optional): caller-ID → name via `file` (vCard/CSV export),
`google` (People API OAuth), or `carddav` (app password; Nextcloud/Fastmail/
iCloud). Last-9-digit matching. Note: Google app passwords do not work.
- **Web portal**: FastAPI app at `https://vm.txt3.net` — PIN login (mailbox + - **Web portal**: FastAPI app at `https://vm.txt3.net` — PIN login (mailbox +
voicemail PIN from `voicemail.conf`), list, play, download, delete, per-user voicemail PIN from `voicemail.conf`), list, play, download, delete, per-user
settings. Zero JavaScript, `script-src 'none'` CSP, loopback-only backend settings. DB-backed sessions. Brute-force lockout per (mailbox, IP).
behind an Apache TLS reverse proxy. Brute-force lockout per (mailbox, IP). - **Storage**: content-addressed audio (decoupled from Asterisk's renumbering).
- **Storage**: SQLite store with content-addressed audio (decoupled from
Asterisk's renumbering) and DB-backed sessions.
- **Backfill**: `vm_import.py` imports and transcribes existing spool recordings; - **Backfill**: `vm_import.py` imports and transcribes existing spool recordings;
idempotent on `(mailbox, origtime, callerid)`. Backfilled 157 historical idempotent on `(mailbox, origtime, callerid)`. Backfilled 157 historical
messages on first run (3m40s, 0 failures). messages on first run (3m40s, 0 failures).
- **Packaging**: venv at `/opt/vm-transcribe`, systemd `vm-portal.service`, - **Packaging**: venv at `/opt/vm-transcribe`, systemd units, Apache vhost +
Apache vhost + certbot TLS, `install.sh`. certbot TLS, `install.sh`.

118
README.md
View File

@ -9,31 +9,16 @@ manage:
- **Emails** a graphically designed `multipart/alternative` notification - **Emails** a graphically designed `multipart/alternative` notification
(plain + HTML) with the recording attached (plain + HTML) with the recording attached
- **DMs Telegram** optionally, as a playable voice note with the summary - **DMs Telegram** optionally, as a playable voice note with the summary
- **Resolves caller ID to a name** from Google Contacts or any CardDAV server - **Resolves caller ID to a name** from the MySQL `contacts` table
- **Serves a web portal** where mailbox users log in with their existing phone - **Serves a React web portal** where mailbox users log in with their existing
PIN to read transcripts, play/download recordings, delete messages and phone PIN to read transcripts, play/download recordings, delete messages,
manage their own notification settings manage contacts and look up unknown numbers
Everything runs on the PBX host. No third-party service sees your voicemail. Everything runs on the PBX host. No third-party service sees your voicemail.
--- ---
## What it looks like ## Architecture (current)
**Email notification** — styled HTML card with metadata, summary panel, intent
chips, a `tel:` callback link, full transcript and the audio attached. A plain
text alternative is always included.
**Telegram** — voice note (ogg/opus) with the summary as its caption, intent
hashtags, and the transcript as a follow-up message.
**Portal** — one card per voicemail: caller, timestamp, duration, summary,
tags, callback link, inline player, collapsible transcript, and Mark read /
Download / Delete.
---
## Architecture
``` ```
incoming call incoming call
@ -49,20 +34,28 @@ Download / Delete.
┌──────────────┬──────────────┬────┴─────────┬──────────────┐ ┌──────────────┬──────────────┬────┴─────────┬──────────────┐
▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
faster-whisper summarise() vm_contacts vm_store vm_telegram faster-whisper summarise() vm_contacts vm_store vm_telegram
(transcribe) + intents (name lookup) (SQLite + (voice note (transcribe) + intents (MySQL name) (MySQL + (voice note
+ numbers audio CAS) DM) + numbers audio CAS) DM)
│ │ │ │
▼ ▼ ▼ ▼
multipart email ──▶ Postfix :25 ┌──────────────┐ multipart email ──▶ Postfix :25 ┌──────────────────────────┐
│ vm_web.py │ │ MySQL (asterisk DB) │
│ (FastAPI) │ │ messages, contacts, CDR │
└──────┬───────┘ └──────┬───────────┬─────────┘
│ :8099 loopback │ │
▼ ┌──────────────────────┘ └──────────────────────┐
Apache (TLS) ▼ ▼
│ ┌──────────────────┐ ┌──────────────────┐
│ vm_api.py :8098 │ JSON API (React SPA) │ vm_web.py :8099 │
│ (FastAPI) │ ◀── React frontend ──┐ │ (FastAPI) │
└────────┬─────────┘ │ │ /audio/ legacy │
│ Apache /api/ proxy │ └────────┬─────────┘
▼ │ │ Apache /audio/ proxy
Apache (TLS) ◀─────────────────────────┘ ▼
│ Apache (TLS)
▼ ▼
https://vm.txt3.net https://vm.txt3.net
(React SPA + JSON API; audio proxied to :8099)
``` ```
**Delivery order is deliberate**: email first, then the database, then Telegram. **Delivery order is deliberate**: email first, then the database, then Telegram.
@ -70,6 +63,13 @@ Each later stage is wrapped so a failure only logs. If anything throws at the
top level, the *original* Asterisk notification is relayed unchanged. A top level, the *original* Asterisk notification is relayed unchanged. A
voicemail notification is never lost because a summariser or an API failed. voicemail notification is never lost because a summariser or an API failed.
**Two backends, one MySQL store.** The *interactive* portal is a React single
page app (built from `/home/jp/Work/voicemail-ui`, served as static files by
Apache) that talks to a JSON API (`vm_api.py` on :8098). The original
server-rendered FastAPI app (`vm_web.py` on :8099) still runs but now only
serves `/audio/` playback (proxied by Apache). Both backends read/write the
same MySQL `asterisk` database.
--- ---
## Components ## Components
@ -77,13 +77,18 @@ voicemail notification is never lost because a summariser or an API failed.
| File | Role | | File | Role |
|---|---| |---|---|
| `src/vm_mailcmd.py` | The `mailcmd` — entry point for every voicemail. Orchestrates everything. | | `src/vm_mailcmd.py` | The `mailcmd` — entry point for every voicemail. Orchestrates everything. |
| `src/vm_store.py` | SQLite store: schema, settings, sessions, content-addressed audio. | | `src/vm_store.py` | Store. **MySQL** (PyMySQL) by default; SQLite fallback only if no `db_secret`/MySQL env. |
| `src/vm_telegram.py` | Telegram delivery, per-mailbox routing. | | `src/vm_api.py` | FastAPI **JSON API** (:8098) for the React portal. Login, messages, settings, contacts CRUD. |
| `src/vm_contacts.py` | Caller-ID → name via local export, Google People API, or CardDAV. | | `src/vm_web.py` | FastAPI HTML portal (:8099). Now primarily serves `/audio/` playback + `/healthz`. |
| `src/vm_web.py` | FastAPI portal: login, list, play, delete, settings. | | `src/vm_contacts.py` | Caller-ID → name/email **from the MySQL `contacts` table** (MySQL-only). |
| `src/vm_auth.py` | Parses `voicemail.conf` so users log in with their phone PIN. | | `src/vm_auth.py` | Parses `voicemail.conf` so users log in with their phone PIN. |
| `src/vm_telegram.py` | Telegram delivery, per-mailbox routing. |
| `src/vm_import.py` | Backfills existing spool recordings into the database. | | `src/vm_import.py` | Backfills existing spool recordings into the database. |
| `src/vm_backfill_cdr.py` | Imports Asterisk `Master.csv` CDR history into MySQL. |
| `src/vm_migrate_sqlite_to_mysql.py` | One-off migration of the legacy SQLite store to MySQL. |
| `src/vm_import_contacts.py` | Imports a vCard/CSV export into the MySQL `contacts` table. |
| `src/vm_tg_setup.py` | Helper: discover Telegram chat IDs, test a route. | | `src/vm_tg_setup.py` | Helper: discover Telegram chat IDs, test a route. |
| `frontend/` | Synced copy of the React frontend (`/home/jp/Work/voicemail-ui`). |
--- ---
@ -92,19 +97,28 @@ voicemail notification is never lost because a summariser or an API failed.
See **[docs/INSTALL.md](docs/INSTALL.md)** for the full walkthrough. Short version: See **[docs/INSTALL.md](docs/INSTALL.md)** for the full walkthrough. Short version:
```bash ```bash
# 1. Backend services (API + legacy audio server)
sudo scripts/install.sh # venv, deps, model cache, mailcmd wiring sudo scripts/install.sh # venv, deps, model cache, mailcmd wiring
sudo cp systemd/vm-api.service /etc/systemd/system/
sudo cp systemd/vm-portal.service /etc/systemd/system/ sudo cp systemd/vm-portal.service /etc/systemd/system/
sudo systemctl enable --now vm-portal sudo systemctl daemon-reload
# then follow docs/INSTALL.md §5 for the Apache vhost + TLS sudo systemctl enable --now vm-api vm-portal
# 2. React frontend (build in /home/jp/Work/voicemail-ui, rsync here)
cd /home/jp/Work/voicemail-ui && npm install && npm run build
sudo -A rsync -a --exclude node_modules --exclude dist \
/home/jp/Work/voicemail-ui/ /home/txt3/domains/vm.txt3.net/public_html/
# 3. Apache vhost + TLS — see docs/INSTALL.md §5
``` ```
Backfill your existing voicemails: Backfill your existing voicemails and CDR:
```bash ```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ V=/opt/vm-transcribe/venv/bin/python3
/opt/vm-transcribe/vm_import.py --dry-run # preview sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --dry-run # preview
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ sudo -u asterisk $V /opt/vm-transcribe/vm_import.py # do it
/opt/vm-transcribe/vm_import.py # do it sudo -u asterisk $V /opt/vm-transcribe/vm_backfill_cdr.py # CDR history
``` ```
--- ---
@ -113,10 +127,10 @@ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
| Document | Contents | | Document | Contents |
|---|---| |---|---|
| [docs/INSTALL.md](docs/INSTALL.md) | Step-by-step install, including TLS ordering | | [docs/INSTALL.md](docs/INSTALL.md) | Step-by-step install, MySQL setup, TLS ordering, frontend build |
| [docs/CONFIGURATION.md](docs/CONFIGURATION.md) | Every config file and option | | [docs/CONFIGURATION.md](docs/CONFIGURATION.md) | MySQL credentials, env vars, every config file and option |
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Design decisions and why | | [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Design decisions and why |
| [docs/OPERATIONS.md](docs/OPERATIONS.md) | Day-to-day running, backup, troubleshooting | | [docs/OPERATIONS.md](docs/OPERATIONS.md) | Day-to-day running, backup (incl. MySQL), troubleshooting |
| [docs/SECURITY.md](docs/SECURITY.md) | Threat model, hardening, privacy | | [docs/SECURITY.md](docs/SECURITY.md) | Threat model, hardening, privacy |
| [docs/TESTING.md](docs/TESTING.md) | How to verify each part | | [docs/TESTING.md](docs/TESTING.md) | How to verify each part |
| [CHANGELOG.md](CHANGELOG.md) | Version history | | [CHANGELOG.md](CHANGELOG.md) | Version history |
@ -126,11 +140,13 @@ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
## Requirements ## Requirements
- Asterisk with `app_voicemail` (file-based spool, not ODBC/IMAP storage) - Asterisk with `app_voicemail` (file-based spool, not ODBC/IMAP storage)
- Python 3.9+ - Python 3.9+ with `pymysql` (backend now MySQL-first)
- MySQL/MariaDB (the same `asterisk` DB Asterisk's CDR uses)
- `ffmpeg` (Telegram voice-note transcoding; `sox` optionally for gsm) - `ffmpeg` (Telegram voice-note transcoding; `sox` optionally for gsm)
- An MTA listening on `localhost:25` (Postfix here) - An MTA listening on `localhost:25` (Postfix here)
- ~200 MB disk for the whisper `base.en` model - ~200 MB disk for the whisper `base.en` model
- Apache with `proxy`, `proxy_http`, `headers`, `rewrite`, `ssl` for the portal - Apache with `proxy`, `proxy_http`, `headers`, `rewrite`, `ssl` for the portal
- Node 18+ + npm for the React frontend build
Runs comfortably on 4 CPU cores with no GPU: ~8 s to transcribe 23 s of audio. Runs comfortably on 4 CPU cores with no GPU: ~8 s to transcribe 23 s of audio.
@ -148,12 +164,16 @@ keeps voicemail content on your own hardware. `summarise()` in
pointing at the wrong recording. Recordings are copied to pointing at the wrong recording. Recordings are copied to
`audio/<sha[:2]>/<sha>.wav`. `audio/<sha[:2]>/<sha>.wav`.
**App passwords cannot read Google Contacts.** Google disabled basic auth for **Contacts live in MySQL.** `vm_contacts.py` resolves caller IDs exclusively
CardDAV/CalDAV/IMAP/SMTP/POP on 2024-09-30. Use the `file` backend (a vCard from the `asterisk.contacts` table (E.164 canonical; `+44`↔`0` normalised so
export) or the `google` backend (OAuth). The `carddav` backend with an app both forms match). Creating/updating a contact backfills matching
password works for Nextcloud, Fastmail and iCloud. `messages.contact_name`/`contact_email`. The older `file` / `google` / `carddav`
backends were removed in favour of the MySQL table.
**The portal ships zero JavaScript**, which lets its CSP be `script-src 'none'`. **The portal ships JavaScript now.** The interactive UI is a React SPA, so the
Apache CSP is relaxed to `script-src 'self' 'unsafe-inline'` (vs the old
zero-JS `script-src 'none'`). See [SECURITY.md](docs/SECURITY.md) for the
updated threat model.
--- ---

89
docs/ARCHITECTURE.md
View File

@ -1,7 +1,8 @@
# Architecture # Architecture
How the pieces fit and *why* they are shaped the way they are. Read this How the pieces fit and *why* they are shaped the way they are. Read this
before changing anything in `vm_mailcmd.py` or `vm_store.py`. before changing anything in `vm_mailcmd.py`, `vm_store.py`, `vm_api.py` or
`vm_contacts.py`.
--- ---
@ -13,12 +14,12 @@ Asterisk ──(RFC822 on stdin)──▶ vm_mailcmd.py
┌──────────┬───────────────┼───────────────┬───────────────┐ ┌──────────┬───────────────┼───────────────┬───────────────┐
▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
transcribe summarise resolve store telegram transcribe summarise resolve store telegram
(whisper) +intents contact (sqlite) (DM) (whisper) +intents contact (MySQL) (DM)
│ │ │ │ │ │ │ │ │ │
└────▶ build multipart email ─▶ Postfix :25 └────▶ build multipart email ─▶ Postfix :25
(plain + HTML + audio attached) (plain + HTML + audio attached)
│ │
└──▶ vm_store.add_message() (best-effort) └──▶ vm_store.add_message() (best-effort, MySQL)
└──▶ vm_telegram.send() (best-effort) └──▶ vm_telegram.send() (best-effort)
``` ```
@ -34,6 +35,58 @@ silently swallow a voicemail.
--- ---
## Storage: MySQL, with a SQLite escape hatch
`vm_store.py` is backend-agnostic. It defaults to **MySQL (PyMySQL)** when a
credentials source is present (`/opt/vm-transcribe/db_secret`, mode `640
root:root`, or `VM_MYSQL_*` env vars). If neither exists it falls back to the
original SQLite file (`VM_DB`).
The same `asterisk` MySQL database is shared with Asterisk's own CDR
(`cdr_adaptive_odbc` + `res_odbc` + the MariaDB ODBC driver), so voicemail
metadata and call records live side by side. This is why the migration kept the
`asterisk` DB and a least-privilege `asterisk` user rather than spinning up a
separate instance.
The connection object returned by `connect()` mimics the `sqlite3` cursor surface
callers use (`.execute()`/`.executemany()`, `.commit()`, `.close()`,
`.fetchone()`, `.fetchall()`, `lastrowid`, and `row['col']` access), so the rest
of the code is written once and works against either backend.
---
## Two backends in front of one store
- **`vm_api.py` (FastAPI, :8098)** — the JSON API consumed by the React SPA.
Login/logout, message list, mark-read, delete, settings, and full contacts
CRUD (`GET/POST /api/contacts`, `GET/PUT/DELETE /api/contacts/{id}`,
`/api/contacts/import_vcf`, `/api/contacts/history/{number}`,
`/api/add_contact`, `/api/contacts/delete_all`). Apache proxies `/api/` here.
- **`vm_web.py` (FastAPI, :8099)** — the original server-rendered HTML portal.
It still runs and now primarily serves `/audio/` playback and `/healthz`;
Apache proxies `/audio/` here. The interactive UI has moved to the React SPA.
Both read and write the same MySQL `messages` / `contacts` tables, so the React
UI and the legacy endpoints never disagree.
---
## Frontend: React SPA + Vite
The portal UI is a React app (`/home/jp/Work/voicemail-ui`) using
`react-router-dom`, TanStack Query, and a Vite build. It is built to `dist/` and
rsync'd to `/home/txt3/domains/vm.txt3.net/public_html/`. Apache serves the
static build and uses an SPA fallback (any non-`/api/`, non-`/audio/` path
returns `index.html`). All data goes through `/api/` → `vm_api.py`.
Because the UI is JavaScript, the Apache CSP is **relaxed** to
`script-src 'self' 'unsafe-inline'` (see SECURITY.md). The dark "cyber" theme
uses neon accents; callers get a dropdown (⋮) on each message/contact row with
**Call back** (`tel:` link from sanitised digits) and **Lookup number** (opens a
who-called.co.uk search in a new tab, using the 0-prefixed UK number).
---
## Why faster-whisper, CPU, int8, base.en ## Why faster-whisper, CPU, int8, base.en
- **faster-whisper** (CTranslate2) is dramatically faster than OpenAI's - **faster-whisper** (CTranslate2) is dramatically faster than OpenAI's
@ -74,23 +127,28 @@ A stored path like `…/INBOX/msg0042.wav` therefore points at a *different*
recording a week later. So: recording a week later. So:
- On store, the audio is copied to - On store, the audio is copied to
`audio/<sha256[:2]>/<sha256>.<ext>` (blake2/sha256 in `vm_store`). `audio/<sha256[:2]>/<sha256>.<ext>`.
- The spool path is kept only as a *delete hint*. - The spool path is kept only as a *delete hint*.
- On delete, the blob is unlinked only if no other row references that hash. - On delete, the blob is unlinked only if no other row references that hash.
This decouples the portal's playback from Asterisk's ever-shifting filenames. This decouples the portal's playback (served by `vm_web.py` :8099 via
`/audio/`) from Asterisk's ever-shifting filenames.
--- ---
## Why the portal is its own service behind Apache ## Contacts: MySQL only
- A long-running `uvicorn` is the natural fit; Asterisk's `mailcmd` model `vm_contacts.py` resolves caller IDs **exclusively** from the MySQL `contacts`
(one short-lived process per call) is wrong for a web app. table. The older `file` / `google` / `carddav` backends were removed.
- It binds **127.0.0.1:8099 only** (`IPAddressAllow=localhost` in the unit), so
the only way in is the Apache reverse proxy, which terminates TLS and enforces - Numbers are stored canonical E.164; `normalize_uk()` collapses `+44…` and
headers. `0…` (and `0044…`) forms so both match.
- Running as the `asterisk` user means it can read `voicemail.conf` (for PIN - Matching uses the full digit string (up to 15 digits), not just last-9, and
auth) and the spool (for future direct-file features) without widening root. guards against emoji/placeholder contacts shadowing real names.
- Creating/updating a contact backfills `messages.contact_name` /
`contact_email` for every callerid that matches (spaces/quotes stripped before
the LIKE).
- `vm_import_contacts.py` loads a vCard/CSV export into the table.
--- ---
@ -98,7 +156,7 @@ This decouples the portal's playback from Asterisk's ever-shifting filenames.
Users log in with their **mailbox number + existing voicemail PIN** parsed from Users log in with their **mailbox number + existing voicemail PIN** parsed from
`/etc/asterisk/voicemail.conf` (`vm_auth.py`). No password database to keep in `/etc/asterisk/voicemail.conf` (`vm_auth.py`). No password database to keep in
sync, no new credentials to provision. Sessions are stored in the DB (random sync, no new credentials to provision. Sessions are stored in MySQL (random
token, `Secure` cookie, `HttpOnly`, `SameSite=lax`, 12 h) so logout and expiry token, `Secure` cookie, `HttpOnly`, `SameSite=lax`, 12 h) so logout and expiry
are enforced server-side, not just by deleting a cookie. are enforced server-side, not just by deleting a cookie.
@ -132,6 +190,7 @@ All audio and transcripts stay on the PBX. The only egress is:
- **Postfix → your MX** for email (the existing path), - **Postfix → your MX** for email (the existing path),
- **Telegram API** (only if you enable Telegram), - **Telegram API** (only if you enable Telegram),
- **Google People API** (only if you enable the `google` contacts backend). - **who-called.co.uk** (only when a user clicks *Lookup number* in the UI — a
manual, user-initiated action that opens a new browser tab).
Nothing is sent to OpenAI, Anthropic, or any transcription vendor. Nothing is sent to OpenAI, Anthropic, or any transcription vendor.

141
docs/CONFIGURATION.md
View File

@ -1,9 +1,8 @@
# Configuration # Configuration
All config lives in `/opt/vm-transcribe/`. Templates are committed as The backend services run from `/opt/vm-transcribe/`. Live secret-bearing files
`*.example`; the live files are created by `install.sh` or by you. Secret-bearing live there (some git-ignored). The repo ships `*.example` templates under
files (`telegram.conf`, `contacts.conf`) are mode `640 root:asterisk` and git- `config/`.
ignored.
Most settings can also be set via environment variables, which is how the systemd Most settings can also be set via environment variables, which is how the systemd
unit wires production values. Env vars override file values when both are present unit wires production values. Env vars override file values when both are present
@ -11,17 +10,47 @@ unless noted.
--- ---
## Environment variables (shared) ## Environment variables
| Variable | Default | Used by | Meaning | ### Shared / store (`vm_store.py`, used by mailcmd, web, api, import)
|---|---|---|---|
| `VM_DB` | `/var/lib/vm-transcribe/voicemail.db` | store, web, import | SQLite path. Created with WAL mode if missing. | | Variable | Default | Meaning |
| `VM_AUDIO_DIR` | `/var/lib/vm-transcribe/audio` | store, web, import | Where copied recordings live (`<sha[:2]>/<sha>.wav`). | |---|---|---|
| `VM_LOG` | `/var/log/asterisk/vm_mailcmd.log` | mailcmd | Log path. | | `VM_DB` | `/var/lib/vm-transcribe/voicemail.db` | SQLite path — **used only as a fallback** when no MySQL is configured. |
| `VM_MODEL_CACHE` | `/opt/vm-transcribe/models` | mailcmd, import | Whisper model download/load dir. | | `VM_AUDIO_DIR` | `/var/lib/vm-transcribe/audio` | Where copied recordings live (`<sha[:2]>/<sha>.wav`). |
| `VM_WHISPER_MODEL` | `base.en` | mailcmd, import | Model name. `tiny`/`base`/`small` trade speed for accuracy. | | `VM_LOG` | `/var/log/asterisk/vm_mailcmd.log` | Log path. |
| `VM_SPOOL` | `/var/spool/asterisk/voicemail` | import | Spool root to scan. | | `VM_MODEL_CACHE` | `/opt/vm-transcribe/models` | Whisper model download/load dir. |
| `VM_ASTERISK_CONF` | `/etc/asterisk/voicemail.conf` | web, auth | For PIN lookup. | | `VM_WHISPER_MODEL` | `base.en` | Model name. `tiny`/`base`/`small` trade speed for accuracy. |
| `VM_SPOOL` | `/var/spool/asterisk/voicemail` | Spool root to scan (import). |
| `VM_ASTERISK_CONF` | `/etc/asterisk/voicemail.conf` | For PIN lookup. |
### MySQL (default store + contacts)
Set these as env vars (the API unit reads `/opt/vm-transcribe/api.env`) **or**
put them in `/opt/vm-transcribe/db_secret` (mode `640 root:root`,
`KEY=VALUE` lines). When a credentials source exists, `vm_store` uses MySQL;
otherwise it falls back to SQLite.
| Variable | Default | Meaning |
|---|---|---|
| `VM_MYSQL_HOST` | `localhost` | MySQL host. |
| `VM_MYSQL_USER` | `asterisk` | Least-privilege DB user (owns the `asterisk` DB; not root). |
| `VM_MYSQL_PASSWORD` | (from `db_secret`) | DB password. |
| `VM_MYSQL_DB` | `asterisk` | Database name — the **same** DB Asterisk's CDR uses. |
`db_secret` key names: `MYSQL_HOST`, `MYSQL_USER`, `MYSQL_PASSWORD`,
`MYSQL_DB`.
### Portal / API hardening
| Var | Default | Meaning |
|---|---|---|
| `VM_SESSION_HOURS` | `12` | Session lifetime. |
| `VM_MAX_FAILS` | `5` | Failed attempts per (mailbox, IP) before locking. |
| `VM_LOCK_MINUTES` | `15` | Lock duration. |
| `VM_INSECURE_COOKIE` | unset | Set to `1`/`yes`/`true` only for plain-HTTP testing; normally cookies are `Secure`. |
| `VM_BASE_PATH` | `""` | Set to e.g. `/voicemail` if served under a sub-path. |
| `VM_CORS_ORIGINS` | `*` | Comma list of allowed CORS origins for the JSON API. |
--- ---
@ -88,41 +117,38 @@ file still delivers the summary.
```ini ```ini
[contacts] [contacts]
backends = file ; space/comma separated: file google carddav enabled = yes
cache_ttl = 86400 ; seconds; cache hits AND misses backends = mysql ; ONLY 'mysql' is implemented (MySQL contacts table).
cache_file = /var/lib/vm-transcribe/contacts-cache.json # The schema still parses file/google/carddav entries,
# but the registry is {"mysql": lookup_mysql} so any
# other value is ignored / logged as unknown.
cache_path = /var/lib/vm-transcribe/contacts_cache.json
cache_ttl = 86400
# --- 'file' backend (recommended): a vCard or CSV export ------------- [file] ; NOT USED — kept for reference only
[file] [google] ; NOT USED — kept for reference only
path = /var/lib/vm-transcribe/contacts.vcf [carddav] ; NOT USED — kept for reference only
# CSV (Google export "All contact data") is auto-detected:
# path = /var/lib/vm-transcribe/contacts.csv
# --- 'google' backend: People API via OAuth ---------------------------
[google]
token_file = /var/lib/vm-transcribe/google-token.json
# Obtain a token with the People API (contacts.readonly) scope. See
# docs/SECURITY.md. No app password here — OAuth only.
# --- 'carddav' backend: app password, Nextcloud/Fastmail/iCloud --------
[carddav]
# Google rejects app passwords (basic auth disabled 2024-09-30); a Google
# URL here is refused with a log warning and the backend is skipped.
url = https://contacts.fastmail.com/dav/addressbooks/user/xxx/Default
username = jp@txt3.com
app_password = ; app-specific password, not your login password
``` ```
**Matching** is on the **last 9 digits** of the caller ID, so `+447700900123`, **Contacts are resolved exclusively from the MySQL `contacts` table** on the
`07700900123` and `447700900123` all collapse to the same contact. Names resolve `asterisk` DB. Numbers are stored canonical E.164; `normalize_uk()` collapses
best-effort; on any backend error the caller ID is used as-is. `+44…` and `0…` (and `0044…`) so both forms match. Matching uses the full digit
string (up to 15 digits), not just last-9. Creating/updating a contact via the
API backfills `messages.contact_name`/`contact_email` for matching callerids.
Load the table from a vCard/CSV export with:
```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
/opt/vm-transcribe/vm_import_contacts.py /path/to/contacts.vcf
```
--- ---
## Portal settings (per mailbox, in the SQLite store) ## Portal settings (per mailbox, in MySQL)
These are edited from the portal's **Settings** page by each user; they are not These are edited from the React portal's **Settings** page by each user; they are
config files. Every value defaults to `yes`. not config files. Every value defaults to `yes`.
| Setting | Effect | | Setting | Effect |
|---|---| |---|---|
@ -134,26 +160,33 @@ config files. Every value defaults to `yes`.
--- ---
## vm-portal.service ## systemd units
Production values are passed as environment in the unit file: Two units, both running as `asterisk` and binding loopback only.
### vm-api.service (:8098 JSON API)
```ini
Environment=VM_SESSION_HOURS=12
EnvironmentFile=-/opt/vm-transcribe/api.env ; MySQL creds + VM_* overrides
# Hardening: ProtectSystem=full, ProtectHome=read-only, NoNewPrivileges,
# PrivateTmp, IPAddressAllow=127.0.0.1
ExecStart=/opt/vm-transcribe/venv/bin/uvicorn vm_api:app --host 127.0.0.1 --port 8098 --workers 2
```
### vm-portal.service (:8099 legacy HTML / audio)
```ini ```ini
Environment=VM_DB=/var/lib/vm-transcribe/voicemail.db Environment=VM_DB=/var/lib/vm-transcribe/voicemail.db
Environment=VM_AUDIO_DIR=/var/lib/vm-transcribe/audio Environment=VM_AUDIO_DIR=/var/lib/vm-transcribe/audio
Environment=VM_ASTERISK_CONF=/etc/asterisk/voicemail.conf Environment=VM_ASTERISK_CONF=/etc/asterisk/voicemail.conf
Environment=VM_SESSION_HOURS=12 Environment=VM_SESSION_HOURS=12
ExecStart=/opt/vm-transcribe/venv/bin/python -m uvicorn vm_web:app \
--host 127.0.0.1 --port 8099 --proxy-headers --forwarded-allow-ips 127.0.0.1
# Hardening: NoNewPrivileges, PrivateTmp, ProtectSystem=full, ProtectHome,
# ReadWritePaths=/var/lib/vm-transcribe /var/log/asterisk,
# IPAddressAllow=localhost, IPAddressDeny=any
``` ```
Login hardening (in-process):
| Var | Default | Meaning |
|---|---|---|
| `VM_MAX_FAILS` | `5` | Failed attempts per (mailbox, IP) before locking. |
| `VM_LOCK_MINUTES` | `15` | Lock duration. |
| `VM_SESSION_HOURS` | `12` | Session lifetime. |
| `VM_INSECURE_COOKIE` | unset | Set to `1` only for plain-HTTP testing; normally cookies are `Secure`. |
| `VM_BASE_PATH` | `""` | Set to e.g. `/voicemail` if the app is served under a sub-path. |
> A service restart clears the in-memory lockout counters. Testing lockout will > A service restart clears the in-memory lockout counters. Testing lockout will
> lock you out of your own verification for `VM_LOCK_MINUTES`. > lock you out of your own verification for `VM_LOCK_MINUTES`.

197
docs/INSTALL.md
View File

@ -1,11 +1,16 @@
# Installation # Installation
Written against Debian 12 + Asterisk 20 + Apache 2.4 + Postfix, running under Written against Debian 12 + Asterisk 20 + Apache 2.4 + Postfix + MySQL/MariaDB,
Virtualmin. Adjust paths if your layout differs. running under Virtualmin. Adjust paths if your layout differs.
Everything installs to `/opt/vm-transcribe` with data in Everything installs to `/opt/vm-transcribe` with audio in
`/var/lib/vm-transcribe`. Nothing is installed into system Python. `/var/lib/vm-transcribe`. Nothing is installed into system Python.
> **Frontend lives in a separate repo.** This backend repo ships a *synced
> copy* under `frontend/`. Build the real app in `/home/jp/Work/voicemail-ui`
> and rsync the result to the web root (see §6). Keep the two in sync with
> `rsync -a /home/jp/Work/voicemail-ui/ frontend/` then commit.
--- ---
## 0. Prerequisites ## 0. Prerequisites
@ -16,7 +21,9 @@ grep -E '^(odbcstorage|imapserver)' /etc/asterisk/voicemail.conf # expect noth
# tools # tools
sudo apt install ffmpeg sox python3-venv # sox optional (gsm) sudo apt install ffmpeg sox python3-venv # sox optional (gsm)
sudo apt install default-mysql-server # or mariadb-server
which certbot # for the portal's TLS which certbot # for the portal's TLS
node --version && npm --version # Node 18+ for the frontend
# an MTA on localhost:25 # an MTA on localhost:25
sudo ss -ltnp | grep ':25 ' sudo ss -ltnp | grep ':25 '
@ -32,25 +39,52 @@ df -h /var
--- ---
## 1. Core pipeline ## 1. MySQL (storage + contacts + CDR)
The backend shares the Asterisk `asterisk` database. Create a least-privilege
user and a credentials file:
```bash
sudo mysql -e "CREATE USER IF NOT EXISTS 'asterisk'@'localhost' IDENTIFIED BY 'PICK_A_STRONG_PASSWORD';"
sudo mysql -e "GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,INDEX,ALTER ON asterisk.* TO 'asterisk'@'localhost';"
sudo mysql -e "FLUSH PRIVILEGES;"
```
Write `/opt/vm-transcribe/db_secret` (mode `640 root:root`):
```bash
sudo install -o root -g root -m 640 /dev/null /opt/vm-transcribe/db_secret
sudo tee /opt/vm-transcribe/db_secret >/dev/null <<'EOF'
MYSQL_HOST=localhost
MYSQL_USER=asterisk
MYSQL_PASSWORD=PICK_A_STRONG_PASSWORD
MYSQL_DB=asterisk
EOF
```
`vm_store` uses MySQL whenever this file (or `VM_MYSQL_*` env vars) exists;
otherwise it falls back to SQLite. The schema is created on first connect.
(Optional) To load call records into the same DB, wire Asterisk CDR via
`cdr_adaptive_odbc` + `res_odbc` + the MariaDB ODBC driver, then run
`src/vm_backfill_cdr.py` to import `Master.csv` history.
---
## 2. Core pipeline (mailcmd + venv)
The `install.sh` script builds the venv, installs `faster-whisper`, deploys the
mailcmd scripts, seeds `telegram.conf`, pre-caches the whisper model **as the
asterisk user**, and rewrites `voicemail.conf`'s `mailcmd=`.
> `install.sh` reads sources from `/home/jp/asterisk-vm` (the deploy path in
> AGENTS.md). Either symlink/clobber that path to this repo, or edit the `SRC=`
> line at the top of `scripts/install.sh` before running.
```bash ```bash
sudo scripts/install.sh sudo scripts/install.sh
``` ```
That script:
1. creates `/opt/vm-transcribe` and `/opt/vm-transcribe/models` owned by `asterisk`
2. builds a venv and installs `faster-whisper`
3. installs the Python modules
4. seeds `telegram.conf` / `contacts.conf` if absent (never overwrites)
5. creates `/var/log/asterisk/vm_mailcmd.log`
6. **pre-downloads the whisper model as the `asterisk` user** — do not skip
this, or the first real voicemail pays a ~150 MB download while the caller
waits for their notification
7. backs up `voicemail.conf` and rewrites `mailcmd=`
8. reloads Asterisk
Verify the wiring: Verify the wiring:
```bash ```bash
@ -64,7 +98,7 @@ Build a fake notification and pipe it through **as the `asterisk` user** —
testing as yourself hides permission problems: testing as yourself hides permission problems:
```bash ```bash
cd /path/to/repo cd /home/jp/Work/asterisk-voicemail
python3 tests/make_test_mail.py /path/to/some.wav you@example.com > /tmp/t.eml python3 tests/make_test_mail.py /path/to/some.wav you@example.com > /tmp/t.eml
sudo chmod 644 /tmp/t.eml sudo chmod 644 /tmp/t.eml
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
@ -76,7 +110,7 @@ You should see `transcribed … chars` then `sent enriched notification to …`.
Confirm the mail actually left: Confirm the mail actually left:
```bash ```bash
sudo grep "to=<you@example.com>" /var/log/mail.log | tail -1 # expect status=sent sudo grep "to=<you@example.com>" /var/log/mail.log | tail -1 # status=sent
``` ```
Also verify the fail-safe — a message with **no** audio must relay unchanged Also verify the fail-safe — a message with **no** audio must relay unchanged
@ -92,7 +126,7 @@ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
--- ---
## 2. Telegram (optional) ## 3. Telegram (optional)
1. Create a bot: message [@BotFather](https://t.me/BotFather) → `/newbot` → copy the token. 1. Create a bot: message [@BotFather](https://t.me/BotFather) → `/newbot` → copy the token.
2. Put it in `/opt/vm-transcribe/telegram.conf` (`sudo`, mode 640 root:asterisk). 2. Put it in `/opt/vm-transcribe/telegram.conf` (`sudo`, mode 640 root:asterisk).
@ -111,95 +145,135 @@ per-mailbox routing.
--- ---
## 3. Contact lookup (optional) ## 4. Contacts (MySQL)
Simplest and most reliable — a local export, no tokens, no rate limits: Contacts live in the MySQL `contacts` table — there is no file/Google/CardDAV
backend anymore. Load your address book from a vCard/CSV export:
1. contacts.google.com → **Export** → **vCard** ```bash
2. `sudo install -o asterisk -g asterisk -m 640 contacts.vcf /var/lib/vm-transcribe/contacts.vcf` sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
3. Confirm `contacts.conf` has `backends = file` and the matching `path`. /opt/vm-transcribe/vm_import_contacts.py /path/to/contacts.vcf
```
Confirm lookup works:
```bash ```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
/opt/vm-transcribe/vm_contacts.py '<07700900123>' /opt/vm-transcribe/vm_contacts.py '<07700900123>'
``` ```
> **Google app passwords do not work for contacts.** Google disabled basic auth
> for CardDAV/CalDAV/IMAP/SMTP/POP on 2024-09-30. Use the `file` or `google`
> (OAuth) backend. The `carddav` backend is for Nextcloud/Fastmail/iCloud.
--- ---
## 4. Portal service ## 5. Backend services
Install both units (API + legacy portal/audio):
```bash ```bash
sudo install -d -o asterisk -g asterisk -m 750 \ sudo install -d -o asterisk -g asterisk -m 750 \
/var/lib/vm-transcribe /var/lib/vm-transcribe/audio /var/lib/vm-transcribe /var/lib/vm-transcribe/audio
sudo /opt/vm-transcribe/venv/bin/pip install fastapi 'uvicorn[standard]' python-multipart sudo /opt/vm-transcribe/venv/bin/pip install fastapi 'uvicorn[standard]' python-multipart pymysql
# API unit reads MySQL creds from api.env
sudo install -o root -g root -m 640 /opt/vm-transcribe/db_secret /opt/vm-transcribe/api.env
sudo install -o root -g root -m 644 systemd/vm-api.service /etc/systemd/system/
sudo install -o root -g root -m 644 systemd/vm-portal.service /etc/systemd/system/ sudo install -o root -g root -m 644 systemd/vm-portal.service /etc/systemd/system/
sudo systemctl daemon-reload sudo systemctl daemon-reload
sudo systemctl enable --now vm-portal sudo systemctl enable --now vm-api vm-portal
sudo systemctl is-active vm-portal sudo systemctl is-active vm-api vm-portal
curl -s http://127.0.0.1:8098/api/healthz # {"ok":true,...}
curl -s http://127.0.0.1:8099/healthz # {"ok":true,"messages":N} curl -s http://127.0.0.1:8099/healthz # {"ok":true,"messages":N}
``` ```
The service runs as `asterisk` (so it can read `voicemail.conf` and the spool) The services run as `asterisk` (so they can read `voicemail.conf` and the spool)
and binds **loopback only** — `IPAddressAllow=localhost` means Apache is the and bind **loopback only** — `IPAddressAllow=localhost` means Apache is the
only possible client. only possible client.
--- ---
## 5. Apache vhost + TLS ## 6. React frontend (build + deploy)
The interactive UI is a React SPA built in `/home/jp/Work/voicemail-ui`:
```bash
cd /home/jp/Work/voicemail-ui
npm install
npm run build # outputs dist/
```
Deploy the static build to the web root (Apache serves it and proxies `/api/`
to :8098 and `/audio/` to :8099):
```bash
sudo -A rsync -a --exclude node_modules --exclude dist \
/home/jp/Work/voicemail-ui/ /home/txt3/domains/vm.txt3.net/public_html/
```
Keep this repo's copy in sync for documentation/commit purposes:
```bash
rsync -a /home/jp/Work/voicemail-ui/ frontend/ # then git commit under frontend/
```
---
## 7. Apache vhost + TLS
**Order matters.** Installing a vhost that references a cert which does not yet **Order matters.** Installing a vhost that references a cert which does not yet
exist breaks `apache2ctl configtest`, which blocks reloads for *every* site on exist breaks `apache2ctl configtest`, which blocks reloads for *every* site on
the box. So: DNS → cert → full vhost. the box. So: DNS → cert → full vhost.
### 5a. DNS ### 7a. DNS
```bash ```bash
dig +short vm.example.com A # must return your ORIGIN ip dig +short vm.txt3.net A # must return your ORIGIN ip
``` ```
Behind Cloudflare, set the record to **DNS only (grey cloud)** first, or the Behind Cloudflare, set the record to **DNS only (grey cloud)** first, or the
HTTP-01 challenge is intercepted by the edge. HTTP-01 challenge is intercepted by the edge.
### 5b. Minimal :80 vhost, so certbot can answer the challenge ### 7b. Minimal :80 vhost, so certbot can answer the challenge
```bash ```bash
sudo install -d -o www-data -g www-data -m 755 /var/www/vm.example.com/public_html sudo install -d -o www-data -g www-data -m 755 /var/www/vm.txt3.net/public_html
sudo cp apache/vm.txt3.net-step1.conf /etc/apache2/sites-available/vm.example.com.conf sudo cp apache/vm.txt3.net-step1.conf /etc/apache2/sites-available/vm.txt3.net.conf
# edit ServerName / paths / IPs to match your host # edit ServerName / paths / IPs to match your host
sudo a2ensite vm.example.com sudo a2ensite vm.txt3.net
sudo apache2ctl configtest && sudo systemctl reload apache2 sudo apache2ctl configtest && sudo systemctl reload apache2
``` ```
### 5c. Issue the certificate ### 7c. Issue the certificate
```bash ```bash
sudo certbot certonly --webroot -w /var/www/vm.example.com/public_html \ sudo certbot certonly --webroot -w /var/www/vm.txt3.net/public_html \
-d vm.example.com --cert-name vm.example.com -d vm.txt3.net --cert-name vm.txt3.net
``` ```
### 5d. Full vhost — HTTPS + reverse proxy ### 7d. Full vhost — HTTPS + reverse proxy
```bash ```bash
sudo a2enmod proxy proxy_http headers rewrite ssl sudo a2enmod proxy proxy_http headers rewrite ssl
sudo cp apache/vm.txt3.net.conf /etc/apache2/sites-available/vm.example.com.conf sudo cp apache/vm.txt3.net.conf /etc/apache2/sites-available/vm.txt3.net.conf
# edit ServerName, cert paths, IPs # edit ServerName, cert paths, IPs, DocumentRoot
sudo install -d /var/www/vm.example.com/public_html/.well-known/acme-challenge sudo install -d /var/www/vm.txt3.net/public_html/.well-known/acme-challenge
sudo apache2ctl configtest && sudo systemctl reload apache2 sudo apache2ctl configtest && sudo systemctl reload apache2
``` ```
### 5e. Verify through the real hostname The shipped `apache/vm.txt3.net.conf` already proxies `/api/` → :8098,
`/audio/` → :8099, serves the React `dist` as `DocumentRoot`, and adds an SPA
fallback (`RewriteRule ^ /index.html`).
### 7e. Verify through the real hostname
Not localhost — the whole point is to exercise Apache, TLS and the proxy: Not localhost — the whole point is to exercise Apache, TLS and the proxy:
```bash ```bash
R="--resolve vm.example.com:443:YOUR.ORIGIN.IP" R="--resolve vm.txt3.net:443:ORIGIN.IP"
curl -s $R https://vm.example.com/healthz curl -s $R https://vm.txt3.net/api/healthz
curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.example.com/login curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.txt3.net/login
curl -s $R -D- -o /dev/null https://vm.example.com/login | grep -i 'strict-transport\|content-security' curl -s $R -D- -o /dev/null https://vm.txt3.net/login | grep -iE 'strict-transport|content-security|x-frame'
# SPA fallback
curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.txt3.net/some/route
``` ```
Behind a CDN, `curl https://host/` tests the *CDN*, not your origin. Always Behind a CDN, `curl https://host/` tests the *CDN*, not your origin. Always
@ -207,13 +281,14 @@ Behind a CDN, `curl https://host/` tests the *CDN*, not your origin. Always
--- ---
## 6. Backfill existing voicemails ## 8. Backfill existing voicemails
```bash ```bash
V=/opt/vm-transcribe/venv/bin/python3 V=/opt/vm-transcribe/venv/bin/python3
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --dry-run # preview sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --dry-run # preview
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --limit 5 # trial sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --limit 5 # trial
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py # all sudo -u asterisk $V /opt/vm-transcribe/vm_import.py # all
sudo -u asterisk $V /opt/vm-transcribe/vm_backfill_cdr.py # CDR history
``` ```
Idempotent — safe to re-run; it resumes rather than duplicating. Expect a Idempotent — safe to re-run; it resumes rather than duplicating. Expect a
@ -224,9 +299,9 @@ on 4 cores.
--- ---
## 7. Log in ## 9. Log in
Browse to `https://vm.example.com/` and log in with a **mailbox number and its Browse to `https://vm.txt3.net/` and log in with a **mailbox number and its
existing voicemail PIN** from `voicemail.conf`. No new passwords are created. existing voicemail PIN** from `voicemail.conf`. No new passwords are created.
--- ---
@ -234,11 +309,13 @@ existing voicemail PIN** from `voicemail.conf`. No new passwords are created.
## Uninstall ## Uninstall
```bash ```bash
sudo systemctl disable --now vm-portal sudo systemctl disable --now vm-api vm-portal
sudo rm /etc/systemd/system/vm-portal.service && sudo systemctl daemon-reload sudo rm /etc/systemd/system/vm-api.service /etc/systemd/system/vm-portal.service
sudo a2dissite vm.example.com && sudo systemctl reload apache2 sudo systemctl daemon-reload
sudo a2dissite vm.txt3.net && sudo systemctl reload apache2
# restore the original mailcmd # restore the original mailcmd
sudo cp /etc/asterisk/voicemail.conf.bak-<timestamp> /etc/asterisk/voicemail.conf sudo cp /etc/asterisk/voicemail.conf.bak-<timestamp> /etc/asterisk/voicemail.conf
sudo asterisk -rx 'voicemail reload' sudo asterisk -rx 'voicemail reload'
sudo rm -rf /opt/vm-transcribe # keep /var/lib/vm-transcribe for the data sudo rm -rf /opt/vm-transcribe # keep /var/lib/vm-transcribe for the data
# MySQL: DROP USER 'asterisk'@'localhost'; (optionally DROP DATABASE asterisk;)
``` ```

76
docs/OPERATIONS.md
View File

@ -4,19 +4,21 @@ Day-to-day running: backups, monitoring, common problems, upgrades.
--- ---
## Service status ## Services
```bash ```bash
sudo systemctl status vm-portal # the portal sudo systemctl status vm-api vm-portal # API (:8098) + legacy portal (:8099)
sudo journalctl -u vm-portal -n 50 # portal logs (uvicorn access/startup) sudo journalctl -u vm-api -n 50 # API logs (uvicorn access/startup)
sudo journalctl -u vm-portal -n 50 # legacy portal logs
sudo tail -f /var/log/asterisk/vm_mailcmd.log # per-voicemail pipeline log sudo tail -f /var/log/asterisk/vm_mailcmd.log # per-voicemail pipeline log
``` ```
Health of the portal (works locally or via the real hostname): Health checks:
```bash ```bash
curl -s http://127.0.0.1:8099/healthz # {"ok":true,"messages":N} curl -s http://127.0.0.1:8098/api/healthz # JSON API health
curl -s --resolve vm.txt3.net:443:ORIGIN_IP https://vm.txt3.net/healthz curl -s http://127.0.0.1:8099/healthz # legacy portal health
curl -s --resolve vm.txt3.net:443:ORIGIN_IP https://vm.txt3.net/api/healthz
``` ```
--- ---
@ -26,7 +28,7 @@ curl -s --resolve vm.txt3.net:443:ORIGIN_IP https://vm.txt3.net/healthz
| Signal | Where | Meaning | | Signal | Where | Meaning |
|---|---|---| |---|---|---|
| `transcribed …` | vm_mailcmd.log | a voicemail was processed OK | | `transcribed …` | vm_mailcmd.log | a voicemail was processed OK |
| `stored message for mailbox …` | vm_mailcmd.log | it also reached the portal DB | | `stored message for mailbox …` | vm_mailcmd.log | it also reached the MySQL store |
| `no audio attachment; relaying original` | vm_mailcmd.log | Asterisk sent a text-only notice (missed-call style) — expected | | `no audio attachment; relaying original` | vm_mailcmd.log | Asterisk sent a text-only notice (missed-call style) — expected |
| `Telegram disabled` / `no route` | vm_mailcmd.log | Telegram skipped (per config) — expected if unset | | `Telegram disabled` / `no route` | vm_mailcmd.log | Telegram skipped (per config) — expected if unset |
| `relaying original, mailcmd error` | vm_mailcmd.log | **pipeline threw**; original email was preserved (fail-safe worked) | | `relaying original, mailcmd error` | vm_mailcmd.log | **pipeline threw**; original email was preserved (fail-safe worked) |
@ -37,23 +39,29 @@ curl -s --resolve vm.txt3.net:443:ORIGIN_IP https://vm.txt3.net/healthz
## Backups ## Backups
Two things to back up — the code is reproducible, the *data* is not: Two things to back up — the code is reproducible, the *data* is not. The MySQL
`asterisk` database now holds `messages`, `contacts` and CDR, so back up the DB
(not a SQLite file):
```bash ```bash
# database + audio (once or twice a day is plenty) # MySQL dump (messages + contacts + cdr) — run as root
sudo -u asterisk tar czf /backup/vm-$(date +%F).tgz \ sudo mysqldump --single-transaction asterisk \
-C / var/lib/vm-transcribe/voicemail.db var/lib/vm-transcribe/audio > /backup/vm-mysql-$(date +%F).sql
# audio blobs (still on disk)
sudo -u asterisk tar czf /backup/vm-audio-$(date +%F).tgz -C / var/lib/vm-transcribe/audio
# config (the live, secret-bearing files) # config (the live, secret-bearing files)
sudo tar czf /backup/vm-conf-$(date +%F).tgz \ sudo tar czf /backup/vm-conf-$(date +%F).tgz \
/opt/vm-transcribe/telegram.conf /opt/vm-transcribe/contacts.conf /opt/vm-transcribe/telegram.conf /opt/vm-transcribe/contacts.conf \
/opt/vm-transcribe/db_secret /opt/vm-transcribe/api.env
``` ```
The whisper model cache (`/opt/vm-transcribe/models`) is reproducible — no need The whisper model cache (`/opt/vm-transcribe/models`) is reproducible — no need
to back it up, just re-run `install.sh`. to back it up, just re-run `install.sh`.
Restoring: stop the service, extract, `systemctl start vm-portal`. The DB schema Restoring: stop the services, load the dump into MySQL, extract audio to
is created on first connect, but **preserve the existing file** to keep history. `/var/lib/vm-transcribe/audio`, `systemctl start vm-api vm-portal`.
--- ---
@ -76,15 +84,25 @@ storage`. Check `df -h /var` first when the pipeline looks dead.
```bash ```bash
git -C /home/jp/Work/asterisk-voicemail pull git -C /home/jp/Work/asterisk-voicemail pull
cd /home/jp/Work/asterisk-voicemail cd /home/jp/Work/asterisk-voicemail
# backend
sudo cp src/*.py /opt/vm-transcribe/ sudo cp src/*.py /opt/vm-transcribe/
sudo /opt/vm-transcribe/venv/bin/pip install -U faster-whisper # occasionally sudo /opt/vm-transcribe/venv/bin/pip install -U faster-whisper # occasionally
sudo systemctl restart vm-portal sudo systemctl restart vm-api vm-portal
# frontend (build in the UI repo, rsync the dist)
cd /home/jp/Work/voicemail-ui && npm install && npm run build
sudo -A rsync -a --exclude node_modules --exclude dist \
/home/jp/Work/voicemail-ui/ /home/txt3/domains/vm.txt3.net/public_html/
``` ```
`install.sh` is idempotent-ish for the first install but is **not** a general `install.sh` is idempotent-ish for the first install but is **not** a general
upgrade tool — it backs up `voicemail.conf` each run, so avoid re-running it upgrade tool — it backs up `voicemail.conf` each run, so avoid re-running it
blindly. For upgrades, copy `src/*.py` as above. blindly. For upgrades, copy `src/*.py` as above.
> Note: the deploy paths in AGENTS.md use `/home/jp/asterisk-vm` and
> `/home/jp/Work/voicemail-ui` respectively — keep the two in sync.
--- ---
## Common problems ## Common problems
@ -96,20 +114,27 @@ blindly. For upgrades, copy `src/*.py` as above.
4. `sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 -c 'import faster_whisper'` — is the venv intact? 4. `sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 -c 'import faster_whisper'` — is the venv intact?
**Portal returns 502 / can't connect.** **Portal returns 502 / can't connect.**
The uvicorn backend is down or not on loopback: `sudo systemctl status vm-portal`; The API backend is down or not on loopback: `sudo systemctl status vm-api`;
`ss -ltnp | grep 8099`. Also confirm Apache has `proxy`/`proxy_http` enabled. `ss -ltnp | grep 8098`. Also confirm Apache has `proxy`/`proxy_http` enabled and
that `/api/` is proxied to `127.0.0.1:8098`. (Legacy `/audio/` 502 → check
`vm-portal` on :8099.)
**Login fails for a real mailbox.** **Login fails for a real mailbox.**
The PIN in `voicemail.conf` is what the portal checks — not anything in the The PIN in `voicemail.conf` is what the API checks — not anything in the DB. If
portal. If you changed a voicemail PIN, that change is picked up immediately you changed a voicemail PIN, that change is picked up immediately (the file is
(the file is reparsed each login). If the mailbox line is commented out or in a reparsed each login). If the mailbox line is commented out or in a context the
context the parser doesn't reach, login fails. Test the parser directly: parser doesn't reach, login fails. Test the parser directly:
```bash ```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
-c "import sys; sys.path.insert(0,'/opt/vm-transcribe'); import vm_auth; print(vm_auth.check_login('7940','5159'))" -c "import sys; sys.path.insert(0,'/opt/vm-transcribe'); import vm_auth; print(vm_auth.check_login('7940','5159'))"
``` ```
**MySQL connection errors in the API.**
Check `/opt/vm-transcribe/db_secret` (or `api.env`) and that the `asterisk` DB
user exists with access to the `asterisk` database. The API logs a clear
`vm_store unavailable` message if MySQL is unreachable.
**Certbot renewal.** **Certbot renewal.**
Renewal runs from `/etc/cron.d/certbot`; `certbot.timer` is masked (normal on Renewal runs from `/etc/cron.d/certbot`; `certbot.timer` is masked (normal on
Debian's package). Test non-destructively in the background (it can be slow): Debian's package). Test non-destructively in the background (it can be slow):
@ -119,7 +144,7 @@ sudo certbot renew --cert-name vm.txt3.net --dry-run &
``` ```
**I locked myself out testing lockout.** **I locked myself out testing lockout.**
Restart the service: `sudo systemctl restart vm-portal`. In-memory counters clear. Restart the service: `sudo systemctl restart vm-api`. In-memory counters clear.
**Backfill shows many `no_speech`.** **Backfill shows many `no_speech`.**
Expected. 44-byte WAVs (`duration=0`) are hung-up calls with no audio. Not a bug. Expected. 44-byte WAVs (`duration=0`) are hung-up calls with no audio. Not a bug.
@ -135,4 +160,9 @@ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_impor
``` ```
Useful after importing a large batch of new spool messages, or after a fresh Useful after importing a large batch of new spool messages, or after a fresh
install on a box with existing voicemails. install on a box with existing voicemails. To also load CDR history from
`Master.csv`:
```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_backfill_cdr.py
```

97
docs/SECURITY.md
View File

@ -13,8 +13,8 @@ their messages — often personal or commercial. The portal also proves who
1. An attacker reads someone else's voicemail (broken authz or IDOR). 1. An attacker reads someone else's voicemail (broken authz or IDOR).
2. An attacker brute-forces a 4-digit PIN at scale. 2. An attacker brute-forces a 4-digit PIN at scale.
3. A stored credential (Telegram token, contacts app password) leaks. 3. A stored credential (MySQL password, Telegram token, contacts export) leaks.
4. XSS / injection poisons the HTML shown to a user. 4. XSS / injection poisons the HTML/JSON shown to a user.
5. The recording audio leaks off-box. 5. The recording audio leaks off-box.
--- ---
@ -23,43 +23,50 @@ their messages — often personal or commercial. The portal also proves who
**Authentication — mailbox PIN from voicemail.conf.** **Authentication — mailbox PIN from voicemail.conf.**
No new password store to breach; the portal trusts Asterisk's PINs. Sessions are No new password store to breach; the portal trusts Asterisk's PINs. Sessions are
DB-backed (random token, `Secure`+`HttpOnly`+`SameSite=lax`, 12 h). Logout and MySQL-backed (random token, `Secure`+`HttpOnly`+`SameSite=lax`, 12 h). Logout
expiry are enforced server-side, not merely by dropping the cookie. and expiry are enforced server-side, not merely by dropping the cookie.
**Authorization — every query is mailbox-scoped.** **Authorization — every query is mailbox-scoped.**
`vm_store` and `vm_web` filter every read and delete by the session's mailbox. `vm_store` and `vm_api` filter every read and delete by the session's mailbox.
Cross-mailbox reads/deletes return **404**. There is a test that asserts this Cross-mailbox reads/deletes return **404**. There is a test that asserts this
([TESTING.md](TESTING.md)). A mailbox owner can only ever see their own messages. ([TESTING.md](TESTING.md)). A mailbox owner can only ever see their own messages
(and their own contacts).
**Brute-force lockout.** **Brute-force lockout.**
5 failed attempts per `(mailbox, IP)` → 15-minute lockout. The correct PIN is 5 failed attempts per `(mailbox, IP)` → 15-minute lockout. The correct PIN is
also refused while locked. State is in-memory (single worker); a restart clears also refused while locked. State is in-memory (single worker per service); a
it. restart clears it.
**Transport — TLS terminated at Apache, with headers.** **Transport — TLS terminated at Apache, with headers.**
`Strict-Transport-Security`, `X-Frame-Options: DENY`, `X-Content-Type-Options: `Strict-Transport-Security`, `X-Frame-Options: DENY`,
nosniff`, `Referrer-Policy`, and a tight CSP: `X-Content-Type-Options: nosniff`, `Referrer-Policy`, and a CSP:
``` ```text
default-src 'self'; style-src 'self' 'unsafe-inline'; default-src 'self'; style-src 'self' 'unsafe-inline';
media-src 'self'; img-src 'self' data:; img-src 'self' data:; font-src 'self';
script-src 'none'; frame-ancestors 'none'; media-src 'self'; connect-src 'self' http://127.0.0.1:8098;
base-uri 'none'; form-action 'self' script-src 'self' 'unsafe-inline'; frame-ancestors 'none'; base-uri 'none'
``` ```
**No JavaScript in the portal at all**, which is what lets `script-src 'none'` **The portal now ships JavaScript** (a React SPA), so the CSP is *relaxed* from
be genuinely enforceable — there is nothing to inject. the original `script-src 'none'` to `script-src 'self' 'unsafe-inline'`. Inline
scripts in the built bundle are still same-origin; `base-uri 'none'` and
`frame-ancestors 'none'` remain to blunt injection / framing. If you can build
the frontend without inline scripts, tighten this back to `'self'` only.
**Network exposure is minimal.** **Network exposure is minimal.**
The uvicorn backend binds **loopback only** (`IPAddressAllow=localhost` in the Both uvicorn backends bind **loopback only**
unit). The only ingress is Apache. The systemd unit also sets (`IPAddressAllow=127.0.0.1` / `localhost` in the units). The only ingress is
`NoNewPrivileges`, `ProtectSystem=full`, `ProtectHome`, `PrivateTmp`, Apache. The `vm-api` unit also sets `ProtectSystem=full`, `ProtectHome=read-only`,
`ProtectKernelTunables`, `ProtectControlGroups`, `RestrictSUIDSGID`, and writable `NoNewPrivileges`, `PrivateTmp`; `vm-portal` adds `ProtectKernelTunables`,
paths are limited to `/var/lib/vm-transcribe` and `/var/log/asterisk`. `ProtectControlGroups`, `RestrictSUIDSGID`, `IPAddressDeny=any`, and writable
paths limited to `/var/lib/vm-transcribe` and `/var/log/asterisk`.
**Secrets are separated and git-ignored.** **Secrets are separated and git-ignored.**
`telegram.conf`, `contacts.conf`, `*.db`, `audio/` are in `.gitignore`. The `telegram.conf`, `contacts.conf`, `db_secret`, `api.env`, `*.db`, `audio/` are
repo ships `*.example` templates only. Both secret files are mode `640 root:asterisk`. in `.gitignore`. The repo ships `*.example` templates only. `db_secret` and
`telegram.conf` are mode `640 root:asterisk`; `api.env` is mode `640`
root:root.
**Fail-safe preserves mail.** **Fail-safe preserves mail.**
A pipeline exception relays the original Asterisk message unchanged — we never A pipeline exception relays the original Asterisk message unchanged — we never
@ -78,22 +85,22 @@ access log. Recommended.
Acceptable (same host, no network path), but if you ever run uvicorn on another Acceptable (same host, no network path), but if you ever run uvicorn on another
host, use a unix socket or mTLS. host, use a unix socket or mTLS.
**(R3) Session token in SQLite.** If the DB file is stolen, sessions are **(R3) Session token in MySQL.** If the DB is stolen, sessions are replayable
replayable until they expire. DB is `640 asterisk:asterisk` and not web-served. until they expire. The `asterisk` DB is not web-served and the `asterisk` user is
For higher assurance, store sessions in a server-side cache with shorter TTLs. least-privilege; keep `db_secret` at `640 root:root`. For higher assurance,
shorten `VM_SESSION_HOURS` or store sessions in a server-side cache with short
TTLs.
**(R4) Contact resolver tokens.** A Google OAuth token or a CardDAV app password **(R4) Contact data.** Contacts now live in the MySQL `contacts` table (no live
grants read access to your address book. Scope the Google token to Google/CardDAV token at all — the old OAuth/app-password backends were removed).
`contacts.readonly`, use a dedicated app password (never your login password), The only secret around contacts is the DB password. Prefer the MySQL backend
and keep these files `640`. Prefer the `file` backend (a periodic vCard export) — (which we are already on) — there is no external address-book token to leak.
no live token at all.
**(R5) Telegram bot token.** Whoever holds it can post as your bot. Keep **(R5) Telegram bot token.** Whoever holds it can post as your bot. Keep
`telegram.conf` `640 root:asterisk`; rotate via @BotFather if leaked. `telegram.conf` `640 root:asterisk`; rotate via @BotFather if leaked.
**(R6) Content-addressed audio filenames.** The sha256 of the audio is in the URL **(R6) Content-addressed audio filenames.** The sha256 of the audio is used to
(`/audio/<sha>`). An attacker who guesses a sha could fetch that recording locate the blob. The portal checks the session's mailbox *owns* that message
without a session. The portal checks the session's mailbox *owns* that message
before serving, so this is not directly exploitable — but consider a random before serving, so this is not directly exploitable — but consider a random
per-message token instead of the content hash if you want defence in depth. per-message token instead of the content hash if you want defence in depth.
@ -101,17 +108,17 @@ per-message token instead of the content hash if you want defence in depth.
audio streams through the edge. Grey-cloud it (DNS only) to keep voice data off audio streams through the edge. Grey-cloud it (DNS only) to keep voice data off
the CDN. Your call. the CDN. Your call.
**(R8) who-called.co.uk lookups.** The *Lookup number* action opens a new browser
tab to who-called.co.uk using the caller's sanitised digits. This is a
user-initiated click from the UI — no data is sent server-side. It does mean the
user's browser (and the caller's number) reach a third-party site; acceptable for
a manual reverse-lookup, but worth knowing.
--- ---
## Obtaining a Google token (if you use the `google` backend) ## Past Google-contacts note (no longer applicable)
1. Google Cloud console → OAuth consent screen (External) → add your account as The old `file` / `google` / `carddav` contact backends were removed in favour of
a test user. the MySQL `contacts` table. Historically, Google disabled basic auth for
2. Credentials → OAuth client ID → **Desktop app**. CardDAV/CalDAV/IMAP on **2024-09-30**, so an app-password read of Google
3. Scope `https://www.googleapis.com/auth/contacts.readonly`. Contacts never worked — that is moot now that contacts are a local MySQL table.
4. Authorize once (the token is written to `google-token.json`, mode `640`).
The People API is used, not CardDAV, because basic auth / app passwords were
disabled by Google on **2024-09-30**.
There is no support for an app-password read of Google Contacts — it does not
work. Use the `file` or `google` backend.

71
docs/TESTING.md
View File

@ -1,7 +1,7 @@
# Testing # Testing
How to verify each part of the system. Do the mailcmd tests **as the `asterisk` How to verify each part of the system. Do the mailcmd tests **as the `asterisk`**
user**, not as yourself — that's where permission bugs hide. user, not as yourself — that's where permission bugs hide.
--- ---
@ -10,7 +10,8 @@ user**, not as yourself — that's where permission bugs hide.
```bash ```bash
df -h /var # must not be full (Postfix 452) df -h /var # must not be full (Postfix 452)
grep '^mailcmd' /etc/asterisk/voicemail.conf grep '^mailcmd' /etc/asterisk/voicemail.conf
sudo systemctl is-active vm-portal sudo systemctl is-active vm-api vm-portal
curl -s http://127.0.0.1:8098/api/healthz
curl -s http://127.0.0.1:8099/healthz curl -s http://127.0.0.1:8099/healthz
``` ```
@ -29,7 +30,7 @@ group id negative; unmapped → default; `enabled=no` → no route; caption clip
to 1017 chars on a word boundary when over 1024; a 374 KB wav → 64 KB `OggS` to 1017 chars on a word boundary when over 1024; a 374 KB wav → 64 KB `OggS`
opus. opus.
`test_contacts.py` asserts: `+447700900123`, `07700900123`, `447700900123` all `test_contacts.py` asserts: `+447****0123`, `07700900123`, `447700900123` all
match one contact (last-9-digit key); multi-TEL cards; Google CSV `:::` split. match one contact (last-9-digit key); multi-TEL cards; Google CSV `:::` split.
--- ---
@ -64,8 +65,9 @@ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
### Fail-safe: a thrown error relays the original ### Fail-safe: a thrown error relays the original
Temporarily point `VM_DB` at an unreadable path; the script should fall back to Temporarily point `VM_DB` at an unreadable path (or MySQL offline); the script
relaying the original Asterisk message and log `relaying original, mailcmd error`. should fall back to relaying the original Asterisk message and log
`relaying original, mailcmd error`.
--- ---
@ -83,47 +85,49 @@ need a token to verify the config parser.
--- ---
## 4. Contacts ## 4. Contacts (MySQL)
```bash ```bash
# resolve a caller ID against the MySQL contacts table
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
/opt/vm-transcribe/vm_contacts.py '<07700900123>' /opt/vm-transcribe/vm_contacts.py '<07700900123>'
# resolves against the configured backend; prints name or the raw caller id # prints (name, email) or the raw caller id
# load an export
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
/opt/vm-transcribe/vm_import_contacts.py /path/to/contacts.vcf
``` ```
--- ---
## 5. Portal — auth, authz, playback, delete ## 5. JSON API — auth, authz, playback, delete
Run the app locally (or against the live service) and exercise it with curl. Run the API locally (or against the live service) and exercise it with curl.
Use `VM_INSECURE_COOKIE=1` only for plain-HTTP local tests so the `Secure` Use `VM_INSECURE_COOKIE=1` only for plain-HTTP local tests so the `Secure`
cookie can be set. cookie can be set.
```bash ```bash
B=http://127.0.0.1:8098
# login ok # login ok
curl -s -c /tmp/j -o /dev/null -w '%{redirect_url}\n' \ curl -s -c /tmp/j -o /dev/null -w '%{redirect_url}\n' -d 'mailbox=7940&pin=5159' $B/api/login
-d 'mailbox=7940&pin=5159' http://127.0.0.1:8099/login
# wrong pin rejected # wrong pin rejected
curl -s -o /dev/null -w '%{http_code}\n' -d 'mailbox=7940&pin=1111' http://127.0.0.1:8099/login curl -s -o /dev/null -w '%{http_code}\n' -d 'mailbox=7940&pin=1111' $B/api/login
# authenticated list # authenticated message list
curl -s -b /tmp/j http://127.0.0.1:8099/ | grep -c 'class="card' curl -s -b /tmp/j $B/api/messages | head -c 400
# audio streams # contacts CRUD
curl -s -b /tmp/j -o /tmp/x.wav http://127.0.0.1:8099/audio/1 curl -s -b /tmp/j $B/api/contacts | head -c 400
file /tmp/x.wav # RIFF WAVE, 8000 Hz mono
# delete
curl -s -b /tmp/j -X POST -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8099/delete/1
``` ```
### Authorization (IDOR) — must 404 ### Authorization (IDOR) — must 404
A session for mailbox A must never reach mailbox B's data. With a cookie for A session for mailbox A must never reach mailbox B's data. With a cookie for
mailbox 7940, hitting `/audio/<id-owned-by-1001>` and `/delete/<id-owned-by-1001>` mailbox 7940, hitting `/api/messages/{id-owned-by-1001}` (or its audio/delete
must both return **404/303**, never serve or delete the other mailbox's message. variants) must return **404**, never serve or delete the other mailbox's message.
Add this assertion whenever you change `vm_store` or `vm_web`. Add this assertion whenever you change `vm_store` or `vm_api`.
--- ---
@ -131,13 +135,20 @@ Add this assertion whenever you change `vm_store` or `vm_web`.
```bash ```bash
R="--resolve vm.txt3.net:443:ORIGIN.IP" R="--resolve vm.txt3.net:443:ORIGIN.IP"
curl -s $R https://vm.txt3.net/healthz curl -s $R https://vm.txt3.net/api/healthz
curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.txt3.net/login curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.txt3.net/login
curl -s $R -D- -o /dev/null https://vm.txt3.net/login | grep -iE 'strict-transport|content-security|x-frame' curl -s $R -D- -o /dev/null https://vm.txt3.net/login | grep -iE 'strict-transport|content-security|x-frame'
# http -> https redirect # http -> https redirect
curl -s $R -o /dev/null -w '%{redirect_url}\n' http://vm.txt3.net/ curl -s $R -o /dev/null -w '%{redirect_url}\n' http://vm.txt3.net/
``` ```
The React SPA is served as static files; all data flows through `/api/`. Confirm
the SPA fallback returns `index.html` for arbitrary client-side routes:
```bash
curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.txt3.net/some/client/route
```
Behind a CDN, `--resolve` to the **origin** IP; otherwise you are testing the CDN, Behind a CDN, `--resolve` to the **origin** IP; otherwise you are testing the CDN,
not your server. not your server.
@ -155,5 +166,11 @@ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
# real: 3 imported, others skipped; re-running is a no-op (idempotent) # real: 3 imported, others skipped; re-running is a no-op (idempotent)
``` ```
Verify they appear in the portal and are playable (§5). Expect a `no_speech` Verify they appear via the API/messages endpoint and are playable (§5). Expect a
count — 44-byte WAVs are hung-up calls, not failures. `no_speech` count — 44-byte WAVs are hung-up calls, not failures.
CDR history import:
```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_backfill_cdr.py
```