diff --git a/CHANGELOG.md b/CHANGELOG.md index 66262c0..db8fbfc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,27 @@ # Changelog +## 1.2.0 — 2026-08-13 + +- **React SPA frontend + JSON API.** Added `vm_api.py` (FastAPI JSON API on + :8098) and a React frontend (`/home/jp/Work/voicemail-ui`, served as static + files with an SPA fallback by Apache). The portal UI is now a JavaScript SPA + (react-router-dom + TanStack Query + Vite), not the zero-JS server-rendered + app. `vm_web.py` (:8099) remains to serve `/audio/` playback and `/healthz`. +- **Apache vhost updated** to proxy `/api/` → :8098, `/audio/` → :8099, serve + the React `dist`, and SPA-fallback to `index.html`. CSP relaxed to + `script-src 'self' 'unsafe-inline'` (was `script-src 'none'`) because the UI + is now JavaScript. +- **Per-message / per-contact dropdown menus** in the UI (⋮): **Call back** + opens a `tel:` link from the sanitised caller digits; **Lookup number** opens + a who-called.co.uk reverse-lookup in a new tab, using the **0-prefixed** UK + number (swap `+44` → `0` before lookup, search by sanitised digits only). + Applied to both the Messages and Contacts pages. +- **Contacts backfill on write.** Creating/updating a contact via the API now + updates matching `messages.contact_name`/`contact_email` (spaces/quotes + stripped before the LIKE; matches both E.164 and 0-prefix variants). +- Documentation (`README.md`, `docs/*`) brought in line with the MySQL + + React + `vm_api.py` architecture. + ## 1.1.0 — 2026-08-13 - **Storage migrated to MySQL**: `vm_store` now defaults to MySQL (PyMySQL) @@ -17,6 +39,9 @@ cannot shadow real names. - **Migrations**: `vm_migrate_sqlite_to_mysql.py` and `vm_backfill_cdr.py` added under `src/` for first-run and CDR backfill. +- **Contacts backend consolidated to MySQL only.** The `file` / `google` / + `carddav` resolver backends were removed; `vm_contacts` now resolves + exclusively from the MySQL `contacts` table. ## 1.0.0 — 2026-08-13 @@ -31,17 +56,12 @@ Initial build on mail.txt3.net (Debian 12, Asterisk 20, Apache 2.4, Postfix). any error (fail-safe). - **Telegram** (optional): per-mailbox routed voice-note DM with summary caption and transcript follow-up; degrades sendVoice → sendDocument → sendMessage. -- **Contacts** (optional): caller-ID → name via `file` (vCard/CSV export), - `google` (People API OAuth), or `carddav` (app password; Nextcloud/Fastmail/ - iCloud). Last-9-digit matching. Note: Google app passwords do not work. - **Web portal**: FastAPI app at `https://vm.txt3.net` — PIN login (mailbox + voicemail PIN from `voicemail.conf`), list, play, download, delete, per-user - settings. Zero JavaScript, `script-src 'none'` CSP, loopback-only backend - behind an Apache TLS reverse proxy. Brute-force lockout per (mailbox, IP). -- **Storage**: SQLite store with content-addressed audio (decoupled from - Asterisk's renumbering) and DB-backed sessions. + settings. DB-backed sessions. Brute-force lockout per (mailbox, IP). +- **Storage**: content-addressed audio (decoupled from Asterisk's renumbering). - **Backfill**: `vm_import.py` imports and transcribes existing spool recordings; idempotent on `(mailbox, origtime, callerid)`. Backfilled 157 historical messages on first run (3m40s, 0 failures). -- **Packaging**: venv at `/opt/vm-transcribe`, systemd `vm-portal.service`, - Apache vhost + certbot TLS, `install.sh`. +- **Packaging**: venv at `/opt/vm-transcribe`, systemd units, Apache vhost + + certbot TLS, `install.sh`. diff --git a/README.md b/README.md index 1b748e6..966239e 100644 --- a/README.md +++ b/README.md @@ -9,60 +9,53 @@ manage: - **Emails** a graphically designed `multipart/alternative` notification (plain + HTML) with the recording attached - **DMs Telegram** optionally, as a playable voice note with the summary -- **Resolves caller ID to a name** from Google Contacts or any CardDAV server -- **Serves a web portal** where mailbox users log in with their existing phone - PIN to read transcripts, play/download recordings, delete messages and - manage their own notification settings +- **Resolves caller ID to a name** from the MySQL `contacts` table +- **Serves a React web portal** where mailbox users log in with their existing + phone PIN to read transcripts, play/download recordings, delete messages, + manage contacts and look up unknown numbers Everything runs on the PBX host. No third-party service sees your voicemail. --- -## What it looks like - -**Email notification** — styled HTML card with metadata, summary panel, intent -chips, a `tel:` callback link, full transcript and the audio attached. A plain -text alternative is always included. - -**Telegram** — voice note (ogg/opus) with the summary as its caption, intent -hashtags, and the transcript as a follow-up message. - -**Portal** — one card per voicemail: caller, timestamp, duration, summary, -tags, callback link, inline player, collapsible transcript, and Mark read / -Download / Delete. - ---- - -## Architecture +## Architecture (current) ``` - incoming call - │ - ▼ + incoming call + │ + ▼ ┌─────────────┐ voicemail.conf: mailcmd=… vm_mailcmd.py │ Asterisk │ ─────────────────────────────┐ - └─────────────┘ pipes an RFC822 message │ - (notification + audio) ▼ + └─────────────┘ pipes an RFC822 message │ + (notification + audio) ▼ ┌───────────────┐ │ vm_mailcmd.py │ └───────┬───────┘ ┌──────────────┬──────────────┬────┴─────────┬──────────────┐ ▼ ▼ ▼ ▼ ▼ faster-whisper summarise() vm_contacts vm_store vm_telegram - (transcribe) + intents (name lookup) (SQLite + (voice note + (transcribe) + intents (MySQL name) (MySQL + (voice note + numbers audio CAS) DM) │ │ ▼ ▼ - multipart email ──▶ Postfix :25 ┌──────────────┐ - │ vm_web.py │ - │ (FastAPI) │ - └──────┬───────┘ - │ :8099 loopback - ▼ - Apache (TLS) - │ - ▼ - https://vm.txt3.net + multipart email ──▶ Postfix :25 ┌──────────────────────────┐ + │ MySQL (asterisk DB) │ + │ messages, contacts, CDR │ + └──────┬───────────┬─────────┘ + │ │ + ┌──────────────────────┘ └──────────────────────┐ + ▼ ▼ + ┌──────────────────┐ ┌──────────────────┐ + │ vm_api.py :8098 │ JSON API (React SPA) │ vm_web.py :8099 │ + │ (FastAPI) │ ◀── React frontend ──┐ │ (FastAPI) │ + └────────┬─────────┘ │ │ /audio/ legacy │ + │ Apache /api/ proxy │ └────────┬─────────┘ + ▼ │ │ Apache /audio/ proxy + Apache (TLS) ◀─────────────────────────┘ ▼ + │ Apache (TLS) + ▼ + https://vm.txt3.net + (React SPA + JSON API; audio proxied to :8099) ``` **Delivery order is deliberate**: email first, then the database, then Telegram. @@ -70,6 +63,13 @@ Each later stage is wrapped so a failure only logs. If anything throws at the top level, the *original* Asterisk notification is relayed unchanged. A voicemail notification is never lost because a summariser or an API failed. +**Two backends, one MySQL store.** The *interactive* portal is a React single +page app (built from `/home/jp/Work/voicemail-ui`, served as static files by +Apache) that talks to a JSON API (`vm_api.py` on :8098). The original +server-rendered FastAPI app (`vm_web.py` on :8099) still runs but now only +serves `/audio/` playback (proxied by Apache). Both backends read/write the +same MySQL `asterisk` database. + --- ## Components @@ -77,13 +77,18 @@ voicemail notification is never lost because a summariser or an API failed. | File | Role | |---|---| | `src/vm_mailcmd.py` | The `mailcmd` — entry point for every voicemail. Orchestrates everything. | -| `src/vm_store.py` | SQLite store: schema, settings, sessions, content-addressed audio. | -| `src/vm_telegram.py` | Telegram delivery, per-mailbox routing. | -| `src/vm_contacts.py` | Caller-ID → name via local export, Google People API, or CardDAV. | -| `src/vm_web.py` | FastAPI portal: login, list, play, delete, settings. | +| `src/vm_store.py` | Store. **MySQL** (PyMySQL) by default; SQLite fallback only if no `db_secret`/MySQL env. | +| `src/vm_api.py` | FastAPI **JSON API** (:8098) for the React portal. Login, messages, settings, contacts CRUD. | +| `src/vm_web.py` | FastAPI HTML portal (:8099). Now primarily serves `/audio/` playback + `/healthz`. | +| `src/vm_contacts.py` | Caller-ID → name/email **from the MySQL `contacts` table** (MySQL-only). | | `src/vm_auth.py` | Parses `voicemail.conf` so users log in with their phone PIN. | +| `src/vm_telegram.py` | Telegram delivery, per-mailbox routing. | | `src/vm_import.py` | Backfills existing spool recordings into the database. | +| `src/vm_backfill_cdr.py` | Imports Asterisk `Master.csv` CDR history into MySQL. | +| `src/vm_migrate_sqlite_to_mysql.py` | One-off migration of the legacy SQLite store to MySQL. | +| `src/vm_import_contacts.py` | Imports a vCard/CSV export into the MySQL `contacts` table. | | `src/vm_tg_setup.py` | Helper: discover Telegram chat IDs, test a route. | +| `frontend/` | Synced copy of the React frontend (`/home/jp/Work/voicemail-ui`). | --- @@ -92,19 +97,28 @@ voicemail notification is never lost because a summariser or an API failed. See **[docs/INSTALL.md](docs/INSTALL.md)** for the full walkthrough. Short version: ```bash +# 1. Backend services (API + legacy audio server) sudo scripts/install.sh # venv, deps, model cache, mailcmd wiring +sudo cp systemd/vm-api.service /etc/systemd/system/ sudo cp systemd/vm-portal.service /etc/systemd/system/ -sudo systemctl enable --now vm-portal -# then follow docs/INSTALL.md §5 for the Apache vhost + TLS +sudo systemctl daemon-reload +sudo systemctl enable --now vm-api vm-portal + +# 2. React frontend (build in /home/jp/Work/voicemail-ui, rsync here) +cd /home/jp/Work/voicemail-ui && npm install && npm run build +sudo -A rsync -a --exclude node_modules --exclude dist \ + /home/jp/Work/voicemail-ui/ /home/txt3/domains/vm.txt3.net/public_html/ + +# 3. Apache vhost + TLS — see docs/INSTALL.md §5 ``` -Backfill your existing voicemails: +Backfill your existing voicemails and CDR: ```bash -sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ - /opt/vm-transcribe/vm_import.py --dry-run # preview -sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ - /opt/vm-transcribe/vm_import.py # do it +V=/opt/vm-transcribe/venv/bin/python3 +sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --dry-run # preview +sudo -u asterisk $V /opt/vm-transcribe/vm_import.py # do it +sudo -u asterisk $V /opt/vm-transcribe/vm_backfill_cdr.py # CDR history ``` --- @@ -113,10 +127,10 @@ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ | Document | Contents | |---|---| -| [docs/INSTALL.md](docs/INSTALL.md) | Step-by-step install, including TLS ordering | -| [docs/CONFIGURATION.md](docs/CONFIGURATION.md) | Every config file and option | +| [docs/INSTALL.md](docs/INSTALL.md) | Step-by-step install, MySQL setup, TLS ordering, frontend build | +| [docs/CONFIGURATION.md](docs/CONFIGURATION.md) | MySQL credentials, env vars, every config file and option | | [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Design decisions and why | -| [docs/OPERATIONS.md](docs/OPERATIONS.md) | Day-to-day running, backup, troubleshooting | +| [docs/OPERATIONS.md](docs/OPERATIONS.md) | Day-to-day running, backup (incl. MySQL), troubleshooting | | [docs/SECURITY.md](docs/SECURITY.md) | Threat model, hardening, privacy | | [docs/TESTING.md](docs/TESTING.md) | How to verify each part | | [CHANGELOG.md](CHANGELOG.md) | Version history | @@ -126,11 +140,13 @@ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ ## Requirements - Asterisk with `app_voicemail` (file-based spool, not ODBC/IMAP storage) -- Python 3.9+ +- Python 3.9+ with `pymysql` (backend now MySQL-first) +- MySQL/MariaDB (the same `asterisk` DB Asterisk's CDR uses) - `ffmpeg` (Telegram voice-note transcoding; `sox` optionally for gsm) - An MTA listening on `localhost:25` (Postfix here) - ~200 MB disk for the whisper `base.en` model - Apache with `proxy`, `proxy_http`, `headers`, `rewrite`, `ssl` for the portal +- Node 18+ + npm for the React frontend build Runs comfortably on 4 CPU cores with no GPU: ~8 s to transcribe 23 s of audio. @@ -148,12 +164,16 @@ keeps voicemail content on your own hardware. `summarise()` in pointing at the wrong recording. Recordings are copied to `audio//.wav`. -**App passwords cannot read Google Contacts.** Google disabled basic auth for -CardDAV/CalDAV/IMAP/SMTP/POP on 2024-09-30. Use the `file` backend (a vCard -export) or the `google` backend (OAuth). The `carddav` backend with an app -password works for Nextcloud, Fastmail and iCloud. +**Contacts live in MySQL.** `vm_contacts.py` resolves caller IDs exclusively +from the `asterisk.contacts` table (E.164 canonical; `+44`↔`0` normalised so +both forms match). Creating/updating a contact backfills matching +`messages.contact_name`/`contact_email`. The older `file` / `google` / `carddav` +backends were removed in favour of the MySQL table. -**The portal ships zero JavaScript**, which lets its CSP be `script-src 'none'`. +**The portal ships JavaScript now.** The interactive UI is a React SPA, so the +Apache CSP is relaxed to `script-src 'self' 'unsafe-inline'` (vs the old +zero-JS `script-src 'none'`). See [SECURITY.md](docs/SECURITY.md) for the +updated threat model. --- diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 9554566..95b5813 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,7 +1,8 @@ # Architecture How the pieces fit and *why* they are shaped the way they are. Read this -before changing anything in `vm_mailcmd.py` or `vm_store.py`. +before changing anything in `vm_mailcmd.py`, `vm_store.py`, `vm_api.py` or +`vm_contacts.py`. --- @@ -13,12 +14,12 @@ Asterisk ──(RFC822 on stdin)──▶ vm_mailcmd.py ┌──────────┬───────────────┼───────────────┬───────────────┐ ▼ ▼ ▼ ▼ ▼ transcribe summarise resolve store telegram - (whisper) +intents contact (sqlite) (DM) + (whisper) +intents contact (MySQL) (DM) │ │ │ │ │ └────▶ build multipart email ─▶ Postfix :25 (plain + HTML + audio attached) │ - └──▶ vm_store.add_message() (best-effort) + └──▶ vm_store.add_message() (best-effort, MySQL) └──▶ vm_telegram.send() (best-effort) ``` @@ -34,6 +35,58 @@ silently swallow a voicemail. --- +## Storage: MySQL, with a SQLite escape hatch + +`vm_store.py` is backend-agnostic. It defaults to **MySQL (PyMySQL)** when a +credentials source is present (`/opt/vm-transcribe/db_secret`, mode `640 +root:root`, or `VM_MYSQL_*` env vars). If neither exists it falls back to the +original SQLite file (`VM_DB`). + +The same `asterisk` MySQL database is shared with Asterisk's own CDR +(`cdr_adaptive_odbc` + `res_odbc` + the MariaDB ODBC driver), so voicemail +metadata and call records live side by side. This is why the migration kept the +`asterisk` DB and a least-privilege `asterisk` user rather than spinning up a +separate instance. + +The connection object returned by `connect()` mimics the `sqlite3` cursor surface +callers use (`.execute()`/`.executemany()`, `.commit()`, `.close()`, +`.fetchone()`, `.fetchall()`, `lastrowid`, and `row['col']` access), so the rest +of the code is written once and works against either backend. + +--- + +## Two backends in front of one store + +- **`vm_api.py` (FastAPI, :8098)** — the JSON API consumed by the React SPA. + Login/logout, message list, mark-read, delete, settings, and full contacts + CRUD (`GET/POST /api/contacts`, `GET/PUT/DELETE /api/contacts/{id}`, + `/api/contacts/import_vcf`, `/api/contacts/history/{number}`, + `/api/add_contact`, `/api/contacts/delete_all`). Apache proxies `/api/` here. +- **`vm_web.py` (FastAPI, :8099)** — the original server-rendered HTML portal. + It still runs and now primarily serves `/audio/` playback and `/healthz`; + Apache proxies `/audio/` here. The interactive UI has moved to the React SPA. + +Both read and write the same MySQL `messages` / `contacts` tables, so the React +UI and the legacy endpoints never disagree. + +--- + +## Frontend: React SPA + Vite + +The portal UI is a React app (`/home/jp/Work/voicemail-ui`) using +`react-router-dom`, TanStack Query, and a Vite build. It is built to `dist/` and +rsync'd to `/home/txt3/domains/vm.txt3.net/public_html/`. Apache serves the +static build and uses an SPA fallback (any non-`/api/`, non-`/audio/` path +returns `index.html`). All data goes through `/api/` → `vm_api.py`. + +Because the UI is JavaScript, the Apache CSP is **relaxed** to +`script-src 'self' 'unsafe-inline'` (see SECURITY.md). The dark "cyber" theme +uses neon accents; callers get a dropdown (⋮) on each message/contact row with +**Call back** (`tel:` link from sanitised digits) and **Lookup number** (opens a +who-called.co.uk search in a new tab, using the 0-prefixed UK number). + +--- + ## Why faster-whisper, CPU, int8, base.en - **faster-whisper** (CTranslate2) is dramatically faster than OpenAI's @@ -74,23 +127,28 @@ A stored path like `…/INBOX/msg0042.wav` therefore points at a *different* recording a week later. So: - On store, the audio is copied to - `audio//.` (blake2/sha256 in `vm_store`). + `audio//.`. - The spool path is kept only as a *delete hint*. - On delete, the blob is unlinked only if no other row references that hash. -This decouples the portal's playback from Asterisk's ever-shifting filenames. +This decouples the portal's playback (served by `vm_web.py` :8099 via +`/audio/`) from Asterisk's ever-shifting filenames. --- -## Why the portal is its own service behind Apache +## Contacts: MySQL only -- A long-running `uvicorn` is the natural fit; Asterisk's `mailcmd` model - (one short-lived process per call) is wrong for a web app. -- It binds **127.0.0.1:8099 only** (`IPAddressAllow=localhost` in the unit), so - the only way in is the Apache reverse proxy, which terminates TLS and enforces - headers. -- Running as the `asterisk` user means it can read `voicemail.conf` (for PIN - auth) and the spool (for future direct-file features) without widening root. +`vm_contacts.py` resolves caller IDs **exclusively** from the MySQL `contacts` +table. The older `file` / `google` / `carddav` backends were removed. + +- Numbers are stored canonical E.164; `normalize_uk()` collapses `+44…` and + `0…` (and `0044…`) forms so both match. +- Matching uses the full digit string (up to 15 digits), not just last-9, and + guards against emoji/placeholder contacts shadowing real names. +- Creating/updating a contact backfills `messages.contact_name` / + `contact_email` for every callerid that matches (spaces/quotes stripped before + the LIKE). +- `vm_import_contacts.py` loads a vCard/CSV export into the table. --- @@ -98,7 +156,7 @@ This decouples the portal's playback from Asterisk's ever-shifting filenames. Users log in with their **mailbox number + existing voicemail PIN** parsed from `/etc/asterisk/voicemail.conf` (`vm_auth.py`). No password database to keep in -sync, no new credentials to provision. Sessions are stored in the DB (random +sync, no new credentials to provision. Sessions are stored in MySQL (random token, `Secure` cookie, `HttpOnly`, `SameSite=lax`, 12 h) so logout and expiry are enforced server-side, not just by deleting a cookie. @@ -132,6 +190,7 @@ All audio and transcripts stay on the PBX. The only egress is: - **Postfix → your MX** for email (the existing path), - **Telegram API** (only if you enable Telegram), -- **Google People API** (only if you enable the `google` contacts backend). +- **who-called.co.uk** (only when a user clicks *Lookup number* in the UI — a + manual, user-initiated action that opens a new browser tab). Nothing is sent to OpenAI, Anthropic, or any transcription vendor. diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 70940c8..506a5fc 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -1,9 +1,8 @@ # Configuration -All config lives in `/opt/vm-transcribe/`. Templates are committed as -`*.example`; the live files are created by `install.sh` or by you. Secret-bearing -files (`telegram.conf`, `contacts.conf`) are mode `640 root:asterisk` and git- -ignored. +The backend services run from `/opt/vm-transcribe/`. Live secret-bearing files +live there (some git-ignored). The repo ships `*.example` templates under +`config/`. Most settings can also be set via environment variables, which is how the systemd unit wires production values. Env vars override file values when both are present @@ -11,17 +10,47 @@ unless noted. --- -## Environment variables (shared) +## Environment variables -| Variable | Default | Used by | Meaning | -|---|---|---|---| -| `VM_DB` | `/var/lib/vm-transcribe/voicemail.db` | store, web, import | SQLite path. Created with WAL mode if missing. | -| `VM_AUDIO_DIR` | `/var/lib/vm-transcribe/audio` | store, web, import | Where copied recordings live (`/.wav`). | -| `VM_LOG` | `/var/log/asterisk/vm_mailcmd.log` | mailcmd | Log path. | -| `VM_MODEL_CACHE` | `/opt/vm-transcribe/models` | mailcmd, import | Whisper model download/load dir. | -| `VM_WHISPER_MODEL` | `base.en` | mailcmd, import | Model name. `tiny`/`base`/`small` trade speed for accuracy. | -| `VM_SPOOL` | `/var/spool/asterisk/voicemail` | import | Spool root to scan. | -| `VM_ASTERISK_CONF` | `/etc/asterisk/voicemail.conf` | web, auth | For PIN lookup. | +### Shared / store (`vm_store.py`, used by mailcmd, web, api, import) + +| Variable | Default | Meaning | +|---|---|---| +| `VM_DB` | `/var/lib/vm-transcribe/voicemail.db` | SQLite path — **used only as a fallback** when no MySQL is configured. | +| `VM_AUDIO_DIR` | `/var/lib/vm-transcribe/audio` | Where copied recordings live (`/.wav`). | +| `VM_LOG` | `/var/log/asterisk/vm_mailcmd.log` | Log path. | +| `VM_MODEL_CACHE` | `/opt/vm-transcribe/models` | Whisper model download/load dir. | +| `VM_WHISPER_MODEL` | `base.en` | Model name. `tiny`/`base`/`small` trade speed for accuracy. | +| `VM_SPOOL` | `/var/spool/asterisk/voicemail` | Spool root to scan (import). | +| `VM_ASTERISK_CONF` | `/etc/asterisk/voicemail.conf` | For PIN lookup. | + +### MySQL (default store + contacts) + +Set these as env vars (the API unit reads `/opt/vm-transcribe/api.env`) **or** +put them in `/opt/vm-transcribe/db_secret` (mode `640 root:root`, +`KEY=VALUE` lines). When a credentials source exists, `vm_store` uses MySQL; +otherwise it falls back to SQLite. + +| Variable | Default | Meaning | +|---|---|---| +| `VM_MYSQL_HOST` | `localhost` | MySQL host. | +| `VM_MYSQL_USER` | `asterisk` | Least-privilege DB user (owns the `asterisk` DB; not root). | +| `VM_MYSQL_PASSWORD` | (from `db_secret`) | DB password. | +| `VM_MYSQL_DB` | `asterisk` | Database name — the **same** DB Asterisk's CDR uses. | + +`db_secret` key names: `MYSQL_HOST`, `MYSQL_USER`, `MYSQL_PASSWORD`, +`MYSQL_DB`. + +### Portal / API hardening + +| Var | Default | Meaning | +|---|---|---| +| `VM_SESSION_HOURS` | `12` | Session lifetime. | +| `VM_MAX_FAILS` | `5` | Failed attempts per (mailbox, IP) before locking. | +| `VM_LOCK_MINUTES` | `15` | Lock duration. | +| `VM_INSECURE_COOKIE` | unset | Set to `1`/`yes`/`true` only for plain-HTTP testing; normally cookies are `Secure`. | +| `VM_BASE_PATH` | `""` | Set to e.g. `/voicemail` if served under a sub-path. | +| `VM_CORS_ORIGINS` | `*` | Comma list of allowed CORS origins for the JSON API. | --- @@ -88,41 +117,38 @@ file still delivers the summary. ```ini [contacts] -backends = file ; space/comma separated: file google carddav -cache_ttl = 86400 ; seconds; cache hits AND misses -cache_file = /var/lib/vm-transcribe/contacts-cache.json +enabled = yes +backends = mysql ; ONLY 'mysql' is implemented (MySQL contacts table). + # The schema still parses file/google/carddav entries, + # but the registry is {"mysql": lookup_mysql} so any + # other value is ignored / logged as unknown. +cache_path = /var/lib/vm-transcribe/contacts_cache.json +cache_ttl = 86400 -# --- 'file' backend (recommended): a vCard or CSV export ------------- -[file] -path = /var/lib/vm-transcribe/contacts.vcf -# CSV (Google export "All contact data") is auto-detected: -# path = /var/lib/vm-transcribe/contacts.csv - -# --- 'google' backend: People API via OAuth --------------------------- -[google] -token_file = /var/lib/vm-transcribe/google-token.json -# Obtain a token with the People API (contacts.readonly) scope. See -# docs/SECURITY.md. No app password here — OAuth only. - -# --- 'carddav' backend: app password, Nextcloud/Fastmail/iCloud -------- -[carddav] -# Google rejects app passwords (basic auth disabled 2024-09-30); a Google -# URL here is refused with a log warning and the backend is skipped. -url = https://contacts.fastmail.com/dav/addressbooks/user/xxx/Default -username = jp@txt3.com -app_password = ; app-specific password, not your login password +[file] ; NOT USED — kept for reference only +[google] ; NOT USED — kept for reference only +[carddav] ; NOT USED — kept for reference only ``` -**Matching** is on the **last 9 digits** of the caller ID, so `+447700900123`, -`07700900123` and `447700900123` all collapse to the same contact. Names resolve -best-effort; on any backend error the caller ID is used as-is. +**Contacts are resolved exclusively from the MySQL `contacts` table** on the +`asterisk` DB. Numbers are stored canonical E.164; `normalize_uk()` collapses +`+44…` and `0…` (and `0044…`) so both forms match. Matching uses the full digit +string (up to 15 digits), not just last-9. Creating/updating a contact via the +API backfills `messages.contact_name`/`contact_email` for matching callerids. + +Load the table from a vCard/CSV export with: + +```bash +sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ + /opt/vm-transcribe/vm_import_contacts.py /path/to/contacts.vcf +``` --- -## Portal settings (per mailbox, in the SQLite store) +## Portal settings (per mailbox, in MySQL) -These are edited from the portal's **Settings** page by each user; they are not -config files. Every value defaults to `yes`. +These are edited from the React portal's **Settings** page by each user; they are +not config files. Every value defaults to `yes`. | Setting | Effect | |---|---| @@ -134,26 +160,33 @@ config files. Every value defaults to `yes`. --- -## vm-portal.service +## systemd units -Production values are passed as environment in the unit file: +Two units, both running as `asterisk` and binding loopback only. + +### vm-api.service (:8098 JSON API) + +```ini +Environment=VM_SESSION_HOURS=12 +EnvironmentFile=-/opt/vm-transcribe/api.env ; MySQL creds + VM_* overrides +# Hardening: ProtectSystem=full, ProtectHome=read-only, NoNewPrivileges, +# PrivateTmp, IPAddressAllow=127.0.0.1 +ExecStart=/opt/vm-transcribe/venv/bin/uvicorn vm_api:app --host 127.0.0.1 --port 8098 --workers 2 +``` + +### vm-portal.service (:8099 legacy HTML / audio) ```ini Environment=VM_DB=/var/lib/vm-transcribe/voicemail.db Environment=VM_AUDIO_DIR=/var/lib/vm-transcribe/audio Environment=VM_ASTERISK_CONF=/etc/asterisk/voicemail.conf Environment=VM_SESSION_HOURS=12 +ExecStart=/opt/vm-transcribe/venv/bin/python -m uvicorn vm_web:app \ + --host 127.0.0.1 --port 8099 --proxy-headers --forwarded-allow-ips 127.0.0.1 +# Hardening: NoNewPrivileges, PrivateTmp, ProtectSystem=full, ProtectHome, +# ReadWritePaths=/var/lib/vm-transcribe /var/log/asterisk, +# IPAddressAllow=localhost, IPAddressDeny=any ``` -Login hardening (in-process): - -| Var | Default | Meaning | -|---|---|---| -| `VM_MAX_FAILS` | `5` | Failed attempts per (mailbox, IP) before locking. | -| `VM_LOCK_MINUTES` | `15` | Lock duration. | -| `VM_SESSION_HOURS` | `12` | Session lifetime. | -| `VM_INSECURE_COOKIE` | unset | Set to `1` only for plain-HTTP testing; normally cookies are `Secure`. | -| `VM_BASE_PATH` | `""` | Set to e.g. `/voicemail` if the app is served under a sub-path. | - > A service restart clears the in-memory lockout counters. Testing lockout will > lock you out of your own verification for `VM_LOCK_MINUTES`. diff --git a/docs/INSTALL.md b/docs/INSTALL.md index 7248aca..011f5cd 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -1,11 +1,16 @@ # Installation -Written against Debian 12 + Asterisk 20 + Apache 2.4 + Postfix, running under -Virtualmin. Adjust paths if your layout differs. +Written against Debian 12 + Asterisk 20 + Apache 2.4 + Postfix + MySQL/MariaDB, +running under Virtualmin. Adjust paths if your layout differs. -Everything installs to `/opt/vm-transcribe` with data in +Everything installs to `/opt/vm-transcribe` with audio in `/var/lib/vm-transcribe`. Nothing is installed into system Python. +> **Frontend lives in a separate repo.** This backend repo ships a *synced +> copy* under `frontend/`. Build the real app in `/home/jp/Work/voicemail-ui` +> and rsync the result to the web root (see §6). Keep the two in sync with +> `rsync -a /home/jp/Work/voicemail-ui/ frontend/` then commit. + --- ## 0. Prerequisites @@ -16,7 +21,9 @@ grep -E '^(odbcstorage|imapserver)' /etc/asterisk/voicemail.conf # expect noth # tools sudo apt install ffmpeg sox python3-venv # sox optional (gsm) +sudo apt install default-mysql-server # or mariadb-server which certbot # for the portal's TLS +node --version && npm --version # Node 18+ for the frontend # an MTA on localhost:25 sudo ss -ltnp | grep ':25 ' @@ -32,25 +39,52 @@ df -h /var --- -## 1. Core pipeline +## 1. MySQL (storage + contacts + CDR) + +The backend shares the Asterisk `asterisk` database. Create a least-privilege +user and a credentials file: + +```bash +sudo mysql -e "CREATE USER IF NOT EXISTS 'asterisk'@'localhost' IDENTIFIED BY 'PICK_A_STRONG_PASSWORD';" +sudo mysql -e "GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,INDEX,ALTER ON asterisk.* TO 'asterisk'@'localhost';" +sudo mysql -e "FLUSH PRIVILEGES;" +``` + +Write `/opt/vm-transcribe/db_secret` (mode `640 root:root`): + +```bash +sudo install -o root -g root -m 640 /dev/null /opt/vm-transcribe/db_secret +sudo tee /opt/vm-transcribe/db_secret >/dev/null <<'EOF' +MYSQL_HOST=localhost +MYSQL_USER=asterisk +MYSQL_PASSWORD=PICK_A_STRONG_PASSWORD +MYSQL_DB=asterisk +EOF +``` + +`vm_store` uses MySQL whenever this file (or `VM_MYSQL_*` env vars) exists; +otherwise it falls back to SQLite. The schema is created on first connect. + +(Optional) To load call records into the same DB, wire Asterisk CDR via +`cdr_adaptive_odbc` + `res_odbc` + the MariaDB ODBC driver, then run +`src/vm_backfill_cdr.py` to import `Master.csv` history. + +--- + +## 2. Core pipeline (mailcmd + venv) + +The `install.sh` script builds the venv, installs `faster-whisper`, deploys the +mailcmd scripts, seeds `telegram.conf`, pre-caches the whisper model **as the +asterisk user**, and rewrites `voicemail.conf`'s `mailcmd=`. + +> `install.sh` reads sources from `/home/jp/asterisk-vm` (the deploy path in +> AGENTS.md). Either symlink/clobber that path to this repo, or edit the `SRC=` +> line at the top of `scripts/install.sh` before running. ```bash sudo scripts/install.sh ``` -That script: - -1. creates `/opt/vm-transcribe` and `/opt/vm-transcribe/models` owned by `asterisk` -2. builds a venv and installs `faster-whisper` -3. installs the Python modules -4. seeds `telegram.conf` / `contacts.conf` if absent (never overwrites) -5. creates `/var/log/asterisk/vm_mailcmd.log` -6. **pre-downloads the whisper model as the `asterisk` user** — do not skip - this, or the first real voicemail pays a ~150 MB download while the caller - waits for their notification -7. backs up `voicemail.conf` and rewrites `mailcmd=` -8. reloads Asterisk - Verify the wiring: ```bash @@ -64,7 +98,7 @@ Build a fake notification and pipe it through **as the `asterisk` user** — testing as yourself hides permission problems: ```bash -cd /path/to/repo +cd /home/jp/Work/asterisk-voicemail python3 tests/make_test_mail.py /path/to/some.wav you@example.com > /tmp/t.eml sudo chmod 644 /tmp/t.eml sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ @@ -76,7 +110,7 @@ You should see `transcribed … chars` then `sent enriched notification to …`. Confirm the mail actually left: ```bash -sudo grep "to=" /var/log/mail.log | tail -1 # expect status=sent +sudo grep "to=" /var/log/mail.log | tail -1 # status=sent ``` Also verify the fail-safe — a message with **no** audio must relay unchanged @@ -92,7 +126,7 @@ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ --- -## 2. Telegram (optional) +## 3. Telegram (optional) 1. Create a bot: message [@BotFather](https://t.me/BotFather) → `/newbot` → copy the token. 2. Put it in `/opt/vm-transcribe/telegram.conf` (`sudo`, mode 640 root:asterisk). @@ -111,95 +145,135 @@ per-mailbox routing. --- -## 3. Contact lookup (optional) +## 4. Contacts (MySQL) -Simplest and most reliable — a local export, no tokens, no rate limits: +Contacts live in the MySQL `contacts` table — there is no file/Google/CardDAV +backend anymore. Load your address book from a vCard/CSV export: -1. contacts.google.com → **Export** → **vCard** -2. `sudo install -o asterisk -g asterisk -m 640 contacts.vcf /var/lib/vm-transcribe/contacts.vcf` -3. Confirm `contacts.conf` has `backends = file` and the matching `path`. +```bash +sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ + /opt/vm-transcribe/vm_import_contacts.py /path/to/contacts.vcf +``` + +Confirm lookup works: ```bash sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ /opt/vm-transcribe/vm_contacts.py '<07700900123>' ``` -> **Google app passwords do not work for contacts.** Google disabled basic auth -> for CardDAV/CalDAV/IMAP/SMTP/POP on 2024-09-30. Use the `file` or `google` -> (OAuth) backend. The `carddav` backend is for Nextcloud/Fastmail/iCloud. - --- -## 4. Portal service +## 5. Backend services + +Install both units (API + legacy portal/audio): ```bash sudo install -d -o asterisk -g asterisk -m 750 \ /var/lib/vm-transcribe /var/lib/vm-transcribe/audio -sudo /opt/vm-transcribe/venv/bin/pip install fastapi 'uvicorn[standard]' python-multipart +sudo /opt/vm-transcribe/venv/bin/pip install fastapi 'uvicorn[standard]' python-multipart pymysql + +# API unit reads MySQL creds from api.env +sudo install -o root -g root -m 640 /opt/vm-transcribe/db_secret /opt/vm-transcribe/api.env +sudo install -o root -g root -m 644 systemd/vm-api.service /etc/systemd/system/ sudo install -o root -g root -m 644 systemd/vm-portal.service /etc/systemd/system/ sudo systemctl daemon-reload -sudo systemctl enable --now vm-portal -sudo systemctl is-active vm-portal -curl -s http://127.0.0.1:8099/healthz # {"ok":true,"messages":N} +sudo systemctl enable --now vm-api vm-portal +sudo systemctl is-active vm-api vm-portal + +curl -s http://127.0.0.1:8098/api/healthz # {"ok":true,...} +curl -s http://127.0.0.1:8099/healthz # {"ok":true,"messages":N} ``` -The service runs as `asterisk` (so it can read `voicemail.conf` and the spool) -and binds **loopback only** — `IPAddressAllow=localhost` means Apache is the +The services run as `asterisk` (so they can read `voicemail.conf` and the spool) +and bind **loopback only** — `IPAddressAllow=localhost` means Apache is the only possible client. --- -## 5. Apache vhost + TLS +## 6. React frontend (build + deploy) + +The interactive UI is a React SPA built in `/home/jp/Work/voicemail-ui`: + +```bash +cd /home/jp/Work/voicemail-ui +npm install +npm run build # outputs dist/ +``` + +Deploy the static build to the web root (Apache serves it and proxies `/api/` +to :8098 and `/audio/` to :8099): + +```bash +sudo -A rsync -a --exclude node_modules --exclude dist \ + /home/jp/Work/voicemail-ui/ /home/txt3/domains/vm.txt3.net/public_html/ +``` + +Keep this repo's copy in sync for documentation/commit purposes: + +```bash +rsync -a /home/jp/Work/voicemail-ui/ frontend/ # then git commit under frontend/ +``` + +--- + +## 7. Apache vhost + TLS **Order matters.** Installing a vhost that references a cert which does not yet exist breaks `apache2ctl configtest`, which blocks reloads for *every* site on the box. So: DNS → cert → full vhost. -### 5a. DNS +### 7a. DNS ```bash -dig +short vm.example.com A # must return your ORIGIN ip +dig +short vm.txt3.net A # must return your ORIGIN ip ``` Behind Cloudflare, set the record to **DNS only (grey cloud)** first, or the HTTP-01 challenge is intercepted by the edge. -### 5b. Minimal :80 vhost, so certbot can answer the challenge +### 7b. Minimal :80 vhost, so certbot can answer the challenge ```bash -sudo install -d -o www-data -g www-data -m 755 /var/www/vm.example.com/public_html -sudo cp apache/vm.txt3.net-step1.conf /etc/apache2/sites-available/vm.example.com.conf +sudo install -d -o www-data -g www-data -m 755 /var/www/vm.txt3.net/public_html +sudo cp apache/vm.txt3.net-step1.conf /etc/apache2/sites-available/vm.txt3.net.conf # edit ServerName / paths / IPs to match your host -sudo a2ensite vm.example.com +sudo a2ensite vm.txt3.net sudo apache2ctl configtest && sudo systemctl reload apache2 ``` -### 5c. Issue the certificate +### 7c. Issue the certificate ```bash -sudo certbot certonly --webroot -w /var/www/vm.example.com/public_html \ - -d vm.example.com --cert-name vm.example.com +sudo certbot certonly --webroot -w /var/www/vm.txt3.net/public_html \ + -d vm.txt3.net --cert-name vm.txt3.net ``` -### 5d. Full vhost — HTTPS + reverse proxy +### 7d. Full vhost — HTTPS + reverse proxy ```bash sudo a2enmod proxy proxy_http headers rewrite ssl -sudo cp apache/vm.txt3.net.conf /etc/apache2/sites-available/vm.example.com.conf -# edit ServerName, cert paths, IPs -sudo install -d /var/www/vm.example.com/public_html/.well-known/acme-challenge +sudo cp apache/vm.txt3.net.conf /etc/apache2/sites-available/vm.txt3.net.conf +# edit ServerName, cert paths, IPs, DocumentRoot +sudo install -d /var/www/vm.txt3.net/public_html/.well-known/acme-challenge sudo apache2ctl configtest && sudo systemctl reload apache2 ``` -### 5e. Verify through the real hostname +The shipped `apache/vm.txt3.net.conf` already proxies `/api/` → :8098, +`/audio/` → :8099, serves the React `dist` as `DocumentRoot`, and adds an SPA +fallback (`RewriteRule ^ /index.html`). + +### 7e. Verify through the real hostname Not localhost — the whole point is to exercise Apache, TLS and the proxy: ```bash -R="--resolve vm.example.com:443:YOUR.ORIGIN.IP" -curl -s $R https://vm.example.com/healthz -curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.example.com/login -curl -s $R -D- -o /dev/null https://vm.example.com/login | grep -i 'strict-transport\|content-security' +R="--resolve vm.txt3.net:443:ORIGIN.IP" +curl -s $R https://vm.txt3.net/api/healthz +curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.txt3.net/login +curl -s $R -D- -o /dev/null https://vm.txt3.net/login | grep -iE 'strict-transport|content-security|x-frame' +# SPA fallback +curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.txt3.net/some/route ``` Behind a CDN, `curl https://host/` tests the *CDN*, not your origin. Always @@ -207,13 +281,14 @@ Behind a CDN, `curl https://host/` tests the *CDN*, not your origin. Always --- -## 6. Backfill existing voicemails +## 8. Backfill existing voicemails ```bash V=/opt/vm-transcribe/venv/bin/python3 sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --dry-run # preview sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --limit 5 # trial sudo -u asterisk $V /opt/vm-transcribe/vm_import.py # all +sudo -u asterisk $V /opt/vm-transcribe/vm_backfill_cdr.py # CDR history ``` Idempotent — safe to re-run; it resumes rather than duplicating. Expect a @@ -224,9 +299,9 @@ on 4 cores. --- -## 7. Log in +## 9. Log in -Browse to `https://vm.example.com/` and log in with a **mailbox number and its +Browse to `https://vm.txt3.net/` and log in with a **mailbox number and its existing voicemail PIN** from `voicemail.conf`. No new passwords are created. --- @@ -234,11 +309,13 @@ existing voicemail PIN** from `voicemail.conf`. No new passwords are created. ## Uninstall ```bash -sudo systemctl disable --now vm-portal -sudo rm /etc/systemd/system/vm-portal.service && sudo systemctl daemon-reload -sudo a2dissite vm.example.com && sudo systemctl reload apache2 +sudo systemctl disable --now vm-api vm-portal +sudo rm /etc/systemd/system/vm-api.service /etc/systemd/system/vm-portal.service +sudo systemctl daemon-reload +sudo a2dissite vm.txt3.net && sudo systemctl reload apache2 # restore the original mailcmd sudo cp /etc/asterisk/voicemail.conf.bak- /etc/asterisk/voicemail.conf sudo asterisk -rx 'voicemail reload' sudo rm -rf /opt/vm-transcribe # keep /var/lib/vm-transcribe for the data +# MySQL: DROP USER 'asterisk'@'localhost'; (optionally DROP DATABASE asterisk;) ``` diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index cf11022..4a43863 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -4,19 +4,21 @@ Day-to-day running: backups, monitoring, common problems, upgrades. --- -## Service status +## Services ```bash -sudo systemctl status vm-portal # the portal -sudo journalctl -u vm-portal -n 50 # portal logs (uvicorn access/startup) +sudo systemctl status vm-api vm-portal # API (:8098) + legacy portal (:8099) +sudo journalctl -u vm-api -n 50 # API logs (uvicorn access/startup) +sudo journalctl -u vm-portal -n 50 # legacy portal logs sudo tail -f /var/log/asterisk/vm_mailcmd.log # per-voicemail pipeline log ``` -Health of the portal (works locally or via the real hostname): +Health checks: ```bash -curl -s http://127.0.0.1:8099/healthz # {"ok":true,"messages":N} -curl -s --resolve vm.txt3.net:443:ORIGIN_IP https://vm.txt3.net/healthz +curl -s http://127.0.0.1:8098/api/healthz # JSON API health +curl -s http://127.0.0.1:8099/healthz # legacy portal health +curl -s --resolve vm.txt3.net:443:ORIGIN_IP https://vm.txt3.net/api/healthz ``` --- @@ -26,7 +28,7 @@ curl -s --resolve vm.txt3.net:443:ORIGIN_IP https://vm.txt3.net/healthz | Signal | Where | Meaning | |---|---|---| | `transcribed …` | vm_mailcmd.log | a voicemail was processed OK | -| `stored message for mailbox …` | vm_mailcmd.log | it also reached the portal DB | +| `stored message for mailbox …` | vm_mailcmd.log | it also reached the MySQL store | | `no audio attachment; relaying original` | vm_mailcmd.log | Asterisk sent a text-only notice (missed-call style) — expected | | `Telegram disabled` / `no route` | vm_mailcmd.log | Telegram skipped (per config) — expected if unset | | `relaying original, mailcmd error` | vm_mailcmd.log | **pipeline threw**; original email was preserved (fail-safe worked) | @@ -37,23 +39,29 @@ curl -s --resolve vm.txt3.net:443:ORIGIN_IP https://vm.txt3.net/healthz ## Backups -Two things to back up — the code is reproducible, the *data* is not: +Two things to back up — the code is reproducible, the *data* is not. The MySQL +`asterisk` database now holds `messages`, `contacts` and CDR, so back up the DB +(not a SQLite file): ```bash -# database + audio (once or twice a day is plenty) -sudo -u asterisk tar czf /backup/vm-$(date +%F).tgz \ - -C / var/lib/vm-transcribe/voicemail.db var/lib/vm-transcribe/audio +# MySQL dump (messages + contacts + cdr) — run as root +sudo mysqldump --single-transaction asterisk \ + > /backup/vm-mysql-$(date +%F).sql + +# audio blobs (still on disk) +sudo -u asterisk tar czf /backup/vm-audio-$(date +%F).tgz -C / var/lib/vm-transcribe/audio # config (the live, secret-bearing files) sudo tar czf /backup/vm-conf-$(date +%F).tgz \ - /opt/vm-transcribe/telegram.conf /opt/vm-transcribe/contacts.conf + /opt/vm-transcribe/telegram.conf /opt/vm-transcribe/contacts.conf \ + /opt/vm-transcribe/db_secret /opt/vm-transcribe/api.env ``` The whisper model cache (`/opt/vm-transcribe/models`) is reproducible — no need to back it up, just re-run `install.sh`. -Restoring: stop the service, extract, `systemctl start vm-portal`. The DB schema -is created on first connect, but **preserve the existing file** to keep history. +Restoring: stop the services, load the dump into MySQL, extract audio to +`/var/lib/vm-transcribe/audio`, `systemctl start vm-api vm-portal`. --- @@ -76,15 +84,25 @@ storage`. Check `df -h /var` first when the pipeline looks dead. ```bash git -C /home/jp/Work/asterisk-voicemail pull cd /home/jp/Work/asterisk-voicemail + +# backend sudo cp src/*.py /opt/vm-transcribe/ sudo /opt/vm-transcribe/venv/bin/pip install -U faster-whisper # occasionally -sudo systemctl restart vm-portal +sudo systemctl restart vm-api vm-portal + +# frontend (build in the UI repo, rsync the dist) +cd /home/jp/Work/voicemail-ui && npm install && npm run build +sudo -A rsync -a --exclude node_modules --exclude dist \ + /home/jp/Work/voicemail-ui/ /home/txt3/domains/vm.txt3.net/public_html/ ``` `install.sh` is idempotent-ish for the first install but is **not** a general upgrade tool — it backs up `voicemail.conf` each run, so avoid re-running it blindly. For upgrades, copy `src/*.py` as above. +> Note: the deploy paths in AGENTS.md use `/home/jp/asterisk-vm` and +> `/home/jp/Work/voicemail-ui` respectively — keep the two in sync. + --- ## Common problems @@ -96,20 +114,27 @@ blindly. For upgrades, copy `src/*.py` as above. 4. `sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 -c 'import faster_whisper'` — is the venv intact? **Portal returns 502 / can't connect.** -The uvicorn backend is down or not on loopback: `sudo systemctl status vm-portal`; -`ss -ltnp | grep 8099`. Also confirm Apache has `proxy`/`proxy_http` enabled. +The API backend is down or not on loopback: `sudo systemctl status vm-api`; +`ss -ltnp | grep 8098`. Also confirm Apache has `proxy`/`proxy_http` enabled and +that `/api/` is proxied to `127.0.0.1:8098`. (Legacy `/audio/` 502 → check +`vm-portal` on :8099.) **Login fails for a real mailbox.** -The PIN in `voicemail.conf` is what the portal checks — not anything in the -portal. If you changed a voicemail PIN, that change is picked up immediately -(the file is reparsed each login). If the mailbox line is commented out or in a -context the parser doesn't reach, login fails. Test the parser directly: +The PIN in `voicemail.conf` is what the API checks — not anything in the DB. If +you changed a voicemail PIN, that change is picked up immediately (the file is +reparsed each login). If the mailbox line is commented out or in a context the +parser doesn't reach, login fails. Test the parser directly: ```bash sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ -c "import sys; sys.path.insert(0,'/opt/vm-transcribe'); import vm_auth; print(vm_auth.check_login('7940','5159'))" ``` +**MySQL connection errors in the API.** +Check `/opt/vm-transcribe/db_secret` (or `api.env`) and that the `asterisk` DB +user exists with access to the `asterisk` database. The API logs a clear +`vm_store unavailable` message if MySQL is unreachable. + **Certbot renewal.** Renewal runs from `/etc/cron.d/certbot`; `certbot.timer` is masked (normal on Debian's package). Test non-destructively in the background (it can be slow): @@ -119,7 +144,7 @@ sudo certbot renew --cert-name vm.txt3.net --dry-run & ``` **I locked myself out testing lockout.** -Restart the service: `sudo systemctl restart vm-portal`. In-memory counters clear. +Restart the service: `sudo systemctl restart vm-api`. In-memory counters clear. **Backfill shows many `no_speech`.** Expected. 44-byte WAVs (`duration=0`) are hung-up calls with no audio. Not a bug. @@ -135,4 +160,9 @@ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_impor ``` Useful after importing a large batch of new spool messages, or after a fresh -install on a box with existing voicemails. +install on a box with existing voicemails. To also load CDR history from +`Master.csv`: + +```bash +sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_backfill_cdr.py +``` diff --git a/docs/SECURITY.md b/docs/SECURITY.md index c4b129f..1602900 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -13,8 +13,8 @@ their messages — often personal or commercial. The portal also proves who 1. An attacker reads someone else's voicemail (broken authz or IDOR). 2. An attacker brute-forces a 4-digit PIN at scale. -3. A stored credential (Telegram token, contacts app password) leaks. -4. XSS / injection poisons the HTML shown to a user. +3. A stored credential (MySQL password, Telegram token, contacts export) leaks. +4. XSS / injection poisons the HTML/JSON shown to a user. 5. The recording audio leaks off-box. --- @@ -23,43 +23,50 @@ their messages — often personal or commercial. The portal also proves who **Authentication — mailbox PIN from voicemail.conf.** No new password store to breach; the portal trusts Asterisk's PINs. Sessions are -DB-backed (random token, `Secure`+`HttpOnly`+`SameSite=lax`, 12 h). Logout and -expiry are enforced server-side, not merely by dropping the cookie. +MySQL-backed (random token, `Secure`+`HttpOnly`+`SameSite=lax`, 12 h). Logout +and expiry are enforced server-side, not merely by dropping the cookie. **Authorization — every query is mailbox-scoped.** -`vm_store` and `vm_web` filter every read and delete by the session's mailbox. +`vm_store` and `vm_api` filter every read and delete by the session's mailbox. Cross-mailbox reads/deletes return **404**. There is a test that asserts this -([TESTING.md](TESTING.md)). A mailbox owner can only ever see their own messages. +([TESTING.md](TESTING.md)). A mailbox owner can only ever see their own messages +(and their own contacts). **Brute-force lockout.** 5 failed attempts per `(mailbox, IP)` → 15-minute lockout. The correct PIN is -also refused while locked. State is in-memory (single worker); a restart clears -it. +also refused while locked. State is in-memory (single worker per service); a +restart clears it. **Transport — TLS terminated at Apache, with headers.** -`Strict-Transport-Security`, `X-Frame-Options: DENY`, `X-Content-Type-Options: -nosniff`, `Referrer-Policy`, and a tight CSP: +`Strict-Transport-Security`, `X-Frame-Options: DENY`, +`X-Content-Type-Options: nosniff`, `Referrer-Policy`, and a CSP: -``` +```text default-src 'self'; style-src 'self' 'unsafe-inline'; -media-src 'self'; img-src 'self' data:; -script-src 'none'; frame-ancestors 'none'; -base-uri 'none'; form-action 'self' +img-src 'self' data:; font-src 'self'; +media-src 'self'; connect-src 'self' http://127.0.0.1:8098; +script-src 'self' 'unsafe-inline'; frame-ancestors 'none'; base-uri 'none' ``` -**No JavaScript in the portal at all**, which is what lets `script-src 'none'` -be genuinely enforceable — there is nothing to inject. +**The portal now ships JavaScript** (a React SPA), so the CSP is *relaxed* from +the original `script-src 'none'` to `script-src 'self' 'unsafe-inline'`. Inline +scripts in the built bundle are still same-origin; `base-uri 'none'` and +`frame-ancestors 'none'` remain to blunt injection / framing. If you can build +the frontend without inline scripts, tighten this back to `'self'` only. **Network exposure is minimal.** -The uvicorn backend binds **loopback only** (`IPAddressAllow=localhost` in the -unit). The only ingress is Apache. The systemd unit also sets -`NoNewPrivileges`, `ProtectSystem=full`, `ProtectHome`, `PrivateTmp`, -`ProtectKernelTunables`, `ProtectControlGroups`, `RestrictSUIDSGID`, and writable -paths are limited to `/var/lib/vm-transcribe` and `/var/log/asterisk`. +Both uvicorn backends bind **loopback only** +(`IPAddressAllow=127.0.0.1` / `localhost` in the units). The only ingress is +Apache. The `vm-api` unit also sets `ProtectSystem=full`, `ProtectHome=read-only`, +`NoNewPrivileges`, `PrivateTmp`; `vm-portal` adds `ProtectKernelTunables`, +`ProtectControlGroups`, `RestrictSUIDSGID`, `IPAddressDeny=any`, and writable +paths limited to `/var/lib/vm-transcribe` and `/var/log/asterisk`. **Secrets are separated and git-ignored.** -`telegram.conf`, `contacts.conf`, `*.db`, `audio/` are in `.gitignore`. The -repo ships `*.example` templates only. Both secret files are mode `640 root:asterisk`. +`telegram.conf`, `contacts.conf`, `db_secret`, `api.env`, `*.db`, `audio/` are +in `.gitignore`. The repo ships `*.example` templates only. `db_secret` and +`telegram.conf` are mode `640 root:asterisk`; `api.env` is mode `640` +root:root. **Fail-safe preserves mail.** A pipeline exception relays the original Asterisk message unchanged — we never @@ -78,22 +85,22 @@ access log. Recommended. Acceptable (same host, no network path), but if you ever run uvicorn on another host, use a unix socket or mTLS. -**(R3) Session token in SQLite.** If the DB file is stolen, sessions are -replayable until they expire. DB is `640 asterisk:asterisk` and not web-served. -For higher assurance, store sessions in a server-side cache with shorter TTLs. +**(R3) Session token in MySQL.** If the DB is stolen, sessions are replayable +until they expire. The `asterisk` DB is not web-served and the `asterisk` user is +least-privilege; keep `db_secret` at `640 root:root`. For higher assurance, +shorten `VM_SESSION_HOURS` or store sessions in a server-side cache with short +TTLs. -**(R4) Contact resolver tokens.** A Google OAuth token or a CardDAV app password -grants read access to your address book. Scope the Google token to -`contacts.readonly`, use a dedicated app password (never your login password), -and keep these files `640`. Prefer the `file` backend (a periodic vCard export) — -no live token at all. +**(R4) Contact data.** Contacts now live in the MySQL `contacts` table (no live +Google/CardDAV token at all — the old OAuth/app-password backends were removed). +The only secret around contacts is the DB password. Prefer the MySQL backend +(which we are already on) — there is no external address-book token to leak. **(R5) Telegram bot token.** Whoever holds it can post as your bot. Keep `telegram.conf` `640 root:asterisk`; rotate via @BotFather if leaked. -**(R6) Content-addressed audio filenames.** The sha256 of the audio is in the URL -(`/audio/`). An attacker who guesses a sha could fetch that recording -without a session. The portal checks the session's mailbox *owns* that message +**(R6) Content-addressed audio filenames.** The sha256 of the audio is used to +locate the blob. The portal checks the session's mailbox *owns* that message before serving, so this is not directly exploitable — but consider a random per-message token instead of the content hash if you want defence in depth. @@ -101,17 +108,17 @@ per-message token instead of the content hash if you want defence in depth. audio streams through the edge. Grey-cloud it (DNS only) to keep voice data off the CDN. Your call. +**(R8) who-called.co.uk lookups.** The *Lookup number* action opens a new browser +tab to who-called.co.uk using the caller's sanitised digits. This is a +user-initiated click from the UI — no data is sent server-side. It does mean the +user's browser (and the caller's number) reach a third-party site; acceptable for +a manual reverse-lookup, but worth knowing. + --- -## Obtaining a Google token (if you use the `google` backend) +## Past Google-contacts note (no longer applicable) -1. Google Cloud console → OAuth consent screen (External) → add your account as - a test user. -2. Credentials → OAuth client ID → **Desktop app**. -3. Scope `https://www.googleapis.com/auth/contacts.readonly`. -4. Authorize once (the token is written to `google-token.json`, mode `640`). - The People API is used, not CardDAV, because basic auth / app passwords were - disabled by Google on **2024-09-30**. - -There is no support for an app-password read of Google Contacts — it does not -work. Use the `file` or `google` backend. +The old `file` / `google` / `carddav` contact backends were removed in favour of +the MySQL `contacts` table. Historically, Google disabled basic auth for +CardDAV/CalDAV/IMAP on **2024-09-30**, so an app-password read of Google +Contacts never worked — that is moot now that contacts are a local MySQL table. diff --git a/docs/TESTING.md b/docs/TESTING.md index 71ffd5b..06368ab 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -1,7 +1,7 @@ # Testing -How to verify each part of the system. Do the mailcmd tests **as the `asterisk` -user**, not as yourself — that's where permission bugs hide. +How to verify each part of the system. Do the mailcmd tests **as the `asterisk`** +user, not as yourself — that's where permission bugs hide. --- @@ -10,7 +10,8 @@ user**, not as yourself — that's where permission bugs hide. ```bash df -h /var # must not be full (Postfix 452) grep '^mailcmd' /etc/asterisk/voicemail.conf -sudo systemctl is-active vm-portal +sudo systemctl is-active vm-api vm-portal +curl -s http://127.0.0.1:8098/api/healthz curl -s http://127.0.0.1:8099/healthz ``` @@ -29,7 +30,7 @@ group id negative; unmapped → default; `enabled=no` → no route; caption clip to 1017 chars on a word boundary when over 1024; a 374 KB wav → 64 KB `OggS` opus. -`test_contacts.py` asserts: `+447700900123`, `07700900123`, `447700900123` all +`test_contacts.py` asserts: `+447****0123`, `07700900123`, `447700900123` all match one contact (last-9-digit key); multi-TEL cards; Google CSV `:::` split. --- @@ -64,8 +65,9 @@ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ ### Fail-safe: a thrown error relays the original -Temporarily point `VM_DB` at an unreadable path; the script should fall back to -relaying the original Asterisk message and log `relaying original, mailcmd error`. +Temporarily point `VM_DB` at an unreadable path (or MySQL offline); the script +should fall back to relaying the original Asterisk message and log +`relaying original, mailcmd error`. --- @@ -83,47 +85,49 @@ need a token to verify the config parser. --- -## 4. Contacts +## 4. Contacts (MySQL) ```bash +# resolve a caller ID against the MySQL contacts table sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ /opt/vm-transcribe/vm_contacts.py '<07700900123>' -# resolves against the configured backend; prints name or the raw caller id +# prints (name, email) or the raw caller id + +# load an export +sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ + /opt/vm-transcribe/vm_import_contacts.py /path/to/contacts.vcf ``` --- -## 5. Portal — auth, authz, playback, delete +## 5. JSON API — auth, authz, playback, delete -Run the app locally (or against the live service) and exercise it with curl. +Run the API locally (or against the live service) and exercise it with curl. Use `VM_INSECURE_COOKIE=1` only for plain-HTTP local tests so the `Secure` cookie can be set. ```bash +B=http://127.0.0.1:8098 + # login ok -curl -s -c /tmp/j -o /dev/null -w '%{redirect_url}\n' \ - -d 'mailbox=7940&pin=5159' http://127.0.0.1:8099/login +curl -s -c /tmp/j -o /dev/null -w '%{redirect_url}\n' -d 'mailbox=7940&pin=5159' $B/api/login # wrong pin rejected -curl -s -o /dev/null -w '%{http_code}\n' -d 'mailbox=7940&pin=1111' http://127.0.0.1:8099/login +curl -s -o /dev/null -w '%{http_code}\n' -d 'mailbox=7940&pin=1111' $B/api/login -# authenticated list -curl -s -b /tmp/j http://127.0.0.1:8099/ | grep -c 'class="card' +# authenticated message list +curl -s -b /tmp/j $B/api/messages | head -c 400 -# audio streams -curl -s -b /tmp/j -o /tmp/x.wav http://127.0.0.1:8099/audio/1 -file /tmp/x.wav # RIFF WAVE, 8000 Hz mono - -# delete -curl -s -b /tmp/j -X POST -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8099/delete/1 +# contacts CRUD +curl -s -b /tmp/j $B/api/contacts | head -c 400 ``` ### Authorization (IDOR) — must 404 A session for mailbox A must never reach mailbox B's data. With a cookie for -mailbox 7940, hitting `/audio/` and `/delete/` -must both return **404/303**, never serve or delete the other mailbox's message. -Add this assertion whenever you change `vm_store` or `vm_web`. +mailbox 7940, hitting `/api/messages/{id-owned-by-1001}` (or its audio/delete +variants) must return **404**, never serve or delete the other mailbox's message. +Add this assertion whenever you change `vm_store` or `vm_api`. --- @@ -131,13 +135,20 @@ Add this assertion whenever you change `vm_store` or `vm_web`. ```bash R="--resolve vm.txt3.net:443:ORIGIN.IP" -curl -s $R https://vm.txt3.net/healthz +curl -s $R https://vm.txt3.net/api/healthz curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.txt3.net/login curl -s $R -D- -o /dev/null https://vm.txt3.net/login | grep -iE 'strict-transport|content-security|x-frame' # http -> https redirect curl -s $R -o /dev/null -w '%{redirect_url}\n' http://vm.txt3.net/ ``` +The React SPA is served as static files; all data flows through `/api/`. Confirm +the SPA fallback returns `index.html` for arbitrary client-side routes: + +```bash +curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.txt3.net/some/client/route +``` + Behind a CDN, `--resolve` to the **origin** IP; otherwise you are testing the CDN, not your server. @@ -155,5 +166,11 @@ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \ # real: 3 imported, others skipped; re-running is a no-op (idempotent) ``` -Verify they appear in the portal and are playable (§5). Expect a `no_speech` -count — 44-byte WAVs are hung-up calls, not failures. +Verify they appear via the API/messages endpoint and are playable (§5). Expect a +`no_speech` count — 44-byte WAVs are hung-up calls, not failures. + +CDR history import: + +```bash +sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_backfill_cdr.py +```