jp/asterisk-voicemail
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update all documentation to current MySQL + React SPA + vm_api.py architecture

Browse Source 
README, docs/{ARCHITECTURE,CONFIGURATION,INSTALL,OPERATIONS,SECURITY,TESTING}
and CHANGELOG were stale: they described the original SQLite + CardDAV +
zero-JS vm_web portal. Brought them in line with the actual code:
- MySQL (PyMySQL) as the default store; SQLite fallback only
- vm_contacts resolves from the MySQL contacts table (file/google/carddav removed)
- React SPA frontend + vm_api.py JSON API (:8098); vm_web.py now serves /audio/
- Apache vhost proxies /api/ -> :8098, /audio/ -> :8099, serves the SPA
- CSP relaxed to script-src 'self' 'unsafe-inline' (UI is now JavaScript)
- Added CHANGELOG 1.2.0 entry for the React/vm_api frontend work
This commit is contained in:
jp
2026-08-13 20:20:39 +01:00
parent 3155020f67
commit b8c60605fd
8 changed files with 553 additions and 290 deletions
Download Patch File Download Diff File Expand all files Collapse all files

132
README.md
View File

@ -9,60 +9,53 @@ manage:
- **Emails** a graphically designed `multipart/alternative` notification
(plain + HTML) with the recording attached
- **DMs Telegram** optionally, as a playable voice note with the summary
- **Resolves caller ID to a name** from Google Contacts or any CardDAV server
- **Serves a web portal** where mailbox users log in with their existing phone
PIN to read transcripts, play/download recordings, delete messages and
manage their own notification settings
- **Resolves caller ID to a name** from the MySQL `contacts` table
- **Serves a React web portal** where mailbox users log in with their existing
phone PIN to read transcripts, play/download recordings, delete messages,
manage contacts and look up unknown numbers
Everything runs on the PBX host. No third-party service sees your voicemail.
---
## What it looks like
**Email notification** — styled HTML card with metadata, summary panel, intent
chips, a `tel:` callback link, full transcript and the audio attached. A plain
text alternative is always included.
**Telegram** — voice note (ogg/opus) with the summary as its caption, intent
hashtags, and the transcript as a follow-up message.
**Portal** — one card per voicemail: caller, timestamp, duration, summary,
tags, callback link, inline player, collapsible transcript, and Mark read /
Download / Delete.
---
## Architecture
## Architecture (current)
```
incoming call
│
▼
incoming call
│
▼
┌─────────────┐ voicemail.conf: mailcmd=… vm_mailcmd.py
│ Asterisk │ ─────────────────────────────┐
└─────────────┘ pipes an RFC822 message │
(notification + audio) ▼
└─────────────┘ pipes an RFC822 message │
(notification + audio) ▼
┌───────────────┐
│ vm_mailcmd.py │
└───────┬───────┘
┌──────────────┬──────────────┬────┴─────────┬──────────────┐
▼ ▼ ▼ ▼ ▼
faster-whisper summarise() vm_contacts vm_store vm_telegram
(transcribe) + intents (name lookup) (SQLite + (voice note
(transcribe) + intents (MySQL name) (MySQL + (voice note
+ numbers audio CAS) DM)
│ │
▼ ▼
multipart email ──▶ Postfix :25 ┌──────────────┐
│ vm_web.py │
│ (FastAPI) │
└──────┬───────┘
│ :8099 loopback
▼
Apache (TLS)
│
▼
https://vm.txt3.net
multipart email ──▶ Postfix :25 ┌──────────────────────────┐
│ MySQL (asterisk DB) │
│ messages, contacts, CDR │
└──────┬───────────┬─────────┘
│ │
┌──────────────────────┘ └──────────────────────┐
▼ ▼
┌──────────────────┐ ┌──────────────────┐
│ vm_api.py :8098 │ JSON API (React SPA) │ vm_web.py :8099 │
│ (FastAPI) │ ◀── React frontend ──┐ │ (FastAPI) │
└────────┬─────────┘ │ │ /audio/ legacy │
│ Apache /api/ proxy │ └────────┬─────────┘
▼ │ │ Apache /audio/ proxy
Apache (TLS) ◀─────────────────────────┘ ▼
│ Apache (TLS)
▼
https://vm.txt3.net
(React SPA + JSON API; audio proxied to :8099)
```
**Delivery order is deliberate**: email first, then the database, then Telegram.
@ -70,6 +63,13 @@ Each later stage is wrapped so a failure only logs. If anything throws at the
top level, the *original* Asterisk notification is relayed unchanged. A
voicemail notification is never lost because a summariser or an API failed.
**Two backends, one MySQL store.** The *interactive* portal is a React single
page app (built from `/home/jp/Work/voicemail-ui`, served as static files by
Apache) that talks to a JSON API (`vm_api.py` on :8098). The original
server-rendered FastAPI app (`vm_web.py` on :8099) still runs but now only
serves `/audio/` playback (proxied by Apache). Both backends read/write the
same MySQL `asterisk` database.
---
## Components
@ -77,13 +77,18 @@ voicemail notification is never lost because a summariser or an API failed.
| File | Role |
|---|---|
| `src/vm_mailcmd.py` | The `mailcmd` — entry point for every voicemail. Orchestrates everything. |
| `src/vm_store.py` | SQLite store: schema, settings, sessions, content-addressed audio. |
| `src/vm_telegram.py` | Telegram delivery, per-mailbox routing. |
| `src/vm_contacts.py` | Caller-ID → name via local export, Google People API, or CardDAV. |
| `src/vm_web.py` | FastAPI portal: login, list, play, delete, settings. |
| `src/vm_store.py` | Store. **MySQL** (PyMySQL) by default; SQLite fallback only if no `db_secret`/MySQL env. |
| `src/vm_api.py` | FastAPI **JSON API** (:8098) for the React portal. Login, messages, settings, contacts CRUD. |
| `src/vm_web.py` | FastAPI HTML portal (:8099). Now primarily serves `/audio/` playback + `/healthz`. |
| `src/vm_contacts.py` | Caller-ID → name/email **from the MySQL `contacts` table** (MySQL-only). |
| `src/vm_auth.py` | Parses `voicemail.conf` so users log in with their phone PIN. |
| `src/vm_telegram.py` | Telegram delivery, per-mailbox routing. |
| `src/vm_import.py` | Backfills existing spool recordings into the database. |
| `src/vm_backfill_cdr.py` | Imports Asterisk `Master.csv` CDR history into MySQL. |
| `src/vm_migrate_sqlite_to_mysql.py` | One-off migration of the legacy SQLite store to MySQL. |
| `src/vm_import_contacts.py` | Imports a vCard/CSV export into the MySQL `contacts` table. |
| `src/vm_tg_setup.py` | Helper: discover Telegram chat IDs, test a route. |
| `frontend/` | Synced copy of the React frontend (`/home/jp/Work/voicemail-ui`). |
---
@ -92,19 +97,28 @@ voicemail notification is never lost because a summariser or an API failed.
See **[docs/INSTALL.md](docs/INSTALL.md)** for the full walkthrough. Short version:
```bash
# 1. Backend services (API + legacy audio server)
sudo scripts/install.sh # venv, deps, model cache, mailcmd wiring
sudo cp systemd/vm-api.service /etc/systemd/system/
sudo cp systemd/vm-portal.service /etc/systemd/system/
sudo systemctl enable --now vm-portal
# then follow docs/INSTALL.md §5 for the Apache vhost + TLS
sudo systemctl daemon-reload
sudo systemctl enable --now vm-api vm-portal
# 2. React frontend (build in /home/jp/Work/voicemail-ui, rsync here)
cd /home/jp/Work/voicemail-ui && npm install && npm run build
sudo -A rsync -a --exclude node_modules --exclude dist \
/home/jp/Work/voicemail-ui/ /home/txt3/domains/vm.txt3.net/public_html/
# 3. Apache vhost + TLS — see docs/INSTALL.md §5
```
Backfill your existing voicemails:
Backfill your existing voicemails and CDR:
```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
/opt/vm-transcribe/vm_import.py --dry-run # preview
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
/opt/vm-transcribe/vm_import.py # do it
V=/opt/vm-transcribe/venv/bin/python3
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --dry-run # preview
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py # do it
sudo -u asterisk $V /opt/vm-transcribe/vm_backfill_cdr.py # CDR history
```
---
@ -113,10 +127,10 @@ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
| Document | Contents |
|---|---|
| [docs/INSTALL.md](docs/INSTALL.md) | Step-by-step install, including TLS ordering |
| [docs/CONFIGURATION.md](docs/CONFIGURATION.md) | Every config file and option |
| [docs/INSTALL.md](docs/INSTALL.md) | Step-by-step install, MySQL setup, TLS ordering, frontend build |
| [docs/CONFIGURATION.md](docs/CONFIGURATION.md) | MySQL credentials, env vars, every config file and option |
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Design decisions and why |
| [docs/OPERATIONS.md](docs/OPERATIONS.md) | Day-to-day running, backup, troubleshooting |
| [docs/OPERATIONS.md](docs/OPERATIONS.md) | Day-to-day running, backup (incl. MySQL), troubleshooting |
| [docs/SECURITY.md](docs/SECURITY.md) | Threat model, hardening, privacy |
| [docs/TESTING.md](docs/TESTING.md) | How to verify each part |
| [CHANGELOG.md](CHANGELOG.md) | Version history |
@ -126,11 +140,13 @@ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
## Requirements
- Asterisk with `app_voicemail` (file-based spool, not ODBC/IMAP storage)
- Python 3.9+
- Python 3.9+ with `pymysql` (backend now MySQL-first)
- MySQL/MariaDB (the same `asterisk` DB Asterisk's CDR uses)
- `ffmpeg` (Telegram voice-note transcoding; `sox` optionally for gsm)
- An MTA listening on `localhost:25` (Postfix here)
- ~200 MB disk for the whisper `base.en` model
- Apache with `proxy`, `proxy_http`, `headers`, `rewrite`, `ssl` for the portal
- Node 18+ + npm for the React frontend build
Runs comfortably on 4 CPU cores with no GPU: ~8 s to transcribe 23 s of audio.
@ -148,12 +164,16 @@ keeps voicemail content on your own hardware. `summarise()` in
pointing at the wrong recording. Recordings are copied to
`audio/<sha[:2]>/<sha>.wav`.
**App passwords cannot read Google Contacts.** Google disabled basic auth for
CardDAV/CalDAV/IMAP/SMTP/POP on 2024-09-30. Use the `file` backend (a vCard
export) or the `google` backend (OAuth). The `carddav` backend with an app
password works for Nextcloud, Fastmail and iCloud.
**Contacts live in MySQL.** `vm_contacts.py` resolves caller IDs exclusively
from the `asterisk.contacts` table (E.164 canonical; `+44`↔`0` normalised so
both forms match). Creating/updating a contact backfills matching
`messages.contact_name`/`contact_email`. The older `file` / `google` / `carddav`
backends were removed in favour of the MySQL table.
**The portal ships zero JavaScript**, which lets its CSP be `script-src 'none'`.
**The portal ships JavaScript now.** The interactive UI is a React SPA, so the
Apache CSP is relaxed to `script-src 'self' 'unsafe-inline'` (vs the old
zero-JS `script-src 'none'`). See [SECURITY.md](docs/SECURITY.md) for the
updated threat model.
---