jp
b8c60605fd
README, docs/{ARCHITECTURE,CONFIGURATION,INSTALL,OPERATIONS,SECURITY,TESTING}
and CHANGELOG were stale: they described the original SQLite + CardDAV +
zero-JS vm_web portal. Brought them in line with the actual code:
- MySQL (PyMySQL) as the default store; SQLite fallback only
- vm_contacts resolves from the MySQL contacts table (file/google/carddav removed)
- React SPA frontend + vm_api.py JSON API (:8098); vm_web.py now serves /audio/
- Apache vhost proxies /api/ -> :8098, /audio/ -> :8099, serves the SPA
- CSP relaxed to script-src 'self' 'unsafe-inline' (UI is now JavaScript)
- Added CHANGELOG 1.2.0 entry for the React/vm_api frontend work
183 lines
9.5 KiB
Markdown
183 lines
9.5 KiB
Markdown
# Asterisk Voicemail Transcription & Portal
|
|
|
|
Turns Asterisk voicemail into something you can actually read, search and
|
|
manage:
|
|
|
|
- **Transcribes** each recording locally with [faster-whisper](https://github.com/SYSTRAN/faster-whisper) (CPU, no cloud, no API key)
|
|
- **Summarises** it, tags intent (callback requested, urgent, invoice…) and
|
|
extracts callback numbers — including spoken-out digits
|
|
- **Emails** a graphically designed `multipart/alternative` notification
|
|
(plain + HTML) with the recording attached
|
|
- **DMs Telegram** optionally, as a playable voice note with the summary
|
|
- **Resolves caller ID to a name** from the MySQL `contacts` table
|
|
- **Serves a React web portal** where mailbox users log in with their existing
|
|
phone PIN to read transcripts, play/download recordings, delete messages,
|
|
manage contacts and look up unknown numbers
|
|
|
|
Everything runs on the PBX host. No third-party service sees your voicemail.
|
|
|
|
---
|
|
|
|
## Architecture (current)
|
|
|
|
```
|
|
incoming call
|
|
│
|
|
▼
|
|
┌─────────────┐ voicemail.conf: mailcmd=… vm_mailcmd.py
|
|
│ Asterisk │ ─────────────────────────────┐
|
|
└─────────────┘ pipes an RFC822 message │
|
|
(notification + audio) ▼
|
|
┌───────────────┐
|
|
│ vm_mailcmd.py │
|
|
└───────┬───────┘
|
|
┌──────────────┬──────────────┬────┴─────────┬──────────────┐
|
|
▼ ▼ ▼ ▼ ▼
|
|
faster-whisper summarise() vm_contacts vm_store vm_telegram
|
|
(transcribe) + intents (MySQL name) (MySQL + (voice note
|
|
+ numbers audio CAS) DM)
|
|
│ │
|
|
▼ ▼
|
|
multipart email ──▶ Postfix :25 ┌──────────────────────────┐
|
|
│ MySQL (asterisk DB) │
|
|
│ messages, contacts, CDR │
|
|
└──────┬───────────┬─────────┘
|
|
│ │
|
|
┌──────────────────────┘ └──────────────────────┐
|
|
▼ ▼
|
|
┌──────────────────┐ ┌──────────────────┐
|
|
│ vm_api.py :8098 │ JSON API (React SPA) │ vm_web.py :8099 │
|
|
│ (FastAPI) │ ◀── React frontend ──┐ │ (FastAPI) │
|
|
└────────┬─────────┘ │ │ /audio/ legacy │
|
|
│ Apache /api/ proxy │ └────────┬─────────┘
|
|
▼ │ │ Apache /audio/ proxy
|
|
Apache (TLS) ◀─────────────────────────┘ ▼
|
|
│ Apache (TLS)
|
|
▼
|
|
https://vm.txt3.net
|
|
(React SPA + JSON API; audio proxied to :8099)
|
|
```
|
|
|
|
**Delivery order is deliberate**: email first, then the database, then Telegram.
|
|
Each later stage is wrapped so a failure only logs. If anything throws at the
|
|
top level, the *original* Asterisk notification is relayed unchanged. A
|
|
voicemail notification is never lost because a summariser or an API failed.
|
|
|
|
**Two backends, one MySQL store.** The *interactive* portal is a React single
|
|
page app (built from `/home/jp/Work/voicemail-ui`, served as static files by
|
|
Apache) that talks to a JSON API (`vm_api.py` on :8098). The original
|
|
server-rendered FastAPI app (`vm_web.py` on :8099) still runs but now only
|
|
serves `/audio/` playback (proxied by Apache). Both backends read/write the
|
|
same MySQL `asterisk` database.
|
|
|
|
---
|
|
|
|
## Components
|
|
|
|
| File | Role |
|
|
|---|---|
|
|
| `src/vm_mailcmd.py` | The `mailcmd` — entry point for every voicemail. Orchestrates everything. |
|
|
| `src/vm_store.py` | Store. **MySQL** (PyMySQL) by default; SQLite fallback only if no `db_secret`/MySQL env. |
|
|
| `src/vm_api.py` | FastAPI **JSON API** (:8098) for the React portal. Login, messages, settings, contacts CRUD. |
|
|
| `src/vm_web.py` | FastAPI HTML portal (:8099). Now primarily serves `/audio/` playback + `/healthz`. |
|
|
| `src/vm_contacts.py` | Caller-ID → name/email **from the MySQL `contacts` table** (MySQL-only). |
|
|
| `src/vm_auth.py` | Parses `voicemail.conf` so users log in with their phone PIN. |
|
|
| `src/vm_telegram.py` | Telegram delivery, per-mailbox routing. |
|
|
| `src/vm_import.py` | Backfills existing spool recordings into the database. |
|
|
| `src/vm_backfill_cdr.py` | Imports Asterisk `Master.csv` CDR history into MySQL. |
|
|
| `src/vm_migrate_sqlite_to_mysql.py` | One-off migration of the legacy SQLite store to MySQL. |
|
|
| `src/vm_import_contacts.py` | Imports a vCard/CSV export into the MySQL `contacts` table. |
|
|
| `src/vm_tg_setup.py` | Helper: discover Telegram chat IDs, test a route. |
|
|
| `frontend/` | Synced copy of the React frontend (`/home/jp/Work/voicemail-ui`). |
|
|
|
|
---
|
|
|
|
## Install
|
|
|
|
See **[docs/INSTALL.md](docs/INSTALL.md)** for the full walkthrough. Short version:
|
|
|
|
```bash
|
|
# 1. Backend services (API + legacy audio server)
|
|
sudo scripts/install.sh # venv, deps, model cache, mailcmd wiring
|
|
sudo cp systemd/vm-api.service /etc/systemd/system/
|
|
sudo cp systemd/vm-portal.service /etc/systemd/system/
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now vm-api vm-portal
|
|
|
|
# 2. React frontend (build in /home/jp/Work/voicemail-ui, rsync here)
|
|
cd /home/jp/Work/voicemail-ui && npm install && npm run build
|
|
sudo -A rsync -a --exclude node_modules --exclude dist \
|
|
/home/jp/Work/voicemail-ui/ /home/txt3/domains/vm.txt3.net/public_html/
|
|
|
|
# 3. Apache vhost + TLS — see docs/INSTALL.md §5
|
|
```
|
|
|
|
Backfill your existing voicemails and CDR:
|
|
|
|
```bash
|
|
V=/opt/vm-transcribe/venv/bin/python3
|
|
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --dry-run # preview
|
|
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py # do it
|
|
sudo -u asterisk $V /opt/vm-transcribe/vm_backfill_cdr.py # CDR history
|
|
```
|
|
|
|
---
|
|
|
|
## Documentation
|
|
|
|
| Document | Contents |
|
|
|---|---|
|
|
| [docs/INSTALL.md](docs/INSTALL.md) | Step-by-step install, MySQL setup, TLS ordering, frontend build |
|
|
| [docs/CONFIGURATION.md](docs/CONFIGURATION.md) | MySQL credentials, env vars, every config file and option |
|
|
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Design decisions and why |
|
|
| [docs/OPERATIONS.md](docs/OPERATIONS.md) | Day-to-day running, backup (incl. MySQL), troubleshooting |
|
|
| [docs/SECURITY.md](docs/SECURITY.md) | Threat model, hardening, privacy |
|
|
| [docs/TESTING.md](docs/TESTING.md) | How to verify each part |
|
|
| [CHANGELOG.md](CHANGELOG.md) | Version history |
|
|
|
|
---
|
|
|
|
## Requirements
|
|
|
|
- Asterisk with `app_voicemail` (file-based spool, not ODBC/IMAP storage)
|
|
- Python 3.9+ with `pymysql` (backend now MySQL-first)
|
|
- MySQL/MariaDB (the same `asterisk` DB Asterisk's CDR uses)
|
|
- `ffmpeg` (Telegram voice-note transcoding; `sox` optionally for gsm)
|
|
- An MTA listening on `localhost:25` (Postfix here)
|
|
- ~200 MB disk for the whisper `base.en` model
|
|
- Apache with `proxy`, `proxy_http`, `headers`, `rewrite`, `ssl` for the portal
|
|
- Node 18+ + npm for the React frontend build
|
|
|
|
Runs comfortably on 4 CPU cores with no GPU: ~8 s to transcribe 23 s of audio.
|
|
|
|
---
|
|
|
|
## Design notes worth knowing
|
|
|
|
**Summarisation is local and extractive.** Frequency-scored sentence selection
|
|
with position/digit weighting, plus regex intent tags. No LLM, by choice — it
|
|
keeps voicemail content on your own hardware. `summarise()` in
|
|
`vm_mailcmd.py` is a single swap-in point if you want an abstractive model.
|
|
|
|
**Audio is content-addressed, not referenced by spool path.** Asterisk renumbers
|
|
`msgNNNN` files when a message is deleted, so a stored path silently starts
|
|
pointing at the wrong recording. Recordings are copied to
|
|
`audio/<sha[:2]>/<sha>.wav`.
|
|
|
|
**Contacts live in MySQL.** `vm_contacts.py` resolves caller IDs exclusively
|
|
from the `asterisk.contacts` table (E.164 canonical; `+44`↔`0` normalised so
|
|
both forms match). Creating/updating a contact backfills matching
|
|
`messages.contact_name`/`contact_email`. The older `file` / `google` / `carddav`
|
|
backends were removed in favour of the MySQL table.
|
|
|
|
**The portal ships JavaScript now.** The interactive UI is a React SPA, so the
|
|
Apache CSP is relaxed to `script-src 'self' 'unsafe-inline'` (vs the old
|
|
zero-JS `script-src 'none'`). See [SECURITY.md](docs/SECURITY.md) for the
|
|
updated threat model.
|
|
|
|
---
|
|
|
|
## Licence
|
|
|
|
MIT — see [LICENSE](LICENSE).
|