jp/asterisk-voicemail
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update all documentation to current MySQL + React SPA + vm_api.py architecture

Browse Source 
README, docs/{ARCHITECTURE,CONFIGURATION,INSTALL,OPERATIONS,SECURITY,TESTING}
and CHANGELOG were stale: they described the original SQLite + CardDAV +
zero-JS vm_web portal. Brought them in line with the actual code:
- MySQL (PyMySQL) as the default store; SQLite fallback only
- vm_contacts resolves from the MySQL contacts table (file/google/carddav removed)
- React SPA frontend + vm_api.py JSON API (:8098); vm_web.py now serves /audio/
- Apache vhost proxies /api/ -> :8098, /audio/ -> :8099, serves the SPA
- CSP relaxed to script-src 'self' 'unsafe-inline' (UI is now JavaScript)
- Added CHANGELOG 1.2.0 entry for the React/vm_api frontend work
This commit is contained in:
jp
2026-08-13 20:20:39 +01:00
parent 3155020f67
commit b8c60605fd
8 changed files with 553 additions and 290 deletions
Download Patch File Download Diff File Expand all files Collapse all files

199
docs/INSTALL.md
View File

@ -1,11 +1,16 @@
# Installation
Written against Debian 12 + Asterisk 20 + Apache 2.4 + Postfix, running under
Virtualmin. Adjust paths if your layout differs.
Written against Debian 12 + Asterisk 20 + Apache 2.4 + Postfix + MySQL/MariaDB,
running under Virtualmin. Adjust paths if your layout differs.
Everything installs to `/opt/vm-transcribe` with data in
Everything installs to `/opt/vm-transcribe` with audio in
`/var/lib/vm-transcribe`. Nothing is installed into system Python.
> **Frontend lives in a separate repo.** This backend repo ships a *synced
> copy* under `frontend/`. Build the real app in `/home/jp/Work/voicemail-ui`
> and rsync the result to the web root (see §6). Keep the two in sync with
> `rsync -a /home/jp/Work/voicemail-ui/ frontend/` then commit.
---
## 0. Prerequisites
@ -16,7 +21,9 @@ grep -E '^(odbcstorage|imapserver)' /etc/asterisk/voicemail.conf # expect noth
# tools
sudo apt install ffmpeg sox python3-venv # sox optional (gsm)
sudo apt install default-mysql-server # or mariadb-server
which certbot # for the portal's TLS
node --version && npm --version # Node 18+ for the frontend
# an MTA on localhost:25
sudo ss -ltnp | grep ':25 '
@ -32,25 +39,52 @@ df -h /var
---
## 1. Core pipeline
## 1. MySQL (storage + contacts + CDR)
The backend shares the Asterisk `asterisk` database. Create a least-privilege
user and a credentials file:
```bash
sudo mysql -e "CREATE USER IF NOT EXISTS 'asterisk'@'localhost' IDENTIFIED BY 'PICK_A_STRONG_PASSWORD';"
sudo mysql -e "GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,INDEX,ALTER ON asterisk.* TO 'asterisk'@'localhost';"
sudo mysql -e "FLUSH PRIVILEGES;"
```
Write `/opt/vm-transcribe/db_secret` (mode `640 root:root`):
```bash
sudo install -o root -g root -m 640 /dev/null /opt/vm-transcribe/db_secret
sudo tee /opt/vm-transcribe/db_secret >/dev/null <<'EOF'
MYSQL_HOST=localhost
MYSQL_USER=asterisk
MYSQL_PASSWORD=PICK_A_STRONG_PASSWORD
MYSQL_DB=asterisk
EOF
```
`vm_store` uses MySQL whenever this file (or `VM_MYSQL_*` env vars) exists;
otherwise it falls back to SQLite. The schema is created on first connect.
(Optional) To load call records into the same DB, wire Asterisk CDR via
`cdr_adaptive_odbc` + `res_odbc` + the MariaDB ODBC driver, then run
`src/vm_backfill_cdr.py` to import `Master.csv` history.
---
## 2. Core pipeline (mailcmd + venv)
The `install.sh` script builds the venv, installs `faster-whisper`, deploys the
mailcmd scripts, seeds `telegram.conf`, pre-caches the whisper model **as the
asterisk user**, and rewrites `voicemail.conf`'s `mailcmd=`.
> `install.sh` reads sources from `/home/jp/asterisk-vm` (the deploy path in
> AGENTS.md). Either symlink/clobber that path to this repo, or edit the `SRC=`
> line at the top of `scripts/install.sh` before running.
```bash
sudo scripts/install.sh
```
That script:
1. creates `/opt/vm-transcribe` and `/opt/vm-transcribe/models` owned by `asterisk`
2. builds a venv and installs `faster-whisper`
3. installs the Python modules
4. seeds `telegram.conf` / `contacts.conf` if absent (never overwrites)
5. creates `/var/log/asterisk/vm_mailcmd.log`
6. **pre-downloads the whisper model as the `asterisk` user** — do not skip
this, or the first real voicemail pays a ~150 MB download while the caller
waits for their notification
7. backs up `voicemail.conf` and rewrites `mailcmd=`
8. reloads Asterisk
Verify the wiring:
```bash
@ -64,7 +98,7 @@ Build a fake notification and pipe it through **as the `asterisk` user** —
testing as yourself hides permission problems:
```bash
cd /path/to/repo
cd /home/jp/Work/asterisk-voicemail
python3 tests/make_test_mail.py /path/to/some.wav you@example.com > /tmp/t.eml
sudo chmod 644 /tmp/t.eml
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
@ -76,7 +110,7 @@ You should see `transcribed … chars` then `sent enriched notification to …`.
Confirm the mail actually left:
```bash
sudo grep "to=<you@example.com>" /var/log/mail.log | tail -1 # expect status=sent
sudo grep "to=<you@example.com>" /var/log/mail.log | tail -1 # status=sent
```
Also verify the fail-safe — a message with **no** audio must relay unchanged
@ -92,7 +126,7 @@ sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
---
## 2. Telegram (optional)
## 3. Telegram (optional)
1. Create a bot: message [@BotFather](https://t.me/BotFather) → `/newbot` → copy the token.
2. Put it in `/opt/vm-transcribe/telegram.conf` (`sudo`, mode 640 root:asterisk).
@ -111,95 +145,135 @@ per-mailbox routing.
---
## 3. Contact lookup (optional)
## 4. Contacts (MySQL)
Simplest and most reliable — a local export, no tokens, no rate limits:
Contacts live in the MySQL `contacts` table — there is no file/Google/CardDAV
backend anymore. Load your address book from a vCard/CSV export:
1. contacts.google.com → **Export** → **vCard**
2. `sudo install -o asterisk -g asterisk -m 640 contacts.vcf /var/lib/vm-transcribe/contacts.vcf`
3. Confirm `contacts.conf` has `backends = file` and the matching `path`.
```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
/opt/vm-transcribe/vm_import_contacts.py /path/to/contacts.vcf
```
Confirm lookup works:
```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
/opt/vm-transcribe/vm_contacts.py '<07700900123>'
```
> **Google app passwords do not work for contacts.** Google disabled basic auth
> for CardDAV/CalDAV/IMAP/SMTP/POP on 2024-09-30. Use the `file` or `google`
> (OAuth) backend. The `carddav` backend is for Nextcloud/Fastmail/iCloud.
---
## 4. Portal service
## 5. Backend services
Install both units (API + legacy portal/audio):
```bash
sudo install -d -o asterisk -g asterisk -m 750 \
/var/lib/vm-transcribe /var/lib/vm-transcribe/audio
sudo /opt/vm-transcribe/venv/bin/pip install fastapi 'uvicorn[standard]' python-multipart
sudo /opt/vm-transcribe/venv/bin/pip install fastapi 'uvicorn[standard]' python-multipart pymysql
# API unit reads MySQL creds from api.env
sudo install -o root -g root -m 640 /opt/vm-transcribe/db_secret /opt/vm-transcribe/api.env
sudo install -o root -g root -m 644 systemd/vm-api.service /etc/systemd/system/
sudo install -o root -g root -m 644 systemd/vm-portal.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now vm-portal
sudo systemctl is-active vm-portal
curl -s http://127.0.0.1:8099/healthz # {"ok":true,"messages":N}
sudo systemctl enable --now vm-api vm-portal
sudo systemctl is-active vm-api vm-portal
curl -s http://127.0.0.1:8098/api/healthz # {"ok":true,...}
curl -s http://127.0.0.1:8099/healthz # {"ok":true,"messages":N}
```
The service runs as `asterisk` (so it can read `voicemail.conf` and the spool)
and binds **loopback only** — `IPAddressAllow=localhost` means Apache is the
The services run as `asterisk` (so they can read `voicemail.conf` and the spool)
and bind **loopback only** — `IPAddressAllow=localhost` means Apache is the
only possible client.
---
## 5. Apache vhost + TLS
## 6. React frontend (build + deploy)
The interactive UI is a React SPA built in `/home/jp/Work/voicemail-ui`:
```bash
cd /home/jp/Work/voicemail-ui
npm install
npm run build # outputs dist/
```
Deploy the static build to the web root (Apache serves it and proxies `/api/`
to :8098 and `/audio/` to :8099):
```bash
sudo -A rsync -a --exclude node_modules --exclude dist \
/home/jp/Work/voicemail-ui/ /home/txt3/domains/vm.txt3.net/public_html/
```
Keep this repo's copy in sync for documentation/commit purposes:
```bash
rsync -a /home/jp/Work/voicemail-ui/ frontend/ # then git commit under frontend/
```
---
## 7. Apache vhost + TLS
**Order matters.** Installing a vhost that references a cert which does not yet
exist breaks `apache2ctl configtest`, which blocks reloads for *every* site on
the box. So: DNS → cert → full vhost.
### 5a. DNS
### 7a. DNS
```bash
dig +short vm.example.com A # must return your ORIGIN ip
dig +short vm.txt3.net A # must return your ORIGIN ip
```
Behind Cloudflare, set the record to **DNS only (grey cloud)** first, or the
HTTP-01 challenge is intercepted by the edge.
### 5b. Minimal :80 vhost, so certbot can answer the challenge
### 7b. Minimal :80 vhost, so certbot can answer the challenge
```bash
sudo install -d -o www-data -g www-data -m 755 /var/www/vm.example.com/public_html
sudo cp apache/vm.txt3.net-step1.conf /etc/apache2/sites-available/vm.example.com.conf
sudo install -d -o www-data -g www-data -m 755 /var/www/vm.txt3.net/public_html
sudo cp apache/vm.txt3.net-step1.conf /etc/apache2/sites-available/vm.txt3.net.conf
# edit ServerName / paths / IPs to match your host
sudo a2ensite vm.example.com
sudo a2ensite vm.txt3.net
sudo apache2ctl configtest && sudo systemctl reload apache2
```
### 5c. Issue the certificate
### 7c. Issue the certificate
```bash
sudo certbot certonly --webroot -w /var/www/vm.example.com/public_html \
-d vm.example.com --cert-name vm.example.com
sudo certbot certonly --webroot -w /var/www/vm.txt3.net/public_html \
-d vm.txt3.net --cert-name vm.txt3.net
```
### 5d. Full vhost — HTTPS + reverse proxy
### 7d. Full vhost — HTTPS + reverse proxy
```bash
sudo a2enmod proxy proxy_http headers rewrite ssl
sudo cp apache/vm.txt3.net.conf /etc/apache2/sites-available/vm.example.com.conf
# edit ServerName, cert paths, IPs
sudo install -d /var/www/vm.example.com/public_html/.well-known/acme-challenge
sudo cp apache/vm.txt3.net.conf /etc/apache2/sites-available/vm.txt3.net.conf
# edit ServerName, cert paths, IPs, DocumentRoot
sudo install -d /var/www/vm.txt3.net/public_html/.well-known/acme-challenge
sudo apache2ctl configtest && sudo systemctl reload apache2
```
### 5e. Verify through the real hostname
The shipped `apache/vm.txt3.net.conf` already proxies `/api/` → :8098,
`/audio/` → :8099, serves the React `dist` as `DocumentRoot`, and adds an SPA
fallback (`RewriteRule ^ /index.html`).
### 7e. Verify through the real hostname
Not localhost — the whole point is to exercise Apache, TLS and the proxy:
```bash
R="--resolve vm.example.com:443:YOUR.ORIGIN.IP"
curl -s $R https://vm.example.com/healthz
curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.example.com/login
curl -s $R -D- -o /dev/null https://vm.example.com/login | grep -i 'strict-transport\|content-security'
R="--resolve vm.txt3.net:443:ORIGIN.IP"
curl -s $R https://vm.txt3.net/api/healthz
curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.txt3.net/login
curl -s $R -D- -o /dev/null https://vm.txt3.net/login | grep -iE 'strict-transport|content-security|x-frame'
# SPA fallback
curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.txt3.net/some/route
```
Behind a CDN, `curl https://host/` tests the *CDN*, not your origin. Always
@ -207,13 +281,14 @@ Behind a CDN, `curl https://host/` tests the *CDN*, not your origin. Always
---
## 6. Backfill existing voicemails
## 8. Backfill existing voicemails
```bash
V=/opt/vm-transcribe/venv/bin/python3
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --dry-run # preview
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --limit 5 # trial
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py # all
sudo -u asterisk $V /opt/vm-transcribe/vm_backfill_cdr.py # CDR history
```
Idempotent — safe to re-run; it resumes rather than duplicating. Expect a
@ -224,9 +299,9 @@ on 4 cores.
---
## 7. Log in
## 9. Log in
Browse to `https://vm.example.com/` and log in with a **mailbox number and its
Browse to `https://vm.txt3.net/` and log in with a **mailbox number and its
existing voicemail PIN** from `voicemail.conf`. No new passwords are created.
---
@ -234,11 +309,13 @@ existing voicemail PIN** from `voicemail.conf`. No new passwords are created.
## Uninstall
```bash
sudo systemctl disable --now vm-portal
sudo rm /etc/systemd/system/vm-portal.service && sudo systemctl daemon-reload
sudo a2dissite vm.example.com && sudo systemctl reload apache2
sudo systemctl disable --now vm-api vm-portal
sudo rm /etc/systemd/system/vm-api.service /etc/systemd/system/vm-portal.service
sudo systemctl daemon-reload
sudo a2dissite vm.txt3.net && sudo systemctl reload apache2
# restore the original mailcmd
sudo cp /etc/asterisk/voicemail.conf.bak-<timestamp> /etc/asterisk/voicemail.conf
sudo asterisk -rx 'voicemail reload'
sudo rm -rf /opt/vm-transcribe # keep /var/lib/vm-transcribe for the data
# MySQL: DROP USER 'asterisk'@'localhost'; (optionally DROP DATABASE asterisk;)
```