jp
857284abbf
- mailcmd replacement (vm_mailcmd.py): faster-whisper transcription (CPU int8), extractive summary + intent tags + spoken-digit number extraction, multipart/alternative HTML email, fail-safe relay of original message - Telegram DM delivery (vm_telegram.py) with per-mailbox routing - Caller-ID -> name (vm_contacts.py): file / google / carddav backends - SQLite store (vm_store.py) with content-addressed audio - FastAPI portal (vm_web.py): PIN login, list/play/delete, per-user settings, zero JS, loopback-only behind Apache TLS - Backfill importer (vm_import.py) for existing spool recordings - systemd unit, Apache vhost + certbot TLS, install.sh - Docs: INSTALL, CONFIGURATION, ARCHITECTURE, OPERATIONS, SECURITY, TESTING Verified end-to-end on mail.txt3.net: 157 historical messages backfilled, live voicemail -> transcribed -> stored -> visible at https://vm.txt3.net.
245 lines
7.7 KiB
Markdown
245 lines
7.7 KiB
Markdown
# Installation
|
|
|
|
Written against Debian 12 + Asterisk 20 + Apache 2.4 + Postfix, running under
|
|
Virtualmin. Adjust paths if your layout differs.
|
|
|
|
Everything installs to `/opt/vm-transcribe` with data in
|
|
`/var/lib/vm-transcribe`. Nothing is installed into system Python.
|
|
|
|
---
|
|
|
|
## 0. Prerequisites
|
|
|
|
```bash
|
|
# Asterisk voicemail must use file-based storage (the default), not ODBC/IMAP
|
|
grep -E '^(odbcstorage|imapserver)' /etc/asterisk/voicemail.conf # expect nothing
|
|
|
|
# tools
|
|
sudo apt install ffmpeg sox python3-venv # sox optional (gsm)
|
|
which certbot # for the portal's TLS
|
|
|
|
# an MTA on localhost:25
|
|
sudo ss -ltnp | grep ':25 '
|
|
```
|
|
|
|
**Check disk space before you start.** A full `/var` makes Postfix reject all
|
|
mail with `452 4.3.1 Insufficient system storage`, which looks like a bug in
|
|
this pipeline but is not:
|
|
|
|
```bash
|
|
df -h /var
|
|
```
|
|
|
|
---
|
|
|
|
## 1. Core pipeline
|
|
|
|
```bash
|
|
sudo scripts/install.sh
|
|
```
|
|
|
|
That script:
|
|
|
|
1. creates `/opt/vm-transcribe` and `/opt/vm-transcribe/models` owned by `asterisk`
|
|
2. builds a venv and installs `faster-whisper`
|
|
3. installs the Python modules
|
|
4. seeds `telegram.conf` / `contacts.conf` if absent (never overwrites)
|
|
5. creates `/var/log/asterisk/vm_mailcmd.log`
|
|
6. **pre-downloads the whisper model as the `asterisk` user** — do not skip
|
|
this, or the first real voicemail pays a ~150 MB download while the caller
|
|
waits for their notification
|
|
7. backs up `voicemail.conf` and rewrites `mailcmd=`
|
|
8. reloads Asterisk
|
|
|
|
Verify the wiring:
|
|
|
|
```bash
|
|
grep '^mailcmd' /etc/asterisk/voicemail.conf
|
|
# mailcmd=/opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_mailcmd.py
|
|
```
|
|
|
|
### Test it before trusting it
|
|
|
|
Build a fake notification and pipe it through **as the `asterisk` user** —
|
|
testing as yourself hides permission problems:
|
|
|
|
```bash
|
|
cd /path/to/repo
|
|
python3 tests/make_test_mail.py /path/to/some.wav you@example.com > /tmp/t.eml
|
|
sudo chmod 644 /tmp/t.eml
|
|
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
|
/opt/vm-transcribe/vm_mailcmd.py < /tmp/t.eml
|
|
sudo tail -5 /var/log/asterisk/vm_mailcmd.log
|
|
```
|
|
|
|
You should see `transcribed … chars` then `sent enriched notification to …`.
|
|
Confirm the mail actually left:
|
|
|
|
```bash
|
|
sudo grep "to=<you@example.com>" /var/log/mail.log | tail -1 # expect status=sent
|
|
```
|
|
|
|
Also verify the fail-safe — a message with **no** audio must relay unchanged
|
|
rather than erroring:
|
|
|
|
```bash
|
|
printf 'From: a@b\nTo: you@example.com\nSubject: no audio\n\nbody\n' > /tmp/n.eml
|
|
sudo chmod 644 /tmp/n.eml
|
|
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
|
/opt/vm-transcribe/vm_mailcmd.py < /tmp/n.eml
|
|
# log: "no audio attachment; relaying original"
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Telegram (optional)
|
|
|
|
1. Create a bot: message [@BotFather](https://t.me/BotFather) → `/newbot` → copy the token.
|
|
2. Put it in `/opt/vm-transcribe/telegram.conf` (`sudo`, mode 640 root:asterisk).
|
|
3. **Each recipient must message the bot once** (`/start`) — Telegram forbids
|
|
bots from initiating conversations.
|
|
4. Discover chat IDs and test:
|
|
|
|
```bash
|
|
sudo /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py ids
|
|
sudo /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py test 1001
|
|
```
|
|
|
|
The `test` subcommand sends a real DM with a playable voice note and exits
|
|
non-zero on failure. See [CONFIGURATION.md](CONFIGURATION.md#telegramconf) for
|
|
per-mailbox routing.
|
|
|
|
---
|
|
|
|
## 3. Contact lookup (optional)
|
|
|
|
Simplest and most reliable — a local export, no tokens, no rate limits:
|
|
|
|
1. contacts.google.com → **Export** → **vCard**
|
|
2. `sudo install -o asterisk -g asterisk -m 640 contacts.vcf /var/lib/vm-transcribe/contacts.vcf`
|
|
3. Confirm `contacts.conf` has `backends = file` and the matching `path`.
|
|
|
|
```bash
|
|
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
|
/opt/vm-transcribe/vm_contacts.py '<07700900123>'
|
|
```
|
|
|
|
> **Google app passwords do not work for contacts.** Google disabled basic auth
|
|
> for CardDAV/CalDAV/IMAP/SMTP/POP on 2024-09-30. Use the `file` or `google`
|
|
> (OAuth) backend. The `carddav` backend is for Nextcloud/Fastmail/iCloud.
|
|
|
|
---
|
|
|
|
## 4. Portal service
|
|
|
|
```bash
|
|
sudo install -d -o asterisk -g asterisk -m 750 \
|
|
/var/lib/vm-transcribe /var/lib/vm-transcribe/audio
|
|
sudo /opt/vm-transcribe/venv/bin/pip install fastapi 'uvicorn[standard]' python-multipart
|
|
sudo install -o root -g root -m 644 systemd/vm-portal.service /etc/systemd/system/
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now vm-portal
|
|
sudo systemctl is-active vm-portal
|
|
curl -s http://127.0.0.1:8099/healthz # {"ok":true,"messages":N}
|
|
```
|
|
|
|
The service runs as `asterisk` (so it can read `voicemail.conf` and the spool)
|
|
and binds **loopback only** — `IPAddressAllow=localhost` means Apache is the
|
|
only possible client.
|
|
|
|
---
|
|
|
|
## 5. Apache vhost + TLS
|
|
|
|
**Order matters.** Installing a vhost that references a cert which does not yet
|
|
exist breaks `apache2ctl configtest`, which blocks reloads for *every* site on
|
|
the box. So: DNS → cert → full vhost.
|
|
|
|
### 5a. DNS
|
|
|
|
```bash
|
|
dig +short vm.example.com A # must return your ORIGIN ip
|
|
```
|
|
|
|
Behind Cloudflare, set the record to **DNS only (grey cloud)** first, or the
|
|
HTTP-01 challenge is intercepted by the edge.
|
|
|
|
### 5b. Minimal :80 vhost, so certbot can answer the challenge
|
|
|
|
```bash
|
|
sudo install -d -o www-data -g www-data -m 755 /var/www/vm.example.com/public_html
|
|
sudo cp apache/vm.txt3.net-step1.conf /etc/apache2/sites-available/vm.example.com.conf
|
|
# edit ServerName / paths / IPs to match your host
|
|
sudo a2ensite vm.example.com
|
|
sudo apache2ctl configtest && sudo systemctl reload apache2
|
|
```
|
|
|
|
### 5c. Issue the certificate
|
|
|
|
```bash
|
|
sudo certbot certonly --webroot -w /var/www/vm.example.com/public_html \
|
|
-d vm.example.com --cert-name vm.example.com
|
|
```
|
|
|
|
### 5d. Full vhost — HTTPS + reverse proxy
|
|
|
|
```bash
|
|
sudo a2enmod proxy proxy_http headers rewrite ssl
|
|
sudo cp apache/vm.txt3.net.conf /etc/apache2/sites-available/vm.example.com.conf
|
|
# edit ServerName, cert paths, IPs
|
|
sudo install -d /var/www/vm.example.com/public_html/.well-known/acme-challenge
|
|
sudo apache2ctl configtest && sudo systemctl reload apache2
|
|
```
|
|
|
|
### 5e. Verify through the real hostname
|
|
|
|
Not localhost — the whole point is to exercise Apache, TLS and the proxy:
|
|
|
|
```bash
|
|
R="--resolve vm.example.com:443:YOUR.ORIGIN.IP"
|
|
curl -s $R https://vm.example.com/healthz
|
|
curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.example.com/login
|
|
curl -s $R -D- -o /dev/null https://vm.example.com/login | grep -i 'strict-transport\|content-security'
|
|
```
|
|
|
|
Behind a CDN, `curl https://host/` tests the *CDN*, not your origin. Always
|
|
`--resolve` to the origin IP when verifying a change.
|
|
|
|
---
|
|
|
|
## 6. Backfill existing voicemails
|
|
|
|
```bash
|
|
V=/opt/vm-transcribe/venv/bin/python3
|
|
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --dry-run # preview
|
|
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --limit 5 # trial
|
|
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py # all
|
|
```
|
|
|
|
Idempotent — safe to re-run; it resumes rather than duplicating. Expect a
|
|
`no_speech` count: 44-byte WAVs are hung-up calls with no audio, not failures.
|
|
|
|
Reckon on ~1.5 s per message plus transcription time; 157 messages took 3m40s
|
|
on 4 cores.
|
|
|
|
---
|
|
|
|
## 7. Log in
|
|
|
|
Browse to `https://vm.example.com/` and log in with a **mailbox number and its
|
|
existing voicemail PIN** from `voicemail.conf`. No new passwords are created.
|
|
|
|
---
|
|
|
|
## Uninstall
|
|
|
|
```bash
|
|
sudo systemctl disable --now vm-portal
|
|
sudo rm /etc/systemd/system/vm-portal.service && sudo systemctl daemon-reload
|
|
sudo a2dissite vm.example.com && sudo systemctl reload apache2
|
|
# restore the original mailcmd
|
|
sudo cp /etc/asterisk/voicemail.conf.bak-<timestamp> /etc/asterisk/voicemail.conf
|
|
sudo asterisk -rx 'voicemail reload'
|
|
sudo rm -rf /opt/vm-transcribe # keep /var/lib/vm-transcribe for the data
|
|
```
|