Initial build: Asterisk voicemail transcription + portal
- mailcmd replacement (vm_mailcmd.py): faster-whisper transcription (CPU int8), extractive summary + intent tags + spoken-digit number extraction, multipart/alternative HTML email, fail-safe relay of original message - Telegram DM delivery (vm_telegram.py) with per-mailbox routing - Caller-ID -> name (vm_contacts.py): file / google / carddav backends - SQLite store (vm_store.py) with content-addressed audio - FastAPI portal (vm_web.py): PIN login, list/play/delete, per-user settings, zero JS, loopback-only behind Apache TLS - Backfill importer (vm_import.py) for existing spool recordings - systemd unit, Apache vhost + certbot TLS, install.sh - Docs: INSTALL, CONFIGURATION, ARCHITECTURE, OPERATIONS, SECURITY, TESTING Verified end-to-end on mail.txt3.net: 157 historical messages backfilled, live voicemail -> transcribed -> stored -> visible at https://vm.txt3.net.
This commit is contained in:
34
.gitignore
vendored
Normal file
34
.gitignore
vendored
Normal file
@ -0,0 +1,34 @@
|
|||||||
|
# Local config with real credentials — never commit these.
|
||||||
|
config/telegram.conf
|
||||||
|
config/contacts.conf
|
||||||
|
*.conf.local
|
||||||
|
|
||||||
|
# Runtime data
|
||||||
|
*.db
|
||||||
|
*.db-wal
|
||||||
|
*.db-shm
|
||||||
|
audio/
|
||||||
|
models/
|
||||||
|
*.log
|
||||||
|
|
||||||
|
# Test artefacts
|
||||||
|
*.eml
|
||||||
|
*.wav
|
||||||
|
*.mp3
|
||||||
|
*.ogg
|
||||||
|
preview.html
|
||||||
|
ui_preview.html
|
||||||
|
tstore.db*
|
||||||
|
taudio/
|
||||||
|
|
||||||
|
# Python
|
||||||
|
__pycache__/
|
||||||
|
*.py[cod]
|
||||||
|
venv/
|
||||||
|
.venv/
|
||||||
|
|
||||||
|
# Editor / OS
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
*.swp
|
||||||
|
.DS_Store
|
||||||
29
CHANGELOG.md
Normal file
29
CHANGELOG.md
Normal file
@ -0,0 +1,29 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## 1.0.0 — 2026-08-13
|
||||||
|
|
||||||
|
Initial build on mail.txt3.net (Debian 12, Asterisk 20, Apache 2.4, Postfix).
|
||||||
|
|
||||||
|
- **Transcription**: per-voicemail `mailcmd` (`vm_mailcmd.py`) transcribes the
|
||||||
|
attached recording with faster-whisper `base.en` (CPU, int8, VAD-filtered).
|
||||||
|
- **Summarisation**: local extractive summariser with intent tags and spoken-
|
||||||
|
digit callback-number extraction. No LLM, by choice.
|
||||||
|
- **Email**: rebuilt `multipart/alternative` notification (plain + styled HTML)
|
||||||
|
with the recording attached; relays the original Asterisk message unchanged on
|
||||||
|
any error (fail-safe).
|
||||||
|
- **Telegram** (optional): per-mailbox routed voice-note DM with summary caption
|
||||||
|
and transcript follow-up; degrades sendVoice → sendDocument → sendMessage.
|
||||||
|
- **Contacts** (optional): caller-ID → name via `file` (vCard/CSV export),
|
||||||
|
`google` (People API OAuth), or `carddav` (app password; Nextcloud/Fastmail/
|
||||||
|
iCloud). Last-9-digit matching. Note: Google app passwords do not work.
|
||||||
|
- **Web portal**: FastAPI app at `https://vm.txt3.net` — PIN login (mailbox +
|
||||||
|
voicemail PIN from `voicemail.conf`), list, play, download, delete, per-user
|
||||||
|
settings. Zero JavaScript, `script-src 'none'` CSP, loopback-only backend
|
||||||
|
behind an Apache TLS reverse proxy. Brute-force lockout per (mailbox, IP).
|
||||||
|
- **Storage**: SQLite store with content-addressed audio (decoupled from
|
||||||
|
Asterisk's renumbering) and DB-backed sessions.
|
||||||
|
- **Backfill**: `vm_import.py` imports and transcribes existing spool recordings;
|
||||||
|
idempotent on `(mailbox, origtime, callerid)`. Backfilled 157 historical
|
||||||
|
messages on first run (3m40s, 0 failures).
|
||||||
|
- **Packaging**: venv at `/opt/vm-transcribe`, systemd `vm-portal.service`,
|
||||||
|
Apache vhost + certbot TLS, `install.sh`.
|
||||||
21
LICENSE
Normal file
21
LICENSE
Normal file
@ -0,0 +1,21 @@
|
|||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2026 jp
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
162
README.md
Normal file
162
README.md
Normal file
@ -0,0 +1,162 @@
|
|||||||
|
# Asterisk Voicemail Transcription & Portal
|
||||||
|
|
||||||
|
Turns Asterisk voicemail into something you can actually read, search and
|
||||||
|
manage:
|
||||||
|
|
||||||
|
- **Transcribes** each recording locally with [faster-whisper](https://github.com/SYSTRAN/faster-whisper) (CPU, no cloud, no API key)
|
||||||
|
- **Summarises** it, tags intent (callback requested, urgent, invoice…) and
|
||||||
|
extracts callback numbers — including spoken-out digits
|
||||||
|
- **Emails** a graphically designed `multipart/alternative` notification
|
||||||
|
(plain + HTML) with the recording attached
|
||||||
|
- **DMs Telegram** optionally, as a playable voice note with the summary
|
||||||
|
- **Resolves caller ID to a name** from Google Contacts or any CardDAV server
|
||||||
|
- **Serves a web portal** where mailbox users log in with their existing phone
|
||||||
|
PIN to read transcripts, play/download recordings, delete messages and
|
||||||
|
manage their own notification settings
|
||||||
|
|
||||||
|
Everything runs on the PBX host. No third-party service sees your voicemail.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What it looks like
|
||||||
|
|
||||||
|
**Email notification** — styled HTML card with metadata, summary panel, intent
|
||||||
|
chips, a `tel:` callback link, full transcript and the audio attached. A plain
|
||||||
|
text alternative is always included.
|
||||||
|
|
||||||
|
**Telegram** — voice note (ogg/opus) with the summary as its caption, intent
|
||||||
|
hashtags, and the transcript as a follow-up message.
|
||||||
|
|
||||||
|
**Portal** — one card per voicemail: caller, timestamp, duration, summary,
|
||||||
|
tags, callback link, inline player, collapsible transcript, and Mark read /
|
||||||
|
Download / Delete.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
incoming call
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────┐ voicemail.conf: mailcmd=… vm_mailcmd.py
|
||||||
|
│ Asterisk │ ─────────────────────────────┐
|
||||||
|
└─────────────┘ pipes an RFC822 message │
|
||||||
|
(notification + audio) ▼
|
||||||
|
┌───────────────┐
|
||||||
|
│ vm_mailcmd.py │
|
||||||
|
└───────┬───────┘
|
||||||
|
┌──────────────┬──────────────┬────┴─────────┬──────────────┐
|
||||||
|
▼ ▼ ▼ ▼ ▼
|
||||||
|
faster-whisper summarise() vm_contacts vm_store vm_telegram
|
||||||
|
(transcribe) + intents (name lookup) (SQLite + (voice note
|
||||||
|
+ numbers audio CAS) DM)
|
||||||
|
│ │
|
||||||
|
▼ ▼
|
||||||
|
multipart email ──▶ Postfix :25 ┌──────────────┐
|
||||||
|
│ vm_web.py │
|
||||||
|
│ (FastAPI) │
|
||||||
|
└──────┬───────┘
|
||||||
|
│ :8099 loopback
|
||||||
|
▼
|
||||||
|
Apache (TLS)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
https://vm.txt3.net
|
||||||
|
```
|
||||||
|
|
||||||
|
**Delivery order is deliberate**: email first, then the database, then Telegram.
|
||||||
|
Each later stage is wrapped so a failure only logs. If anything throws at the
|
||||||
|
top level, the *original* Asterisk notification is relayed unchanged. A
|
||||||
|
voicemail notification is never lost because a summariser or an API failed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|---|---|
|
||||||
|
| `src/vm_mailcmd.py` | The `mailcmd` — entry point for every voicemail. Orchestrates everything. |
|
||||||
|
| `src/vm_store.py` | SQLite store: schema, settings, sessions, content-addressed audio. |
|
||||||
|
| `src/vm_telegram.py` | Telegram delivery, per-mailbox routing. |
|
||||||
|
| `src/vm_contacts.py` | Caller-ID → name via local export, Google People API, or CardDAV. |
|
||||||
|
| `src/vm_web.py` | FastAPI portal: login, list, play, delete, settings. |
|
||||||
|
| `src/vm_auth.py` | Parses `voicemail.conf` so users log in with their phone PIN. |
|
||||||
|
| `src/vm_import.py` | Backfills existing spool recordings into the database. |
|
||||||
|
| `src/vm_tg_setup.py` | Helper: discover Telegram chat IDs, test a route. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
See **[docs/INSTALL.md](docs/INSTALL.md)** for the full walkthrough. Short version:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo scripts/install.sh # venv, deps, model cache, mailcmd wiring
|
||||||
|
sudo cp systemd/vm-portal.service /etc/systemd/system/
|
||||||
|
sudo systemctl enable --now vm-portal
|
||||||
|
# then follow docs/INSTALL.md §5 for the Apache vhost + TLS
|
||||||
|
```
|
||||||
|
|
||||||
|
Backfill your existing voicemails:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||||
|
/opt/vm-transcribe/vm_import.py --dry-run # preview
|
||||||
|
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||||
|
/opt/vm-transcribe/vm_import.py # do it
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
| Document | Contents |
|
||||||
|
|---|---|
|
||||||
|
| [docs/INSTALL.md](docs/INSTALL.md) | Step-by-step install, including TLS ordering |
|
||||||
|
| [docs/CONFIGURATION.md](docs/CONFIGURATION.md) | Every config file and option |
|
||||||
|
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Design decisions and why |
|
||||||
|
| [docs/OPERATIONS.md](docs/OPERATIONS.md) | Day-to-day running, backup, troubleshooting |
|
||||||
|
| [docs/SECURITY.md](docs/SECURITY.md) | Threat model, hardening, privacy |
|
||||||
|
| [docs/TESTING.md](docs/TESTING.md) | How to verify each part |
|
||||||
|
| [CHANGELOG.md](CHANGELOG.md) | Version history |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Asterisk with `app_voicemail` (file-based spool, not ODBC/IMAP storage)
|
||||||
|
- Python 3.9+
|
||||||
|
- `ffmpeg` (Telegram voice-note transcoding; `sox` optionally for gsm)
|
||||||
|
- An MTA listening on `localhost:25` (Postfix here)
|
||||||
|
- ~200 MB disk for the whisper `base.en` model
|
||||||
|
- Apache with `proxy`, `proxy_http`, `headers`, `rewrite`, `ssl` for the portal
|
||||||
|
|
||||||
|
Runs comfortably on 4 CPU cores with no GPU: ~8 s to transcribe 23 s of audio.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Design notes worth knowing
|
||||||
|
|
||||||
|
**Summarisation is local and extractive.** Frequency-scored sentence selection
|
||||||
|
with position/digit weighting, plus regex intent tags. No LLM, by choice — it
|
||||||
|
keeps voicemail content on your own hardware. `summarise()` in
|
||||||
|
`vm_mailcmd.py` is a single swap-in point if you want an abstractive model.
|
||||||
|
|
||||||
|
**Audio is content-addressed, not referenced by spool path.** Asterisk renumbers
|
||||||
|
`msgNNNN` files when a message is deleted, so a stored path silently starts
|
||||||
|
pointing at the wrong recording. Recordings are copied to
|
||||||
|
`audio/<sha[:2]>/<sha>.wav`.
|
||||||
|
|
||||||
|
**App passwords cannot read Google Contacts.** Google disabled basic auth for
|
||||||
|
CardDAV/CalDAV/IMAP/SMTP/POP on 2024-09-30. Use the `file` backend (a vCard
|
||||||
|
export) or the `google` backend (OAuth). The `carddav` backend with an app
|
||||||
|
password works for Nextcloud, Fastmail and iCloud.
|
||||||
|
|
||||||
|
**The portal ships zero JavaScript**, which lets its CSP be `script-src 'none'`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Licence
|
||||||
|
|
||||||
|
MIT — see [LICENSE](LICENSE).
|
||||||
12
apache/vm.txt3.net-step1.conf
Normal file
12
apache/vm.txt3.net-step1.conf
Normal file
@ -0,0 +1,12 @@
|
|||||||
|
<VirtualHost 51.68.212.39:80 [2001:41d0:801:2000::2245]:80>
|
||||||
|
ServerName vm.txt3.net
|
||||||
|
DocumentRoot /home/txt3/domains/vm.txt3.net/public_html
|
||||||
|
ErrorLog /var/log/virtualmin/vm.txt3.net_error_log
|
||||||
|
CustomLog /var/log/virtualmin/vm.txt3.net_access_log combined
|
||||||
|
|
||||||
|
<Directory /home/txt3/domains/vm.txt3.net/public_html>
|
||||||
|
Require all granted
|
||||||
|
Options -Indexes
|
||||||
|
AllowOverride None
|
||||||
|
</Directory>
|
||||||
|
</VirtualHost>
|
||||||
54
apache/vm.txt3.net.conf
Normal file
54
apache/vm.txt3.net.conf
Normal file
@ -0,0 +1,54 @@
|
|||||||
|
<VirtualHost 51.68.212.39:80 [2001:41d0:801:2000::2245]:80>
|
||||||
|
ServerName vm.txt3.net
|
||||||
|
ErrorLog /var/log/virtualmin/vm.txt3.net_error_log
|
||||||
|
CustomLog /var/log/virtualmin/vm.txt3.net_access_log combined
|
||||||
|
|
||||||
|
# Let certbot answer HTTP-01 challenges from the webroot
|
||||||
|
Alias /.well-known/acme-challenge/ /home/txt3/domains/vm.txt3.net/public_html/.well-known/acme-challenge/
|
||||||
|
<Directory /home/txt3/domains/vm.txt3.net/public_html/.well-known/acme-challenge>
|
||||||
|
Require all granted
|
||||||
|
Options -Indexes
|
||||||
|
</Directory>
|
||||||
|
ProxyPass /.well-known !
|
||||||
|
|
||||||
|
# Everything else goes to HTTPS
|
||||||
|
RewriteEngine on
|
||||||
|
RewriteCond %{HTTPS} !=on
|
||||||
|
RewriteRule ^/(?!\.well-known)(.*)$ https://vm.txt3.net/$1 [R=301,L]
|
||||||
|
</VirtualHost>
|
||||||
|
|
||||||
|
<VirtualHost 51.68.212.39:443 [2001:41d0:801:2000::2245]:443>
|
||||||
|
ServerName vm.txt3.net
|
||||||
|
ErrorLog /var/log/virtualmin/vm.txt3.net_error_log
|
||||||
|
CustomLog /var/log/virtualmin/vm.txt3.net_access_log combined
|
||||||
|
|
||||||
|
SSLEngine on
|
||||||
|
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
|
||||||
|
# Replaced by certbot with the vm.txt3.net cert once issued.
|
||||||
|
SSLCertificateFile /etc/letsencrypt/live/vm.txt3.net/fullchain.pem
|
||||||
|
SSLCertificateKeyFile /etc/letsencrypt/live/vm.txt3.net/privkey.pem
|
||||||
|
|
||||||
|
# --- security headers -------------------------------------------------
|
||||||
|
Header always set X-Content-Type-Options "nosniff"
|
||||||
|
Header always set X-Frame-Options "DENY"
|
||||||
|
Header always set Referrer-Policy "strict-origin-when-cross-origin"
|
||||||
|
Header always set Strict-Transport-Security "max-age=15768000"
|
||||||
|
# The app uses only inline <style>, no external or inline JS.
|
||||||
|
Header always set Content-Security-Policy "default-src 'self'; style-src 'self' 'unsafe-inline'; media-src 'self'; img-src 'self' data:; script-src 'none'; frame-ancestors 'none'; base-uri 'none'; form-action 'self'"
|
||||||
|
|
||||||
|
# --- reverse proxy to the voicemail portal ----------------------------
|
||||||
|
ProxyPreserveHost On
|
||||||
|
ProxyRequests Off
|
||||||
|
# Recordings can be a few MB; give uploads/streams room.
|
||||||
|
ProxyTimeout 120
|
||||||
|
|
||||||
|
ProxyPass /.well-known !
|
||||||
|
ProxyPass / http://127.0.0.1:8099/ retry=0
|
||||||
|
ProxyPassReverse / http://127.0.0.1:8099/
|
||||||
|
|
||||||
|
RequestHeader set X-Forwarded-Proto "https"
|
||||||
|
|
||||||
|
<Location />
|
||||||
|
Require all granted
|
||||||
|
</Location>
|
||||||
|
</VirtualHost>
|
||||||
14
config/README.md
Normal file
14
config/README.md
Normal file
@ -0,0 +1,14 @@
|
|||||||
|
# Examples
|
||||||
|
|
||||||
|
These files are templates. **Copy to a live path and fill in secrets** — the
|
||||||
|
live files are git-ignored so credentials never get committed.
|
||||||
|
|
||||||
|
- `config/telegram.conf.example` → `/opt/vm-transcribe/telegram.conf`
|
||||||
|
- `config/contacts.conf.example` → `/opt/vm-transcribe/contacts.conf`
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo install -o root -g asterisk -m 640 config/telegram.conf.example /opt/vm-transcribe/telegram.conf
|
||||||
|
sudo install -o root -g asterisk -m 640 config/contacts.conf.example /opt/vm-transcribe/contacts.conf
|
||||||
|
```
|
||||||
|
|
||||||
|
Then edit the real files with `sudo` and add your bot token / contacts source.
|
||||||
73
config/contacts.conf.example
Normal file
73
config/contacts.conf.example
Normal file
@ -0,0 +1,73 @@
|
|||||||
|
# /opt/vm-transcribe/contacts.conf
|
||||||
|
#
|
||||||
|
# Caller-ID -> name lookup for voicemail notifications.
|
||||||
|
# Re-read on every voicemail; no restart needed.
|
||||||
|
#
|
||||||
|
# chmod 640, owned root:asterisk - may contain a password.
|
||||||
|
|
||||||
|
[contacts]
|
||||||
|
# Master switch. If no, notifications just show the raw caller ID.
|
||||||
|
enabled = yes
|
||||||
|
|
||||||
|
# Which backends to try, in order, comma separated:
|
||||||
|
# file - a local vCard (.vcf) or CSV export. No auth, fast, works offline.
|
||||||
|
# google - Google People API using the Hermes OAuth token (recommended
|
||||||
|
# for Google accounts; see note about app passwords below).
|
||||||
|
# carddav - any CardDAV server with username + app password.
|
||||||
|
backends = file, google
|
||||||
|
|
||||||
|
# Successful and failed lookups are cached here to avoid hammering an API
|
||||||
|
# on every call. Delete the file to force a refresh.
|
||||||
|
cache_path = /var/lib/vm-transcribe/contacts_cache.json
|
||||||
|
cache_ttl = 86400
|
||||||
|
|
||||||
|
# Match on the last N digits of the number, so +447941223856,
|
||||||
|
# 07941223856 and 447941223856 all resolve to the same contact.
|
||||||
|
# 9 is a sane default for UK/US. Lower it only if you get misses.
|
||||||
|
match_digits = 9
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
[file]
|
||||||
|
# vCard (.vcf) or CSV exported from Google Contacts:
|
||||||
|
# contacts.google.com -> Export -> vCard (or Google CSV)
|
||||||
|
# This is the most reliable option: no tokens, no API limits, no network.
|
||||||
|
path = /var/lib/vm-transcribe/contacts.vcf
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
[google]
|
||||||
|
# Uses the OAuth token from the Hermes google-workspace skill, which must be
|
||||||
|
# authorised with the People API / contacts.readonly scope.
|
||||||
|
token_path = /var/lib/vm-transcribe/google_token.json
|
||||||
|
|
||||||
|
# Optional: refresh the local export from the People API on this schedule
|
||||||
|
# (seconds). 0 = only look up on demand. Bulk-syncing to the file backend is
|
||||||
|
# much faster per call than a live query.
|
||||||
|
sync_interval = 0
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
[carddav]
|
||||||
|
# Generic CardDAV with an app password.
|
||||||
|
#
|
||||||
|
# !! IMPORTANT: this will NOT work against Google. Google disabled basic
|
||||||
|
# !! authentication for CardDAV, CalDAV, IMAP, SMTP and POP on 2024-09-30;
|
||||||
|
# !! app passwords are rejected and Google's CardDAV now requires OAuth 2.0.
|
||||||
|
# !! Use the 'google' or 'file' backend for Google Contacts instead.
|
||||||
|
#
|
||||||
|
# These settings do work for Nextcloud, Fastmail, iCloud, Radicale, etc.
|
||||||
|
url =
|
||||||
|
username =
|
||||||
|
|
||||||
|
# App password (NOT the main account password). Generate one in your
|
||||||
|
# provider's security settings.
|
||||||
|
app_password =
|
||||||
|
|
||||||
|
# Some providers need an explicit addressbook path, e.g.
|
||||||
|
# Nextcloud: /remote.php/dav/addressbooks/users/<user>/contacts/
|
||||||
|
# iCloud: discovered automatically from the principal URL
|
||||||
|
addressbook_path =
|
||||||
|
|
||||||
|
verify_tls = yes
|
||||||
|
timeout = 20
|
||||||
51
config/telegram.conf.example
Normal file
51
config/telegram.conf.example
Normal file
@ -0,0 +1,51 @@
|
|||||||
|
# /opt/vm-transcribe/telegram.conf
|
||||||
|
#
|
||||||
|
# Telegram delivery for voicemail notifications.
|
||||||
|
# Reloaded on every voicemail - no restart needed after editing.
|
||||||
|
#
|
||||||
|
# chmod 640, owned root:asterisk - it contains a bot token.
|
||||||
|
|
||||||
|
[telegram]
|
||||||
|
# Master switch. Set to no to disable Telegram entirely (email still sends).
|
||||||
|
enabled = yes
|
||||||
|
|
||||||
|
# Bot token from @BotFather, e.g. 123456789:AAH...
|
||||||
|
token =
|
||||||
|
|
||||||
|
# Where to send when a mailbox has no [mailbox:N] section below.
|
||||||
|
# Leave blank to send nothing for unmapped mailboxes.
|
||||||
|
default_chat_id =
|
||||||
|
|
||||||
|
# Attach the recording as a Telegram voice note (ogg/opus, plays inline).
|
||||||
|
send_audio = yes
|
||||||
|
|
||||||
|
# Include the full transcript as a follow-up message when it is longer
|
||||||
|
# than the summary. Set to no for summary-only.
|
||||||
|
send_transcript = yes
|
||||||
|
|
||||||
|
# Seconds to wait on the Telegram API before giving up. Email is already
|
||||||
|
# sent by this point, so a timeout is never fatal.
|
||||||
|
timeout = 20
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Per-mailbox routing. Section name is [mailbox:<mailbox number>]
|
||||||
|
# chat_id may be a comma-separated list to fan out to several people.
|
||||||
|
#
|
||||||
|
# To find a chat ID: message your bot, then visit
|
||||||
|
# https://api.telegram.org/bot<TOKEN>/getUpdates
|
||||||
|
# and read result[].message.chat.id (negative numbers are groups).
|
||||||
|
#
|
||||||
|
# Any key from [telegram] can be overridden per mailbox.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# [mailbox:1001]
|
||||||
|
# chat_id = 123456789
|
||||||
|
|
||||||
|
# [mailbox:1002]
|
||||||
|
# chat_id = 987654321, 123456789
|
||||||
|
# send_transcript = no
|
||||||
|
|
||||||
|
# [mailbox:1003]
|
||||||
|
# chat_id = -1001234567890
|
||||||
|
# send_audio = no
|
||||||
137
docs/ARCHITECTURE.md
Normal file
137
docs/ARCHITECTURE.md
Normal file
@ -0,0 +1,137 @@
|
|||||||
|
# Architecture
|
||||||
|
|
||||||
|
How the pieces fit and *why* they are shaped the way they are. Read this
|
||||||
|
before changing anything in `vm_mailcmd.py` or `vm_store.py`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
Asterisk ──(RFC822 on stdin)──▶ vm_mailcmd.py
|
||||||
|
│
|
||||||
|
┌──────────┬───────────────┼───────────────┬───────────────┐
|
||||||
|
▼ ▼ ▼ ▼ ▼
|
||||||
|
transcribe summarise resolve store telegram
|
||||||
|
(whisper) +intents contact (sqlite) (DM)
|
||||||
|
│ │ │ │ │
|
||||||
|
└────▶ build multipart email ─▶ Postfix :25
|
||||||
|
(plain + HTML + audio attached)
|
||||||
|
│
|
||||||
|
└──▶ vm_store.add_message() (best-effort)
|
||||||
|
└──▶ vm_telegram.send() (best-effort)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Email is sent before anything else.** The portal and Telegram are conveniences;
|
||||||
|
the email is the contract with the user. If the database is down or Telegram's
|
||||||
|
API throws, the mail has already gone. Each later stage is in its own `try/except`
|
||||||
|
that only logs. The reverse — a transcription error blocking the email — never
|
||||||
|
happens, because the email builder is given whatever text we have, even `""`.
|
||||||
|
|
||||||
|
At the very top, the whole `main()` body is wrapped so that *any* unexpected
|
||||||
|
exception relays the **original** Asterisk message unchanged. Nothing we add can
|
||||||
|
silently swallow a voicemail.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Why faster-whisper, CPU, int8, base.en
|
||||||
|
|
||||||
|
- **faster-whisper** (CTranslate2) is dramatically faster than OpenAI's
|
||||||
|
`whisper` on CPU — ~8 s for 23 s of 8 kHz audio here, versus minutes.
|
||||||
|
- **CPU/int8** because this host has no GPU and the voicemail rate is low. A
|
||||||
|
GPU would be idle 99.9 % of the time.
|
||||||
|
- **base.en** is plenty for clear phone audio; `small.en` only if you see
|
||||||
|
accuracy problems with names/numbers. Set `VM_WHISPER_MODEL`.
|
||||||
|
|
||||||
|
The model is loaded once per process. Asterisk fires a fresh `mailcmd` per
|
||||||
|
voicemail, so there is a ~1 s model-load cost *per message* — acceptable at PBX
|
||||||
|
volumes, and the model is pre-cached at install so the first call doesn't also
|
||||||
|
pay a download.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Why the summary is extractive, not an LLM
|
||||||
|
|
||||||
|
The user explicitly declined both OpenAI and Ollama. `summarise()` is:
|
||||||
|
|
||||||
|
1. split into sentences,
|
||||||
|
2. score each by word frequency (stop-words removed) plus a boost for
|
||||||
|
position (first/last sentence) and for sentences containing digits,
|
||||||
|
3. pick the top sentences, ordered as spoken, capped at ~3 and a length limit,
|
||||||
|
4. wrap with regex intent tags (`Call back requested`, `Urgent`,
|
||||||
|
`Payment / invoice`, …) and number extraction.
|
||||||
|
|
||||||
|
It reads well on phone voicemail and needs no network. The single function
|
||||||
|
`summarise()` is a deliberate **swap-in point** — replace it with a local model
|
||||||
|
or an API call without touching the rest of the pipeline.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Why audio is content-addressed
|
||||||
|
|
||||||
|
Asterisk **renumbers** `msgNNNN` files when a message is deleted from a folder.
|
||||||
|
A stored path like `…/INBOX/msg0042.wav` therefore points at a *different*
|
||||||
|
recording a week later. So:
|
||||||
|
|
||||||
|
- On store, the audio is copied to
|
||||||
|
`audio/<sha256[:2]>/<sha256>.<ext>` (blake2/sha256 in `vm_store`).
|
||||||
|
- The spool path is kept only as a *delete hint*.
|
||||||
|
- On delete, the blob is unlinked only if no other row references that hash.
|
||||||
|
|
||||||
|
This decouples the portal's playback from Asterisk's ever-shifting filenames.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Why the portal is its own service behind Apache
|
||||||
|
|
||||||
|
- A long-running `uvicorn` is the natural fit; Asterisk's `mailcmd` model
|
||||||
|
(one short-lived process per call) is wrong for a web app.
|
||||||
|
- It binds **127.0.0.1:8099 only** (`IPAddressAllow=localhost` in the unit), so
|
||||||
|
the only way in is the Apache reverse proxy, which terminates TLS and enforces
|
||||||
|
headers.
|
||||||
|
- Running as the `asterisk` user means it can read `voicemail.conf` (for PIN
|
||||||
|
auth) and the spool (for future direct-file features) without widening root.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Auth model
|
||||||
|
|
||||||
|
Users log in with their **mailbox number + existing voicemail PIN** parsed from
|
||||||
|
`/etc/asterisk/voicemail.conf` (`vm_auth.py`). No password database to keep in
|
||||||
|
sync, no new credentials to provision. Sessions are stored in the DB (random
|
||||||
|
token, `Secure` cookie, `HttpOnly`, `SameSite=lax`, 12 h) so logout and expiry
|
||||||
|
are enforced server-side, not just by deleting a cookie.
|
||||||
|
|
||||||
|
Every data query is filtered by the session's mailbox. Cross-mailbox access must
|
||||||
|
404 — there is a test for this in [TESTING.md](TESTING.md).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Lockout
|
||||||
|
|
||||||
|
Voicemail PINs are short (often 4 digits) and the portal is internet-facing, so
|
||||||
|
failed logins are throttled in-process: 5 per `(mailbox, IP)` then 15 minutes.
|
||||||
|
State is in memory by design (single worker). A restart clears it — fine, because
|
||||||
|
a restart is an admin action, not an attacker one.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Telegram routing
|
||||||
|
|
||||||
|
`telegram.conf` is re-read on every voicemail (no reload). Resolution order:
|
||||||
|
`[mailbox:N]` `chat_id` → `default_chat_id` → nothing. A chat id may fan out to
|
||||||
|
several recipients (comma list) or be a group (negative). Per-mailbox sections
|
||||||
|
can override any global (`send_audio`, `send_transcript`). Delivery degrades
|
||||||
|
`sendVoice → sendDocument → sendMessage` so a summary always gets through.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Privacy posture
|
||||||
|
|
||||||
|
All audio and transcripts stay on the PBX. The only egress is:
|
||||||
|
|
||||||
|
- **Postfix → your MX** for email (the existing path),
|
||||||
|
- **Telegram API** (only if you enable Telegram),
|
||||||
|
- **Google People API** (only if you enable the `google` contacts backend).
|
||||||
|
|
||||||
|
Nothing is sent to OpenAI, Anthropic, or any transcription vendor.
|
||||||
159
docs/CONFIGURATION.md
Normal file
159
docs/CONFIGURATION.md
Normal file
@ -0,0 +1,159 @@
|
|||||||
|
# Configuration
|
||||||
|
|
||||||
|
All config lives in `/opt/vm-transcribe/`. Templates are committed as
|
||||||
|
`*.example`; the live files are created by `install.sh` or by you. Secret-bearing
|
||||||
|
files (`telegram.conf`, `contacts.conf`) are mode `640 root:asterisk` and git-
|
||||||
|
ignored.
|
||||||
|
|
||||||
|
Most settings can also be set via environment variables, which is how the systemd
|
||||||
|
unit wires production values. Env vars override file values when both are present
|
||||||
|
unless noted.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Environment variables (shared)
|
||||||
|
|
||||||
|
| Variable | Default | Used by | Meaning |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `VM_DB` | `/var/lib/vm-transcribe/voicemail.db` | store, web, import | SQLite path. Created with WAL mode if missing. |
|
||||||
|
| `VM_AUDIO_DIR` | `/var/lib/vm-transcribe/audio` | store, web, import | Where copied recordings live (`<sha[:2]>/<sha>.wav`). |
|
||||||
|
| `VM_LOG` | `/var/log/asterisk/vm_mailcmd.log` | mailcmd | Log path. |
|
||||||
|
| `VM_MODEL_CACHE` | `/opt/vm-transcribe/models` | mailcmd, import | Whisper model download/load dir. |
|
||||||
|
| `VM_WHISPER_MODEL` | `base.en` | mailcmd, import | Model name. `tiny`/`base`/`small` trade speed for accuracy. |
|
||||||
|
| `VM_SPOOL` | `/var/spool/asterisk/voicemail` | import | Spool root to scan. |
|
||||||
|
| `VM_ASTERISK_CONF` | `/etc/asterisk/voicemail.conf` | web, auth | For PIN lookup. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## voicemail.conf (Asterisk)
|
||||||
|
|
||||||
|
One line is all the pipeline needs:
|
||||||
|
|
||||||
|
```ini
|
||||||
|
mailcmd=/opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_mailcmd.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Place it in the `[general]` section (or any context). After editing:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo asterisk -rx 'voicemail reload'
|
||||||
|
```
|
||||||
|
|
||||||
|
`install.sh` backs up the file and adds this line automatically.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## telegram.conf
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[telegram]
|
||||||
|
enabled = yes
|
||||||
|
token = 123456789:AAH... ; from @BotFather
|
||||||
|
default_chat_id = ; fallback for unmapped mailboxes (blank = none)
|
||||||
|
send_audio = yes ; voice note with caption
|
||||||
|
send_transcript = yes ; full transcript as a follow-up message
|
||||||
|
timeout = 20 ; seconds per send attempt
|
||||||
|
|
||||||
|
[mailbox:1001]
|
||||||
|
chat_id = 123456789 ; single recipient
|
||||||
|
|
||||||
|
[mailbox:1002]
|
||||||
|
chat_id = 5551, 5552 ; comma list fans out to several people
|
||||||
|
send_transcript = no ; any global key can be overridden per mailbox
|
||||||
|
|
||||||
|
[mailbox:1003]
|
||||||
|
chat_id = -1001234567890 ; negative = group chat
|
||||||
|
send_audio = no ; text-only DM
|
||||||
|
```
|
||||||
|
|
||||||
|
**Routing rules** (evaluated per voicemail):
|
||||||
|
|
||||||
|
1. Match the `[mailbox:N]` section for the voicemail's mailbox number.
|
||||||
|
2. `chat_id` may be a single id, a comma-separated list (fan-out), or a group id
|
||||||
|
(negative).
|
||||||
|
3. If no mailbox section matches, fall back to `default_chat_id`.
|
||||||
|
4. If `enabled = no` or no route resolves, nothing is sent — email still goes out.
|
||||||
|
5. The file is re-read on every voicemail; no reload needed after editing.
|
||||||
|
|
||||||
|
**Delivery fallback chain** (per chat id, tried in order): `sendVoice` →
|
||||||
|
`sendDocument` → `sendMessage` (text-only). So a Telegram outage or a bad audio
|
||||||
|
file still delivers the summary.
|
||||||
|
|
||||||
|
> Contact resolution note: a user must message the bot once before it can DM
|
||||||
|
> them. Telegram does not permit bots to initiate conversations.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## contacts.conf
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[contacts]
|
||||||
|
backends = file ; space/comma separated: file google carddav
|
||||||
|
cache_ttl = 86400 ; seconds; cache hits AND misses
|
||||||
|
cache_file = /var/lib/vm-transcribe/contacts-cache.json
|
||||||
|
|
||||||
|
# --- 'file' backend (recommended): a vCard or CSV export -------------
|
||||||
|
[file]
|
||||||
|
path = /var/lib/vm-transcribe/contacts.vcf
|
||||||
|
# CSV (Google export "All contact data") is auto-detected:
|
||||||
|
# path = /var/lib/vm-transcribe/contacts.csv
|
||||||
|
|
||||||
|
# --- 'google' backend: People API via OAuth ---------------------------
|
||||||
|
[google]
|
||||||
|
token_file = /var/lib/vm-transcribe/google-token.json
|
||||||
|
# Obtain a token with the People API (contacts.readonly) scope. See
|
||||||
|
# docs/SECURITY.md. No app password here — OAuth only.
|
||||||
|
|
||||||
|
# --- 'carddav' backend: app password, Nextcloud/Fastmail/iCloud --------
|
||||||
|
[carddav]
|
||||||
|
# Google rejects app passwords (basic auth disabled 2024-09-30); a Google
|
||||||
|
# URL here is refused with a log warning and the backend is skipped.
|
||||||
|
url = https://contacts.fastmail.com/dav/addressbooks/user/xxx/Default
|
||||||
|
username = jp@txt3.com
|
||||||
|
app_password = ; app-specific password, not your login password
|
||||||
|
```
|
||||||
|
|
||||||
|
**Matching** is on the **last 9 digits** of the caller ID, so `+447700900123`,
|
||||||
|
`07700900123` and `447700900123` all collapse to the same contact. Names resolve
|
||||||
|
best-effort; on any backend error the caller ID is used as-is.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Portal settings (per mailbox, in the SQLite store)
|
||||||
|
|
||||||
|
These are edited from the portal's **Settings** page by each user; they are not
|
||||||
|
config files. Every value defaults to `yes`.
|
||||||
|
|
||||||
|
| Setting | Effect |
|
||||||
|
|---|---|
|
||||||
|
| `summarise` | Run the summariser (if off, the raw transcript is shown). |
|
||||||
|
| `telegram` | Forward new voicemails to the mailbox's Telegram route (requires telegram.conf). |
|
||||||
|
| `email_notify` | Send the enriched email (if off, the message is stored but not emailed). |
|
||||||
|
| `highlight` | Colour the caller chip when a name is resolved from contacts. |
|
||||||
|
| `max_inbox` | Keep at most N messages; oldest beyond N are auto-deleted (0 = unlimited). |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## vm-portal.service
|
||||||
|
|
||||||
|
Production values are passed as environment in the unit file:
|
||||||
|
|
||||||
|
```ini
|
||||||
|
Environment=VM_DB=/var/lib/vm-transcribe/voicemail.db
|
||||||
|
Environment=VM_AUDIO_DIR=/var/lib/vm-transcribe/audio
|
||||||
|
Environment=VM_ASTERISK_CONF=/etc/asterisk/voicemail.conf
|
||||||
|
Environment=VM_SESSION_HOURS=12
|
||||||
|
```
|
||||||
|
|
||||||
|
Login hardening (in-process):
|
||||||
|
|
||||||
|
| Var | Default | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `VM_MAX_FAILS` | `5` | Failed attempts per (mailbox, IP) before locking. |
|
||||||
|
| `VM_LOCK_MINUTES` | `15` | Lock duration. |
|
||||||
|
| `VM_SESSION_HOURS` | `12` | Session lifetime. |
|
||||||
|
| `VM_INSECURE_COOKIE` | unset | Set to `1` only for plain-HTTP testing; normally cookies are `Secure`. |
|
||||||
|
| `VM_BASE_PATH` | `""` | Set to e.g. `/voicemail` if the app is served under a sub-path. |
|
||||||
|
|
||||||
|
> A service restart clears the in-memory lockout counters. Testing lockout will
|
||||||
|
> lock you out of your own verification for `VM_LOCK_MINUTES`.
|
||||||
244
docs/INSTALL.md
Normal file
244
docs/INSTALL.md
Normal file
@ -0,0 +1,244 @@
|
|||||||
|
# Installation
|
||||||
|
|
||||||
|
Written against Debian 12 + Asterisk 20 + Apache 2.4 + Postfix, running under
|
||||||
|
Virtualmin. Adjust paths if your layout differs.
|
||||||
|
|
||||||
|
Everything installs to `/opt/vm-transcribe` with data in
|
||||||
|
`/var/lib/vm-transcribe`. Nothing is installed into system Python.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0. Prerequisites
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Asterisk voicemail must use file-based storage (the default), not ODBC/IMAP
|
||||||
|
grep -E '^(odbcstorage|imapserver)' /etc/asterisk/voicemail.conf # expect nothing
|
||||||
|
|
||||||
|
# tools
|
||||||
|
sudo apt install ffmpeg sox python3-venv # sox optional (gsm)
|
||||||
|
which certbot # for the portal's TLS
|
||||||
|
|
||||||
|
# an MTA on localhost:25
|
||||||
|
sudo ss -ltnp | grep ':25 '
|
||||||
|
```
|
||||||
|
|
||||||
|
**Check disk space before you start.** A full `/var` makes Postfix reject all
|
||||||
|
mail with `452 4.3.1 Insufficient system storage`, which looks like a bug in
|
||||||
|
this pipeline but is not:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
df -h /var
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Core pipeline
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo scripts/install.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
That script:
|
||||||
|
|
||||||
|
1. creates `/opt/vm-transcribe` and `/opt/vm-transcribe/models` owned by `asterisk`
|
||||||
|
2. builds a venv and installs `faster-whisper`
|
||||||
|
3. installs the Python modules
|
||||||
|
4. seeds `telegram.conf` / `contacts.conf` if absent (never overwrites)
|
||||||
|
5. creates `/var/log/asterisk/vm_mailcmd.log`
|
||||||
|
6. **pre-downloads the whisper model as the `asterisk` user** — do not skip
|
||||||
|
this, or the first real voicemail pays a ~150 MB download while the caller
|
||||||
|
waits for their notification
|
||||||
|
7. backs up `voicemail.conf` and rewrites `mailcmd=`
|
||||||
|
8. reloads Asterisk
|
||||||
|
|
||||||
|
Verify the wiring:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
grep '^mailcmd' /etc/asterisk/voicemail.conf
|
||||||
|
# mailcmd=/opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_mailcmd.py
|
||||||
|
```
|
||||||
|
|
||||||
|
### Test it before trusting it
|
||||||
|
|
||||||
|
Build a fake notification and pipe it through **as the `asterisk` user** —
|
||||||
|
testing as yourself hides permission problems:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /path/to/repo
|
||||||
|
python3 tests/make_test_mail.py /path/to/some.wav you@example.com > /tmp/t.eml
|
||||||
|
sudo chmod 644 /tmp/t.eml
|
||||||
|
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||||
|
/opt/vm-transcribe/vm_mailcmd.py < /tmp/t.eml
|
||||||
|
sudo tail -5 /var/log/asterisk/vm_mailcmd.log
|
||||||
|
```
|
||||||
|
|
||||||
|
You should see `transcribed … chars` then `sent enriched notification to …`.
|
||||||
|
Confirm the mail actually left:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo grep "to=<you@example.com>" /var/log/mail.log | tail -1 # expect status=sent
|
||||||
|
```
|
||||||
|
|
||||||
|
Also verify the fail-safe — a message with **no** audio must relay unchanged
|
||||||
|
rather than erroring:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
printf 'From: a@b\nTo: you@example.com\nSubject: no audio\n\nbody\n' > /tmp/n.eml
|
||||||
|
sudo chmod 644 /tmp/n.eml
|
||||||
|
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||||
|
/opt/vm-transcribe/vm_mailcmd.py < /tmp/n.eml
|
||||||
|
# log: "no audio attachment; relaying original"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Telegram (optional)
|
||||||
|
|
||||||
|
1. Create a bot: message [@BotFather](https://t.me/BotFather) → `/newbot` → copy the token.
|
||||||
|
2. Put it in `/opt/vm-transcribe/telegram.conf` (`sudo`, mode 640 root:asterisk).
|
||||||
|
3. **Each recipient must message the bot once** (`/start`) — Telegram forbids
|
||||||
|
bots from initiating conversations.
|
||||||
|
4. Discover chat IDs and test:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py ids
|
||||||
|
sudo /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py test 1001
|
||||||
|
```
|
||||||
|
|
||||||
|
The `test` subcommand sends a real DM with a playable voice note and exits
|
||||||
|
non-zero on failure. See [CONFIGURATION.md](CONFIGURATION.md#telegramconf) for
|
||||||
|
per-mailbox routing.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Contact lookup (optional)
|
||||||
|
|
||||||
|
Simplest and most reliable — a local export, no tokens, no rate limits:
|
||||||
|
|
||||||
|
1. contacts.google.com → **Export** → **vCard**
|
||||||
|
2. `sudo install -o asterisk -g asterisk -m 640 contacts.vcf /var/lib/vm-transcribe/contacts.vcf`
|
||||||
|
3. Confirm `contacts.conf` has `backends = file` and the matching `path`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||||
|
/opt/vm-transcribe/vm_contacts.py '<07700900123>'
|
||||||
|
```
|
||||||
|
|
||||||
|
> **Google app passwords do not work for contacts.** Google disabled basic auth
|
||||||
|
> for CardDAV/CalDAV/IMAP/SMTP/POP on 2024-09-30. Use the `file` or `google`
|
||||||
|
> (OAuth) backend. The `carddav` backend is for Nextcloud/Fastmail/iCloud.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Portal service
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo install -d -o asterisk -g asterisk -m 750 \
|
||||||
|
/var/lib/vm-transcribe /var/lib/vm-transcribe/audio
|
||||||
|
sudo /opt/vm-transcribe/venv/bin/pip install fastapi 'uvicorn[standard]' python-multipart
|
||||||
|
sudo install -o root -g root -m 644 systemd/vm-portal.service /etc/systemd/system/
|
||||||
|
sudo systemctl daemon-reload
|
||||||
|
sudo systemctl enable --now vm-portal
|
||||||
|
sudo systemctl is-active vm-portal
|
||||||
|
curl -s http://127.0.0.1:8099/healthz # {"ok":true,"messages":N}
|
||||||
|
```
|
||||||
|
|
||||||
|
The service runs as `asterisk` (so it can read `voicemail.conf` and the spool)
|
||||||
|
and binds **loopback only** — `IPAddressAllow=localhost` means Apache is the
|
||||||
|
only possible client.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Apache vhost + TLS
|
||||||
|
|
||||||
|
**Order matters.** Installing a vhost that references a cert which does not yet
|
||||||
|
exist breaks `apache2ctl configtest`, which blocks reloads for *every* site on
|
||||||
|
the box. So: DNS → cert → full vhost.
|
||||||
|
|
||||||
|
### 5a. DNS
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dig +short vm.example.com A # must return your ORIGIN ip
|
||||||
|
```
|
||||||
|
|
||||||
|
Behind Cloudflare, set the record to **DNS only (grey cloud)** first, or the
|
||||||
|
HTTP-01 challenge is intercepted by the edge.
|
||||||
|
|
||||||
|
### 5b. Minimal :80 vhost, so certbot can answer the challenge
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo install -d -o www-data -g www-data -m 755 /var/www/vm.example.com/public_html
|
||||||
|
sudo cp apache/vm.txt3.net-step1.conf /etc/apache2/sites-available/vm.example.com.conf
|
||||||
|
# edit ServerName / paths / IPs to match your host
|
||||||
|
sudo a2ensite vm.example.com
|
||||||
|
sudo apache2ctl configtest && sudo systemctl reload apache2
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5c. Issue the certificate
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo certbot certonly --webroot -w /var/www/vm.example.com/public_html \
|
||||||
|
-d vm.example.com --cert-name vm.example.com
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5d. Full vhost — HTTPS + reverse proxy
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo a2enmod proxy proxy_http headers rewrite ssl
|
||||||
|
sudo cp apache/vm.txt3.net.conf /etc/apache2/sites-available/vm.example.com.conf
|
||||||
|
# edit ServerName, cert paths, IPs
|
||||||
|
sudo install -d /var/www/vm.example.com/public_html/.well-known/acme-challenge
|
||||||
|
sudo apache2ctl configtest && sudo systemctl reload apache2
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5e. Verify through the real hostname
|
||||||
|
|
||||||
|
Not localhost — the whole point is to exercise Apache, TLS and the proxy:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
R="--resolve vm.example.com:443:YOUR.ORIGIN.IP"
|
||||||
|
curl -s $R https://vm.example.com/healthz
|
||||||
|
curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.example.com/login
|
||||||
|
curl -s $R -D- -o /dev/null https://vm.example.com/login | grep -i 'strict-transport\|content-security'
|
||||||
|
```
|
||||||
|
|
||||||
|
Behind a CDN, `curl https://host/` tests the *CDN*, not your origin. Always
|
||||||
|
`--resolve` to the origin IP when verifying a change.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Backfill existing voicemails
|
||||||
|
|
||||||
|
```bash
|
||||||
|
V=/opt/vm-transcribe/venv/bin/python3
|
||||||
|
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --dry-run # preview
|
||||||
|
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --limit 5 # trial
|
||||||
|
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py # all
|
||||||
|
```
|
||||||
|
|
||||||
|
Idempotent — safe to re-run; it resumes rather than duplicating. Expect a
|
||||||
|
`no_speech` count: 44-byte WAVs are hung-up calls with no audio, not failures.
|
||||||
|
|
||||||
|
Reckon on ~1.5 s per message plus transcription time; 157 messages took 3m40s
|
||||||
|
on 4 cores.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Log in
|
||||||
|
|
||||||
|
Browse to `https://vm.example.com/` and log in with a **mailbox number and its
|
||||||
|
existing voicemail PIN** from `voicemail.conf`. No new passwords are created.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Uninstall
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo systemctl disable --now vm-portal
|
||||||
|
sudo rm /etc/systemd/system/vm-portal.service && sudo systemctl daemon-reload
|
||||||
|
sudo a2dissite vm.example.com && sudo systemctl reload apache2
|
||||||
|
# restore the original mailcmd
|
||||||
|
sudo cp /etc/asterisk/voicemail.conf.bak-<timestamp> /etc/asterisk/voicemail.conf
|
||||||
|
sudo asterisk -rx 'voicemail reload'
|
||||||
|
sudo rm -rf /opt/vm-transcribe # keep /var/lib/vm-transcribe for the data
|
||||||
|
```
|
||||||
138
docs/OPERATIONS.md
Normal file
138
docs/OPERATIONS.md
Normal file
@ -0,0 +1,138 @@
|
|||||||
|
# Operations
|
||||||
|
|
||||||
|
Day-to-day running: backups, monitoring, common problems, upgrades.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Service status
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo systemctl status vm-portal # the portal
|
||||||
|
sudo journalctl -u vm-portal -n 50 # portal logs (uvicorn access/startup)
|
||||||
|
sudo tail -f /var/log/asterisk/vm_mailcmd.log # per-voicemail pipeline log
|
||||||
|
```
|
||||||
|
|
||||||
|
Health of the portal (works locally or via the real hostname):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s http://127.0.0.1:8099/healthz # {"ok":true,"messages":N}
|
||||||
|
curl -s --resolve vm.txt3.net:443:ORIGIN_IP https://vm.txt3.net/healthz
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Logs to watch
|
||||||
|
|
||||||
|
| Signal | Where | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `transcribed …` | vm_mailcmd.log | a voicemail was processed OK |
|
||||||
|
| `stored message for mailbox …` | vm_mailcmd.log | it also reached the portal DB |
|
||||||
|
| `no audio attachment; relaying original` | vm_mailcmd.log | Asterisk sent a text-only notice (missed-call style) — expected |
|
||||||
|
| `Telegram disabled` / `no route` | vm_mailcmd.log | Telegram skipped (per config) — expected if unset |
|
||||||
|
| `relaying original, mailcmd error` | vm_mailcmd.log | **pipeline threw**; original email was preserved (fail-safe worked) |
|
||||||
|
| `status=sent` | /var/log/mail.log | the enriched email left Postfix |
|
||||||
|
| `452 4.3.1 Insufficient system storage` | /var/log/mail.log | **/var is full** — see below |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Backups
|
||||||
|
|
||||||
|
Two things to back up — the code is reproducible, the *data* is not:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# database + audio (once or twice a day is plenty)
|
||||||
|
sudo -u asterisk tar czf /backup/vm-$(date +%F).tgz \
|
||||||
|
-C / var/lib/vm-transcribe/voicemail.db var/lib/vm-transcribe/audio
|
||||||
|
|
||||||
|
# config (the live, secret-bearing files)
|
||||||
|
sudo tar czf /backup/vm-conf-$(date +%F).tgz \
|
||||||
|
/opt/vm-transcribe/telegram.conf /opt/vm-transcribe/contacts.conf
|
||||||
|
```
|
||||||
|
|
||||||
|
The whisper model cache (`/opt/vm-transcribe/models`) is reproducible — no need
|
||||||
|
to back it up, just re-run `install.sh`.
|
||||||
|
|
||||||
|
Restoring: stop the service, extract, `systemctl start vm-portal`. The DB schema
|
||||||
|
is created on first connect, but **preserve the existing file** to keep history.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Disk
|
||||||
|
|
||||||
|
`/var/lib/vm-transcribe/audio` grows with every voicemail. At ~290 KB per
|
||||||
|
WAV (8000 Hz, mono, 23 s) that is ~12 MB per 40 messages. The portal's
|
||||||
|
`max_inbox` per-user auto-delete keeps individual mailboxes bounded, but the
|
||||||
|
*total* store only shrinks when messages are deleted (and their audio blob is
|
||||||
|
unreferenced by any other message).
|
||||||
|
|
||||||
|
A full `/var` is the single most common cause of "voicemail emails stopped
|
||||||
|
arriving" — Postfix rejects everything with `452 4.3.1 Insufficient system
|
||||||
|
storage`. Check `df -h /var` first when the pipeline looks dead.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Upgrading
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git -C /home/jp/Work/asterisk-voicemail pull
|
||||||
|
cd /home/jp/Work/asterisk-voicemail
|
||||||
|
sudo cp src/*.py /opt/vm-transcribe/
|
||||||
|
sudo /opt/vm-transcribe/venv/bin/pip install -U faster-whisper # occasionally
|
||||||
|
sudo systemctl restart vm-portal
|
||||||
|
```
|
||||||
|
|
||||||
|
`install.sh` is idempotent-ish for the first install but is **not** a general
|
||||||
|
upgrade tool — it backs up `voicemail.conf` each run, so avoid re-running it
|
||||||
|
blindly. For upgrades, copy `src/*.py` as above.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Common problems
|
||||||
|
|
||||||
|
**No transcription email arrived.**
|
||||||
|
1. `df -h /var` — full disk blocks Postfix.
|
||||||
|
2. `sudo grep '^mailcmd' /etc/asterisk/voicemail.conf` — is it our script?
|
||||||
|
3. `sudo tail /var/log/asterisk/vm_mailcmd.log` — look for `relaying original, mailcmd error`.
|
||||||
|
4. `sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 -c 'import faster_whisper'` — is the venv intact?
|
||||||
|
|
||||||
|
**Portal returns 502 / can't connect.**
|
||||||
|
The uvicorn backend is down or not on loopback: `sudo systemctl status vm-portal`;
|
||||||
|
`ss -ltnp | grep 8099`. Also confirm Apache has `proxy`/`proxy_http` enabled.
|
||||||
|
|
||||||
|
**Login fails for a real mailbox.**
|
||||||
|
The PIN in `voicemail.conf` is what the portal checks — not anything in the
|
||||||
|
portal. If you changed a voicemail PIN, that change is picked up immediately
|
||||||
|
(the file is reparsed each login). If the mailbox line is commented out or in a
|
||||||
|
context the parser doesn't reach, login fails. Test the parser directly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||||
|
-c "import sys; sys.path.insert(0,'/opt/vm-transcribe'); import vm_auth; print(vm_auth.check_login('7940','5159'))"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Certbot renewal.**
|
||||||
|
Renewal runs from `/etc/cron.d/certbot`; `certbot.timer` is masked (normal on
|
||||||
|
Debian's package). Test non-destructively in the background (it can be slow):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo certbot renew --cert-name vm.txt3.net --dry-run &
|
||||||
|
```
|
||||||
|
|
||||||
|
**I locked myself out testing lockout.**
|
||||||
|
Restart the service: `sudo systemctl restart vm-portal`. In-memory counters clear.
|
||||||
|
|
||||||
|
**Backfill shows many `no_speech`.**
|
||||||
|
Expected. 44-byte WAVs (`duration=0`) are hung-up calls with no audio. Not a bug.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Re-running the backfill
|
||||||
|
|
||||||
|
Idempotent — safe any time:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_import.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Useful after importing a large batch of new spool messages, or after a fresh
|
||||||
|
install on a box with existing voicemails.
|
||||||
117
docs/SECURITY.md
Normal file
117
docs/SECURITY.md
Normal file
@ -0,0 +1,117 @@
|
|||||||
|
# Security
|
||||||
|
|
||||||
|
Threat model and the hardening already in place. Read before exposing the portal
|
||||||
|
on a hostile network.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What we are protecting
|
||||||
|
|
||||||
|
Voicemail is sensitive: it contains callers' names, numbers, and the content of
|
||||||
|
their messages — often personal or commercial. The portal also proves who
|
||||||
|
*isn't* a mailbox owner (knowing a PIN is the only gate). Breaches of concern:
|
||||||
|
|
||||||
|
1. An attacker reads someone else's voicemail (broken authz or IDOR).
|
||||||
|
2. An attacker brute-forces a 4-digit PIN at scale.
|
||||||
|
3. A stored credential (Telegram token, contacts app password) leaks.
|
||||||
|
4. XSS / injection poisons the HTML shown to a user.
|
||||||
|
5. The recording audio leaks off-box.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What is already done
|
||||||
|
|
||||||
|
**Authentication — mailbox PIN from voicemail.conf.**
|
||||||
|
No new password store to breach; the portal trusts Asterisk's PINs. Sessions are
|
||||||
|
DB-backed (random token, `Secure`+`HttpOnly`+`SameSite=lax`, 12 h). Logout and
|
||||||
|
expiry are enforced server-side, not merely by dropping the cookie.
|
||||||
|
|
||||||
|
**Authorization — every query is mailbox-scoped.**
|
||||||
|
`vm_store` and `vm_web` filter every read and delete by the session's mailbox.
|
||||||
|
Cross-mailbox reads/deletes return **404**. There is a test that asserts this
|
||||||
|
([TESTING.md](TESTING.md)). A mailbox owner can only ever see their own messages.
|
||||||
|
|
||||||
|
**Brute-force lockout.**
|
||||||
|
5 failed attempts per `(mailbox, IP)` → 15-minute lockout. The correct PIN is
|
||||||
|
also refused while locked. State is in-memory (single worker); a restart clears
|
||||||
|
it.
|
||||||
|
|
||||||
|
**Transport — TLS terminated at Apache, with headers.**
|
||||||
|
`Strict-Transport-Security`, `X-Frame-Options: DENY`, `X-Content-Type-Options:
|
||||||
|
nosniff`, `Referrer-Policy`, and a tight CSP:
|
||||||
|
|
||||||
|
```
|
||||||
|
default-src 'self'; style-src 'self' 'unsafe-inline';
|
||||||
|
media-src 'self'; img-src 'self' data:;
|
||||||
|
script-src 'none'; frame-ancestors 'none';
|
||||||
|
base-uri 'none'; form-action 'self'
|
||||||
|
```
|
||||||
|
|
||||||
|
**No JavaScript in the portal at all**, which is what lets `script-src 'none'`
|
||||||
|
be genuinely enforceable — there is nothing to inject.
|
||||||
|
|
||||||
|
**Network exposure is minimal.**
|
||||||
|
The uvicorn backend binds **loopback only** (`IPAddressAllow=localhost` in the
|
||||||
|
unit). The only ingress is Apache. The systemd unit also sets
|
||||||
|
`NoNewPrivileges`, `ProtectSystem=full`, `ProtectHome`, `PrivateTmp`,
|
||||||
|
`ProtectKernelTunables`, `ProtectControlGroups`, `RestrictSUIDSGID`, and writable
|
||||||
|
paths are limited to `/var/lib/vm-transcribe` and `/var/log/asterisk`.
|
||||||
|
|
||||||
|
**Secrets are separated and git-ignored.**
|
||||||
|
`telegram.conf`, `contacts.conf`, `*.db`, `audio/` are in `.gitignore`. The
|
||||||
|
repo ships `*.example` templates only. Both secret files are mode `640 root:asterisk`.
|
||||||
|
|
||||||
|
**Fail-safe preserves mail.**
|
||||||
|
A pipeline exception relays the original Asterisk message unchanged — we never
|
||||||
|
lose a notification to a bug we introduced.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Residual risks and how to close them
|
||||||
|
|
||||||
|
**(R1) 4-digit PINs.** Lockout helps but a distributed attack from many IPs
|
||||||
|
defeats per-IP throttling. If the portal is on a hostile network, put it behind
|
||||||
|
your VPN (there is already a `t01.vpn.conf` on this host) or fail2ban the Apache
|
||||||
|
access log. Recommended.
|
||||||
|
|
||||||
|
**(R2) No TLS on the loopback hop.** Apache→uvicorn is plain HTTP on localhost.
|
||||||
|
Acceptable (same host, no network path), but if you ever run uvicorn on another
|
||||||
|
host, use a unix socket or mTLS.
|
||||||
|
|
||||||
|
**(R3) Session token in SQLite.** If the DB file is stolen, sessions are
|
||||||
|
replayable until they expire. DB is `640 asterisk:asterisk` and not web-served.
|
||||||
|
For higher assurance, store sessions in a server-side cache with shorter TTLs.
|
||||||
|
|
||||||
|
**(R4) Contact resolver tokens.** A Google OAuth token or a CardDAV app password
|
||||||
|
grants read access to your address book. Scope the Google token to
|
||||||
|
`contacts.readonly`, use a dedicated app password (never your login password),
|
||||||
|
and keep these files `640`. Prefer the `file` backend (a periodic vCard export) —
|
||||||
|
no live token at all.
|
||||||
|
|
||||||
|
**(R5) Telegram bot token.** Whoever holds it can post as your bot. Keep
|
||||||
|
`telegram.conf` `640 root:asterisk`; rotate via @BotFather if leaked.
|
||||||
|
|
||||||
|
**(R6) Content-addressed audio filenames.** The sha256 of the audio is in the URL
|
||||||
|
(`/audio/<sha>`). An attacker who guesses a sha could fetch that recording
|
||||||
|
without a session. The portal checks the session's mailbox *owns* that message
|
||||||
|
before serving, so this is not directly exploitable — but consider a random
|
||||||
|
per-message token instead of the content hash if you want defence in depth.
|
||||||
|
|
||||||
|
**(R7) CDN / edge.** If `vm.txt3.net` is orange-clouded at Cloudflare, recording
|
||||||
|
audio streams through the edge. Grey-cloud it (DNS only) to keep voice data off
|
||||||
|
the CDN. Your call.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Obtaining a Google token (if you use the `google` backend)
|
||||||
|
|
||||||
|
1. Google Cloud console → OAuth consent screen (External) → add your account as
|
||||||
|
a test user.
|
||||||
|
2. Credentials → OAuth client ID → **Desktop app**.
|
||||||
|
3. Scope `https://www.googleapis.com/auth/contacts.readonly`.
|
||||||
|
4. Authorize once (the token is written to `google-token.json`, mode `640`).
|
||||||
|
The People API is used, not CardDAV, because basic auth / app passwords were
|
||||||
|
disabled by Google on **2024-09-30**.
|
||||||
|
|
||||||
|
There is no support for an app-password read of Google Contacts — it does not
|
||||||
|
work. Use the `file` or `google` backend.
|
||||||
159
docs/TESTING.md
Normal file
159
docs/TESTING.md
Normal file
@ -0,0 +1,159 @@
|
|||||||
|
# Testing
|
||||||
|
|
||||||
|
How to verify each part of the system. Do the mailcmd tests **as the `asterisk`
|
||||||
|
user**, not as yourself — that's where permission bugs hide.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0. Pre-checks
|
||||||
|
|
||||||
|
```bash
|
||||||
|
df -h /var # must not be full (Postfix 452)
|
||||||
|
grep '^mailcmd' /etc/asterisk/voicemail.conf
|
||||||
|
sudo systemctl is-active vm-portal
|
||||||
|
curl -s http://127.0.0.1:8099/healthz
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Unit tests (no sudo, no asterisk)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /home/jp/Work/asterisk-voicemail
|
||||||
|
./venv/bin/python tests/test_telegram.py # routing, caption clipping, opus transcode
|
||||||
|
./venv/bin/python tests/test_contacts.py # vCard/CSV parse, digit-normalised match
|
||||||
|
```
|
||||||
|
|
||||||
|
`test_telegram.py` asserts: per-mailbox → one chat; comma list → N chats;
|
||||||
|
group id negative; unmapped → default; `enabled=no` → no route; caption clipped
|
||||||
|
to 1017 chars on a word boundary when over 1024; a 374 KB wav → 64 KB `OggS`
|
||||||
|
opus.
|
||||||
|
|
||||||
|
`test_contacts.py` asserts: `+447700900123`, `07700900123`, `447700900123` all
|
||||||
|
match one contact (last-9-digit key); multi-TEL cards; Google CSV `:::` split.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Build a fake voicemail and run the real pipeline
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# needs ffmpeg; uses test_vm.wav shipped in the repo (or any wav)
|
||||||
|
python3 tests/make_test_mail.py test_vm.wav you@example.com > /tmp/t.eml
|
||||||
|
sudo chmod 644 /tmp/t.eml
|
||||||
|
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||||
|
/opt/vm-transcribe/vm_mailcmd.py < /tmp/t.eml
|
||||||
|
sudo tail -5 /var/log/asterisk/vm_mailcmd.log
|
||||||
|
```
|
||||||
|
|
||||||
|
Expect `transcribed … chars` then `sent enriched notification to you@example.com`.
|
||||||
|
Then confirm real delivery:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo grep "to=<you@example.com>" /var/log/mail.log | tail -1 # status=sent
|
||||||
|
```
|
||||||
|
|
||||||
|
### Fail-safe: a message with no audio
|
||||||
|
|
||||||
|
```bash
|
||||||
|
printf 'From: a@b\nTo: you@example.com\nSubject: no audio\n\nbody\n' > /tmp/n.eml
|
||||||
|
sudo chmod 644 /tmp/n.eml
|
||||||
|
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||||
|
/opt/vm-transcribe/vm_mailcmd.py < /tmp/n.eml
|
||||||
|
# log must show: "no audio attachment; relaying original"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Fail-safe: a thrown error relays the original
|
||||||
|
|
||||||
|
Temporarily point `VM_DB` at an unreadable path; the script should fall back to
|
||||||
|
relaying the original Asterisk message and log `relaying original, mailcmd error`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Telegram (needs a real token)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py ids
|
||||||
|
# each recipient must have messaged the bot once (/start) to appear
|
||||||
|
sudo /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py test 1001
|
||||||
|
# sends a real voice note + summary; exits non-zero on any failure
|
||||||
|
```
|
||||||
|
|
||||||
|
Offline, the routing logic is covered by `tests/test_telegram.py` so you don't
|
||||||
|
need a token to verify the config parser.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Contacts
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||||
|
/opt/vm-transcribe/vm_contacts.py '<07700900123>'
|
||||||
|
# resolves against the configured backend; prints name or the raw caller id
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Portal — auth, authz, playback, delete
|
||||||
|
|
||||||
|
Run the app locally (or against the live service) and exercise it with curl.
|
||||||
|
Use `VM_INSECURE_COOKIE=1` only for plain-HTTP local tests so the `Secure`
|
||||||
|
cookie can be set.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# login ok
|
||||||
|
curl -s -c /tmp/j -o /dev/null -w '%{redirect_url}\n' \
|
||||||
|
-d 'mailbox=7940&pin=5159' http://127.0.0.1:8099/login
|
||||||
|
|
||||||
|
# wrong pin rejected
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' -d 'mailbox=7940&pin=1111' http://127.0.0.1:8099/login
|
||||||
|
|
||||||
|
# authenticated list
|
||||||
|
curl -s -b /tmp/j http://127.0.0.1:8099/ | grep -c 'class="card'
|
||||||
|
|
||||||
|
# audio streams
|
||||||
|
curl -s -b /tmp/j -o /tmp/x.wav http://127.0.0.1:8099/audio/1
|
||||||
|
file /tmp/x.wav # RIFF WAVE, 8000 Hz mono
|
||||||
|
|
||||||
|
# delete
|
||||||
|
curl -s -b /tmp/j -X POST -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8099/delete/1
|
||||||
|
```
|
||||||
|
|
||||||
|
### Authorization (IDOR) — must 404
|
||||||
|
|
||||||
|
A session for mailbox A must never reach mailbox B's data. With a cookie for
|
||||||
|
mailbox 7940, hitting `/audio/<id-owned-by-1001>` and `/delete/<id-owned-by-1001>`
|
||||||
|
must both return **404/303**, never serve or delete the other mailbox's message.
|
||||||
|
Add this assertion whenever you change `vm_store` or `vm_web`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Portal over the real URL (TLS + proxy)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
R="--resolve vm.txt3.net:443:ORIGIN.IP"
|
||||||
|
curl -s $R https://vm.txt3.net/healthz
|
||||||
|
curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.txt3.net/login
|
||||||
|
curl -s $R -D- -o /dev/null https://vm.txt3.net/login | grep -iE 'strict-transport|content-security|x-frame'
|
||||||
|
# http -> https redirect
|
||||||
|
curl -s $R -o /dev/null -w '%{redirect_url}\n' http://vm.txt3.net/
|
||||||
|
```
|
||||||
|
|
||||||
|
Behind a CDN, `--resolve` to the **origin** IP; otherwise you are testing the CDN,
|
||||||
|
not your server.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Backfill
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||||
|
/opt/vm-transcribe/vm_import.py --dry-run --limit 5
|
||||||
|
# preview: lists 5 messages with caller + date, no writes
|
||||||
|
|
||||||
|
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||||
|
/opt/vm-transcribe/vm_import.py --limit 3
|
||||||
|
# real: 3 imported, others skipped; re-running is a no-op (idempotent)
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify they appear in the portal and are playable (§5). Expect a `no_speech`
|
||||||
|
count — 44-byte WAVs are hung-up calls, not failures.
|
||||||
62
scripts/install.sh
Executable file
62
scripts/install.sh
Executable file
@ -0,0 +1,62 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Installs the voicemail transcription mailcmd. Run with sudo.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SRC=/home/jp/asterisk-vm
|
||||||
|
DEST=/opt/vm-transcribe
|
||||||
|
|
||||||
|
echo "== creating $DEST"
|
||||||
|
install -d -o asterisk -g asterisk -m 755 "$DEST" "$DEST/models"
|
||||||
|
|
||||||
|
echo "== python venv + faster-whisper"
|
||||||
|
if [ ! -x "$DEST/venv/bin/python3" ]; then
|
||||||
|
python3 -m venv "$DEST/venv"
|
||||||
|
"$DEST/venv/bin/pip" install -q --upgrade pip
|
||||||
|
"$DEST/venv/bin/pip" install -q faster-whisper
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "== installing script"
|
||||||
|
install -o root -g root -m 755 "$SRC/vm_mailcmd.py" "$DEST/vm_mailcmd.py"
|
||||||
|
install -o root -g root -m 755 "$SRC/vm_telegram.py" "$DEST/vm_telegram.py"
|
||||||
|
install -o root -g root -m 755 "$SRC/vm_tg_setup.py" "$DEST/vm_tg_setup.py"
|
||||||
|
chown -R asterisk:asterisk "$DEST/models"
|
||||||
|
|
||||||
|
echo "== telegram config (not overwritten if it already exists)"
|
||||||
|
if [ ! -f "$DEST/telegram.conf" ]; then
|
||||||
|
install -o root -g asterisk -m 640 "$SRC/telegram.conf" "$DEST/telegram.conf"
|
||||||
|
echo " created $DEST/telegram.conf - add your bot token and chat IDs"
|
||||||
|
else
|
||||||
|
echo " kept existing $DEST/telegram.conf"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "== log file"
|
||||||
|
touch /var/log/asterisk/vm_mailcmd.log
|
||||||
|
chown asterisk:asterisk /var/log/asterisk/vm_mailcmd.log
|
||||||
|
chmod 640 /var/log/asterisk/vm_mailcmd.log
|
||||||
|
|
||||||
|
echo "== pre-downloading the whisper model as the asterisk user"
|
||||||
|
sudo -u asterisk "$DEST/venv/bin/python3" - <<EOF
|
||||||
|
from faster_whisper import WhisperModel
|
||||||
|
WhisperModel("base.en", device="cpu", compute_type="int8", download_root="$DEST/models")
|
||||||
|
print("model cached")
|
||||||
|
EOF
|
||||||
|
|
||||||
|
echo "== patching /etc/asterisk/voicemail.conf"
|
||||||
|
CONF=/etc/asterisk/voicemail.conf
|
||||||
|
cp -a "$CONF" "$CONF.bak-$(date +%Y%m%d%H%M%S)"
|
||||||
|
python3 - <<EOF
|
||||||
|
import re
|
||||||
|
p="$CONF"
|
||||||
|
s=open(p).read()
|
||||||
|
old="mailcmd=sendmail -t"
|
||||||
|
new="mailcmd=$DEST/venv/bin/python3 $DEST/vm_mailcmd.py"
|
||||||
|
assert s.count(old)==1, "anchor not found exactly once: %d" % s.count(old)
|
||||||
|
s=s.replace(old,new)
|
||||||
|
open(p,"w").write(s)
|
||||||
|
print("mailcmd set to:",new)
|
||||||
|
EOF
|
||||||
|
|
||||||
|
echo "== reloading asterisk voicemail"
|
||||||
|
asterisk -rx "voicemail reload" || asterisk -rx "module reload app_voicemail"
|
||||||
|
|
||||||
|
echo "DONE. Watch: tail -f /var/log/asterisk/vm_mailcmd.log"
|
||||||
67
src/vm_auth.py
Normal file
67
src/vm_auth.py
Normal file
@ -0,0 +1,67 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Voicemail portal - parses Asterisk voicemail.conf for mailbox auth.
|
||||||
|
|
||||||
|
voicemail.conf mailbox lines look like:
|
||||||
|
7940 => 5159,Jamie,jp@txt3.com ; 01273 805515
|
||||||
|
mailbox => password,name,email,pager,options
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
|
||||||
|
VM_CONF = os.environ.get("VM_ASTERISK_CONF", "/etc/asterisk/voicemail.conf")
|
||||||
|
|
||||||
|
# Sections that are not mailbox contexts
|
||||||
|
_NON_CONTEXT = {"general", "zonemessages"}
|
||||||
|
|
||||||
|
|
||||||
|
def parse_mailboxes(path=None):
|
||||||
|
"""Return {mailbox: {"pin","name","email","context"}}."""
|
||||||
|
p = path or VM_CONF
|
||||||
|
out = {}
|
||||||
|
context = "default"
|
||||||
|
try:
|
||||||
|
with open(p, encoding="utf-8", errors="replace") as fh:
|
||||||
|
for raw in fh:
|
||||||
|
line = raw.strip()
|
||||||
|
if not line or line.startswith(";") or line.startswith("#"):
|
||||||
|
continue
|
||||||
|
m = re.match(r"^\[([^\]]+)\]", line)
|
||||||
|
if m:
|
||||||
|
context = m.group(1).strip()
|
||||||
|
continue
|
||||||
|
if context in _NON_CONTEXT:
|
||||||
|
continue
|
||||||
|
m = re.match(r"^(\d+)\s*=>\s*(.+)$", line)
|
||||||
|
if not m:
|
||||||
|
continue
|
||||||
|
mbox, rest = m.group(1), m.group(2)
|
||||||
|
rest = rest.split(";", 1)[0].strip() # strip trailing comment
|
||||||
|
parts = [x.strip() for x in rest.split(",")]
|
||||||
|
pin = parts[0] if parts else ""
|
||||||
|
name = parts[1] if len(parts) > 1 else mbox
|
||||||
|
mail = parts[2] if len(parts) > 2 else ""
|
||||||
|
out[mbox] = {"pin": pin, "name": name, "email": mail,
|
||||||
|
"context": context}
|
||||||
|
except FileNotFoundError:
|
||||||
|
pass
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def check_login(mailbox, pin, path=None):
|
||||||
|
"""Constant-time-ish PIN check. Returns the mailbox dict or None."""
|
||||||
|
import hmac
|
||||||
|
boxes = parse_mailboxes(path)
|
||||||
|
info = boxes.get(str(mailbox).strip())
|
||||||
|
if not info or not info.get("pin"):
|
||||||
|
return None
|
||||||
|
if hmac.compare_digest(str(info["pin"]), str(pin).strip()):
|
||||||
|
return info
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
for mb, i in parse_mailboxes().items():
|
||||||
|
print("%-8s %-18s %-28s ctx=%s pin=%s"
|
||||||
|
% (mb, i["name"], i["email"], i["context"], "*" * len(i["pin"])))
|
||||||
334
src/vm_contacts.py
Normal file
334
src/vm_contacts.py
Normal file
@ -0,0 +1,334 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Caller-ID -> contact name resolution for Asterisk voicemail notifications.
|
||||||
|
|
||||||
|
Backends (tried in the order given by contacts.conf 'backends='):
|
||||||
|
file - local vCard (.vcf) or CSV export; no auth, offline, fast.
|
||||||
|
google - Google People API via the Hermes OAuth token.
|
||||||
|
carddav - generic CardDAV with username + app password.
|
||||||
|
|
||||||
|
Everything here is best-effort: a lookup failure returns None and the caller
|
||||||
|
falls back to the raw caller-ID string. Results (hits AND misses) are cached
|
||||||
|
to avoid per-call API traffic.
|
||||||
|
|
||||||
|
NOTE: app-password / basic auth does NOT work against Google - Google disabled
|
||||||
|
it for CardDAV/CalDAV/IMAP on 2024-09-30 and now requires OAuth for CardDAV.
|
||||||
|
The carddav backend is for Nextcloud / Fastmail / iCloud / Radicale etc.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import configparser
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import time
|
||||||
|
import urllib.error
|
||||||
|
import urllib.parse
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
CONF_PATH = os.environ.get("VM_CONTACTS_CONF", "/opt/vm-transcribe/contacts.conf")
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------- helpers
|
||||||
|
def digits_of(s):
|
||||||
|
return re.sub(r"\D", "", s or "")
|
||||||
|
|
||||||
|
|
||||||
|
def extract_number(caller):
|
||||||
|
"""Pull a dialable number out of a CallerID string like 'Dave <079...>'."""
|
||||||
|
if not caller:
|
||||||
|
return ""
|
||||||
|
m = re.search(r"<([^>]+)>", caller)
|
||||||
|
cand = m.group(1) if m else caller
|
||||||
|
return digits_of(cand)
|
||||||
|
|
||||||
|
|
||||||
|
def _key(num_digits, n):
|
||||||
|
d = digits_of(num_digits)
|
||||||
|
return d[-n:] if len(d) >= n else d
|
||||||
|
|
||||||
|
|
||||||
|
class Cache:
|
||||||
|
def __init__(self, path, ttl):
|
||||||
|
self.path, self.ttl, self.data = path, ttl, {}
|
||||||
|
try:
|
||||||
|
with open(path) as fh:
|
||||||
|
self.data = json.load(fh)
|
||||||
|
except Exception:
|
||||||
|
self.data = {}
|
||||||
|
|
||||||
|
def get(self, key):
|
||||||
|
e = self.data.get(key)
|
||||||
|
if not e:
|
||||||
|
return None # unknown -> caller should look up
|
||||||
|
if self.ttl and time.time() - e.get("t", 0) > self.ttl:
|
||||||
|
return None
|
||||||
|
return e # {"t":..., "name": <str or None>}
|
||||||
|
|
||||||
|
def put(self, key, name):
|
||||||
|
self.data[key] = {"t": time.time(), "name": name}
|
||||||
|
try:
|
||||||
|
os.makedirs(os.path.dirname(self.path), exist_ok=True)
|
||||||
|
tmp = self.path + ".tmp"
|
||||||
|
with open(tmp, "w") as fh:
|
||||||
|
json.dump(self.data, fh)
|
||||||
|
os.replace(tmp, self.path)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------- file backend
|
||||||
|
def _parse_vcf(text):
|
||||||
|
"""Return list of (name, [numbers])."""
|
||||||
|
out = []
|
||||||
|
name, nums = None, []
|
||||||
|
for raw in text.splitlines():
|
||||||
|
line = raw.strip()
|
||||||
|
u = line.upper()
|
||||||
|
if u == "BEGIN:VCARD":
|
||||||
|
name, nums = None, []
|
||||||
|
elif u.startswith("FN"):
|
||||||
|
name = line.split(":", 1)[1].strip() if ":" in line else None
|
||||||
|
elif u.startswith("TEL"):
|
||||||
|
if ":" in line:
|
||||||
|
nums.append(line.split(":", 1)[1].strip())
|
||||||
|
elif u == "END:VCARD":
|
||||||
|
if name and nums:
|
||||||
|
out.append((name, nums))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_csv(path):
|
||||||
|
import csv
|
||||||
|
out = []
|
||||||
|
with open(path, newline="", encoding="utf-8", errors="replace") as fh:
|
||||||
|
r = csv.DictReader(fh)
|
||||||
|
cols = r.fieldnames or []
|
||||||
|
name_cols = [c for c in cols if c and ("Name" == c or c.endswith("Name"))]
|
||||||
|
phone_cols = [c for c in cols if c and "Phone" in c and "Value" in c] or \
|
||||||
|
[c for c in cols if c and "Phone" in c]
|
||||||
|
for row in r:
|
||||||
|
name = ""
|
||||||
|
if "Name" in row and row["Name"]:
|
||||||
|
name = row["Name"]
|
||||||
|
else:
|
||||||
|
parts = [row.get(c, "") for c in ("Given Name", "Family Name") if row.get(c)]
|
||||||
|
name = " ".join(parts) or (row.get(name_cols[0], "") if name_cols else "")
|
||||||
|
nums = []
|
||||||
|
for c in phone_cols:
|
||||||
|
v = row.get(c, "")
|
||||||
|
if v:
|
||||||
|
nums.extend(re.split(r"\s*:::\s*|\s*;\s*", v))
|
||||||
|
if name and nums:
|
||||||
|
out.append((name.strip(), nums))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _index(entries, n):
|
||||||
|
idx = {}
|
||||||
|
for name, nums in entries:
|
||||||
|
for num in nums:
|
||||||
|
k = _key(num, n)
|
||||||
|
if k:
|
||||||
|
idx.setdefault(k, name)
|
||||||
|
return idx
|
||||||
|
|
||||||
|
|
||||||
|
def lookup_file(cfg, num_digits, n, log):
|
||||||
|
path = cfg.get("path", "").strip()
|
||||||
|
if not path or not os.path.exists(path):
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
if path.lower().endswith(".csv"):
|
||||||
|
entries = _parse_csv(path)
|
||||||
|
else:
|
||||||
|
with open(path, encoding="utf-8", errors="replace") as fh:
|
||||||
|
entries = _parse_vcf(fh.read())
|
||||||
|
return _index(entries, n).get(_key(num_digits, n))
|
||||||
|
except Exception as e:
|
||||||
|
log("contacts file backend error: %s" % e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------- google backend
|
||||||
|
def _google_access_token(token_path, log):
|
||||||
|
try:
|
||||||
|
with open(token_path) as fh:
|
||||||
|
tok = json.load(fh)
|
||||||
|
except Exception as e:
|
||||||
|
log("google token unreadable (%s): %s" % (token_path, e))
|
||||||
|
return None
|
||||||
|
# try existing token first; refresh if People API 401s
|
||||||
|
at = tok.get("token") or tok.get("access_token")
|
||||||
|
refresh = tok.get("refresh_token")
|
||||||
|
cid = tok.get("client_id")
|
||||||
|
secret = tok.get("client_secret")
|
||||||
|
if at:
|
||||||
|
return at, (refresh, cid, secret, token_path, tok)
|
||||||
|
return _google_refresh((refresh, cid, secret, token_path, tok), log)
|
||||||
|
|
||||||
|
|
||||||
|
def _google_refresh(ctx, log):
|
||||||
|
refresh, cid, secret, token_path, tok = ctx
|
||||||
|
if not (refresh and cid and secret):
|
||||||
|
log("google token missing refresh_token/client_id/client_secret")
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
data = urllib.parse.urlencode({
|
||||||
|
"client_id": cid, "client_secret": secret,
|
||||||
|
"refresh_token": refresh, "grant_type": "refresh_token",
|
||||||
|
}).encode()
|
||||||
|
req = urllib.request.Request("https://oauth2.googleapis.com/token", data=data)
|
||||||
|
with urllib.request.urlopen(req, timeout=20) as r:
|
||||||
|
new = json.loads(r.read().decode())
|
||||||
|
at = new.get("access_token")
|
||||||
|
if at:
|
||||||
|
tok["token"] = at
|
||||||
|
try:
|
||||||
|
with open(token_path, "w") as fh:
|
||||||
|
json.dump(tok, fh)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return at, ctx
|
||||||
|
except Exception as e:
|
||||||
|
log("google token refresh failed: %s" % e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def lookup_google(cfg, num_digits, n, log):
|
||||||
|
token_path = cfg.get("token_path", "").strip()
|
||||||
|
if not token_path or not os.path.exists(token_path):
|
||||||
|
return None
|
||||||
|
got = _google_access_token(token_path, log)
|
||||||
|
if not got:
|
||||||
|
return None
|
||||||
|
at, ctx = got
|
||||||
|
want = _key(num_digits, n)
|
||||||
|
|
||||||
|
def query(access):
|
||||||
|
url = ("https://people.googleapis.com/v1/people:searchContacts"
|
||||||
|
"?query=%s&readMask=names,phoneNumbers"
|
||||||
|
% urllib.parse.quote(num_digits[-7:] or num_digits))
|
||||||
|
req = urllib.request.Request(url, headers={"Authorization": "Bearer %s" % access})
|
||||||
|
with urllib.request.urlopen(req, timeout=20) as r:
|
||||||
|
return json.loads(r.read().decode())
|
||||||
|
|
||||||
|
try:
|
||||||
|
try:
|
||||||
|
d = query(at)
|
||||||
|
except urllib.error.HTTPError as he:
|
||||||
|
if he.code == 401: # refresh once
|
||||||
|
got = _google_refresh(ctx if isinstance(ctx, tuple) and len(ctx) == 5
|
||||||
|
else ctx, log)
|
||||||
|
if not got:
|
||||||
|
return None
|
||||||
|
d = query(got[0])
|
||||||
|
else:
|
||||||
|
raise
|
||||||
|
for res in d.get("results", []):
|
||||||
|
person = res.get("person", {})
|
||||||
|
for ph in person.get("phoneNumbers", []):
|
||||||
|
if _key(ph.get("value", ""), n) == want:
|
||||||
|
names = person.get("names", [])
|
||||||
|
if names:
|
||||||
|
return names[0].get("displayName")
|
||||||
|
except Exception as e:
|
||||||
|
log("google People API error: %s" % e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------- carddav backend
|
||||||
|
def lookup_carddav(cfg, num_digits, n, log):
|
||||||
|
import ssl
|
||||||
|
url = cfg.get("url", "").strip()
|
||||||
|
user = cfg.get("username", "").strip()
|
||||||
|
pw = cfg.get("app_password", "").strip()
|
||||||
|
if not (url and user and pw):
|
||||||
|
return None
|
||||||
|
if "google.com" in url:
|
||||||
|
log("carddav backend points at Google, which rejects app passwords "
|
||||||
|
"since 2024-09-30; use the 'google' or 'file' backend instead")
|
||||||
|
return None
|
||||||
|
|
||||||
|
path = cfg.get("addressbook_path", "").strip()
|
||||||
|
base = url.rstrip("/") + (path if path.startswith("/") else "/" + path if path else "")
|
||||||
|
ctx = None
|
||||||
|
if cfg.get("verify_tls", "yes").lower() in ("no", "false", "0"):
|
||||||
|
ctx = ssl.create_default_context()
|
||||||
|
ctx.check_hostname = False
|
||||||
|
ctx.verify_mode = ssl.CERT_NONE
|
||||||
|
timeout = int(cfg.get("timeout", "20") or 20)
|
||||||
|
auth = base64.b64encode(("%s:%s" % (user, pw)).encode()).decode()
|
||||||
|
|
||||||
|
body = ('<?xml version="1.0"?>'
|
||||||
|
'<C:addressbook-query xmlns:D="DAV:" '
|
||||||
|
'xmlns:C="urn:ietf:params:xml:ns:carddav">'
|
||||||
|
'<D:prop><C:address-data/></D:prop></C:addressbook-query>')
|
||||||
|
try:
|
||||||
|
req = urllib.request.Request(
|
||||||
|
base, data=body.encode(), method="REPORT",
|
||||||
|
headers={"Authorization": "Basic %s" % auth, "Depth": "1",
|
||||||
|
"Content-Type": "application/xml; charset=utf-8"})
|
||||||
|
with urllib.request.urlopen(req, timeout=timeout, context=ctx) as r:
|
||||||
|
xml = r.read().decode("utf-8", "replace")
|
||||||
|
vcards = re.findall(r"BEGIN:VCARD.*?END:VCARD", xml, re.S | re.I)
|
||||||
|
entries = []
|
||||||
|
for v in vcards:
|
||||||
|
entries.extend(_parse_vcf(v.replace(" ", "").replace("\r", "")))
|
||||||
|
return _index(entries, n).get(_key(num_digits, n))
|
||||||
|
except Exception as e:
|
||||||
|
log("carddav backend error: %s" % e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- entrypoint
|
||||||
|
_BACKENDS = {"file": lookup_file, "google": lookup_google, "carddav": lookup_carddav}
|
||||||
|
|
||||||
|
|
||||||
|
def resolve(caller_id, log=print):
|
||||||
|
"""Return a contact name for a CallerID string, or None."""
|
||||||
|
num = extract_number(caller_id)
|
||||||
|
if not num or not os.path.exists(CONF_PATH):
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
cp = configparser.ConfigParser(inline_comment_prefixes=("#", ";"))
|
||||||
|
cp.read(CONF_PATH)
|
||||||
|
if not cp.has_section("contacts") or not cp["contacts"].getboolean("enabled", False):
|
||||||
|
return None
|
||||||
|
g = cp["contacts"]
|
||||||
|
n = int(g.get("match_digits", "9") or 9)
|
||||||
|
cache = Cache(g.get("cache_path", "/var/lib/vm-transcribe/contacts_cache.json"),
|
||||||
|
int(g.get("cache_ttl", "86400") or 86400))
|
||||||
|
|
||||||
|
key = _key(num, n)
|
||||||
|
cached = cache.get(key)
|
||||||
|
if cached is not None:
|
||||||
|
return cached.get("name")
|
||||||
|
|
||||||
|
order = [b.strip() for b in g.get("backends", "file").split(",") if b.strip()]
|
||||||
|
name = None
|
||||||
|
for b in order:
|
||||||
|
fn = _BACKENDS.get(b)
|
||||||
|
if not fn:
|
||||||
|
log("unknown contacts backend: %s" % b)
|
||||||
|
continue
|
||||||
|
sect = cp[b] if cp.has_section(b) else {}
|
||||||
|
try:
|
||||||
|
name = fn(sect, num, n, log)
|
||||||
|
except Exception as e:
|
||||||
|
log("contacts backend %s crashed: %s" % (b, e))
|
||||||
|
name = None
|
||||||
|
if name:
|
||||||
|
log("contacts: %s -> %s (via %s)" % (num, name, b))
|
||||||
|
break
|
||||||
|
|
||||||
|
cache.put(key, name) # cache misses too (name=None)
|
||||||
|
return name
|
||||||
|
except Exception as e:
|
||||||
|
log("contacts resolve error: %s" % e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
import sys
|
||||||
|
print(resolve(sys.argv[1] if len(sys.argv) > 1 else "", log=lambda m: print("[log]", m)))
|
||||||
229
src/vm_import.py
Normal file
229
src/vm_import.py
Normal file
@ -0,0 +1,229 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Backfill existing Asterisk voicemails into the portal database.
|
||||||
|
|
||||||
|
Walks the voicemail spool, reads each message's .txt metadata, transcribes the
|
||||||
|
recording with faster-whisper, summarises it, resolves the caller against
|
||||||
|
contacts, and inserts it into the same SQLite store the live mailcmd writes to.
|
||||||
|
|
||||||
|
Safe to re-run: rows are unique on (mailbox, origtime, callerid), so an
|
||||||
|
interrupted run resumes rather than duplicating. Already-imported messages are
|
||||||
|
skipped without being transcribed again (the expensive part).
|
||||||
|
|
||||||
|
Usage (run as the asterisk user so it can read the spool):
|
||||||
|
vm_import.py --dry-run # show what would be done
|
||||||
|
vm_import.py # import everything
|
||||||
|
vm_import.py --mailbox 7940 # one mailbox
|
||||||
|
vm_import.py --folder INBOX # one folder
|
||||||
|
vm_import.py --limit 10 # first N (useful for a trial run)
|
||||||
|
vm_import.py --no-transcribe # metadata + audio only, no whisper
|
||||||
|
vm_import.py --reverse # newest first
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import configparser
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
|
||||||
|
SPOOL = os.environ.get("VM_SPOOL", "/var/spool/asterisk/voicemail")
|
||||||
|
AUDIO_EXTS = (".wav", ".WAV", ".gsm", ".wav49", ".ogg", ".mp3")
|
||||||
|
|
||||||
|
|
||||||
|
def log(msg):
|
||||||
|
print(msg, flush=True)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_info(path):
|
||||||
|
"""Parse an Asterisk msgNNNN.txt message information file."""
|
||||||
|
out = {}
|
||||||
|
try:
|
||||||
|
cp = configparser.ConfigParser(strict=False, inline_comment_prefixes=None)
|
||||||
|
with open(path, encoding="utf-8", errors="replace") as fh:
|
||||||
|
text = fh.read()
|
||||||
|
cp.read_string(text)
|
||||||
|
if cp.has_section("message"):
|
||||||
|
out = dict(cp["message"])
|
||||||
|
except Exception:
|
||||||
|
# fall back to a naive key=value scrape
|
||||||
|
try:
|
||||||
|
with open(path, encoding="utf-8", errors="replace") as fh:
|
||||||
|
for line in fh:
|
||||||
|
if "=" in line and not line.strip().startswith(";"):
|
||||||
|
k, v = line.split("=", 1)
|
||||||
|
out[k.strip()] = v.strip()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def find_messages(spool, want_mailbox=None, want_folder=None):
|
||||||
|
"""Yield (context, mailbox, folder, basepath, audio_path, info_path)."""
|
||||||
|
if not os.path.isdir(spool):
|
||||||
|
return
|
||||||
|
for context in sorted(os.listdir(spool)):
|
||||||
|
cdir = os.path.join(spool, context)
|
||||||
|
if not os.path.isdir(cdir):
|
||||||
|
continue
|
||||||
|
for mailbox in sorted(os.listdir(cdir)):
|
||||||
|
if want_mailbox and mailbox != want_mailbox:
|
||||||
|
continue
|
||||||
|
mdir = os.path.join(cdir, mailbox)
|
||||||
|
if not os.path.isdir(mdir):
|
||||||
|
continue
|
||||||
|
for folder in sorted(os.listdir(mdir)):
|
||||||
|
if folder in ("tmp",):
|
||||||
|
continue
|
||||||
|
if want_folder and folder != want_folder:
|
||||||
|
continue
|
||||||
|
fdir = os.path.join(mdir, folder)
|
||||||
|
if not os.path.isdir(fdir):
|
||||||
|
continue
|
||||||
|
# group by msgNNNN stem
|
||||||
|
stems = set()
|
||||||
|
for fn in os.listdir(fdir):
|
||||||
|
m = re.match(r"^(msg\d+)\.", fn)
|
||||||
|
if m:
|
||||||
|
stems.add(m.group(1))
|
||||||
|
for stem in sorted(stems):
|
||||||
|
base = os.path.join(fdir, stem)
|
||||||
|
audio = None
|
||||||
|
for ext in AUDIO_EXTS:
|
||||||
|
if os.path.exists(base + ext):
|
||||||
|
audio = base + ext
|
||||||
|
break
|
||||||
|
info = base + ".txt"
|
||||||
|
if audio:
|
||||||
|
yield (context, mailbox, folder, base, audio,
|
||||||
|
info if os.path.exists(info) else None)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--spool", default=SPOOL)
|
||||||
|
ap.add_argument("--mailbox")
|
||||||
|
ap.add_argument("--folder")
|
||||||
|
ap.add_argument("--limit", type=int)
|
||||||
|
ap.add_argument("--dry-run", action="store_true")
|
||||||
|
ap.add_argument("--no-transcribe", action="store_true")
|
||||||
|
ap.add_argument("--reverse", action="store_true")
|
||||||
|
ap.add_argument("--model", default=os.environ.get("VM_WHISPER_MODEL", "base.en"))
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
import vm_store
|
||||||
|
con = vm_store.connect()
|
||||||
|
|
||||||
|
items = list(find_messages(args.spool, args.mailbox, args.folder))
|
||||||
|
if args.reverse:
|
||||||
|
items.reverse()
|
||||||
|
log("found %d message(s) in %s" % (len(items), args.spool))
|
||||||
|
if not items:
|
||||||
|
return 0
|
||||||
|
|
||||||
|
# Which are already imported? Compare on (mailbox, origtime, callerid).
|
||||||
|
existing = set()
|
||||||
|
for r in con.execute("SELECT mailbox, origtime, callerid FROM messages"):
|
||||||
|
existing.add((str(r["mailbox"]), r["origtime"], r["callerid"]))
|
||||||
|
|
||||||
|
# Load whisper once, lazily - it is the slow part.
|
||||||
|
model = None
|
||||||
|
|
||||||
|
def transcribe(path):
|
||||||
|
nonlocal model
|
||||||
|
if model is None:
|
||||||
|
from faster_whisper import WhisperModel
|
||||||
|
log(" loading whisper model %s (cpu/int8)..." % args.model)
|
||||||
|
model = WhisperModel(
|
||||||
|
args.model, device="cpu", compute_type="int8",
|
||||||
|
cpu_threads=max(1, (os.cpu_count() or 2) - 1),
|
||||||
|
download_root=os.environ.get("VM_MODEL_CACHE",
|
||||||
|
"/opt/vm-transcribe/models"))
|
||||||
|
segs, _info = model.transcribe(
|
||||||
|
path, beam_size=1, vad_filter=True,
|
||||||
|
vad_parameters=dict(min_silence_duration_ms=500),
|
||||||
|
condition_on_previous_text=False)
|
||||||
|
return re.sub(r"\s+", " ", " ".join(s.text.strip() for s in segs)).strip()
|
||||||
|
|
||||||
|
# Reuse the live pipeline's summariser / tagger / contact lookup so
|
||||||
|
# backfilled messages look identical to new ones.
|
||||||
|
import vm_mailcmd
|
||||||
|
try:
|
||||||
|
import vm_contacts
|
||||||
|
except Exception:
|
||||||
|
vm_contacts = None
|
||||||
|
|
||||||
|
stats = {"imported": 0, "skipped": 0, "failed": 0, "no_speech": 0}
|
||||||
|
t0 = time.time()
|
||||||
|
todo = items[:args.limit] if args.limit else items
|
||||||
|
|
||||||
|
for i, (context, mailbox, folder, base, audio, info) in enumerate(todo, 1):
|
||||||
|
meta = parse_info(info) if info else {}
|
||||||
|
callerid = meta.get("callerid") or None
|
||||||
|
try:
|
||||||
|
origtime = int(meta.get("origtime") or 0) or int(os.path.getmtime(audio))
|
||||||
|
except Exception:
|
||||||
|
origtime = int(os.path.getmtime(audio))
|
||||||
|
try:
|
||||||
|
duration = int(float(meta.get("duration") or 0)) or None
|
||||||
|
except Exception:
|
||||||
|
duration = None
|
||||||
|
|
||||||
|
tag = "%s/%s/%s" % (mailbox, folder, os.path.basename(base))
|
||||||
|
if (str(mailbox), origtime, callerid) in existing:
|
||||||
|
stats["skipped"] += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
if args.dry_run:
|
||||||
|
log(" [%d/%d] WOULD IMPORT %-28s caller=%-22s %s"
|
||||||
|
% (i, len(todo), tag, callerid or "?",
|
||||||
|
time.strftime("%Y-%m-%d %H:%M", time.localtime(origtime))))
|
||||||
|
stats["imported"] += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
try:
|
||||||
|
transcript = "" if args.no_transcribe else transcribe(audio)
|
||||||
|
if not transcript:
|
||||||
|
stats["no_speech"] += 1
|
||||||
|
summary = vm_mailcmd.summarise(transcript) if transcript else ""
|
||||||
|
intents = vm_mailcmd.detect_intents(transcript) if transcript else []
|
||||||
|
numbers = vm_mailcmd.find_numbers(transcript) if transcript else []
|
||||||
|
|
||||||
|
contact = None
|
||||||
|
if vm_contacts and callerid:
|
||||||
|
try:
|
||||||
|
contact = vm_contacts.resolve(callerid, log=lambda m: None)
|
||||||
|
except Exception:
|
||||||
|
contact = None
|
||||||
|
|
||||||
|
with open(audio, "rb") as fh:
|
||||||
|
audio_bytes = fh.read()
|
||||||
|
|
||||||
|
vm_store.add_message(
|
||||||
|
con, mailbox, callerid=callerid, contact_name=contact,
|
||||||
|
origtime=origtime, duration=duration, transcript=transcript,
|
||||||
|
summary=summary, intents=intents, numbers=numbers,
|
||||||
|
audio_bytes=audio_bytes,
|
||||||
|
audio_ext=os.path.splitext(audio)[1].lstrip(".").lower() or "wav",
|
||||||
|
spool_path=audio, context=context, folder=folder)
|
||||||
|
existing.add((str(mailbox), origtime, callerid))
|
||||||
|
stats["imported"] += 1
|
||||||
|
log(" [%d/%d] %-28s %-22s %3ds %s"
|
||||||
|
% (i, len(todo), tag, (callerid or "?")[:22], duration or 0,
|
||||||
|
(summary[:60] + "...") if len(summary) > 60 else summary))
|
||||||
|
except Exception as e:
|
||||||
|
stats["failed"] += 1
|
||||||
|
log(" [%d/%d] FAILED %s: %s" % (i, len(todo), tag, e))
|
||||||
|
|
||||||
|
con.close()
|
||||||
|
el = time.time() - t0
|
||||||
|
log("\ndone in %dm%02ds - imported=%d skipped=%d no_speech=%d failed=%d"
|
||||||
|
% (el // 60, el % 60, stats["imported"], stats["skipped"],
|
||||||
|
stats["no_speech"], stats["failed"]))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
533
src/vm_mailcmd.py
Executable file
533
src/vm_mailcmd.py
Executable file
@ -0,0 +1,533 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Asterisk voicemail mailcmd replacement.
|
||||||
|
|
||||||
|
Reads the RFC822 message Asterisk pipes to `sendmail -t` on stdin, then:
|
||||||
|
a) transcribes the attached voicemail audio with faster-whisper (CPU, int8)
|
||||||
|
b) produces a short extractive summary + callback-number / intent hints
|
||||||
|
c) rebuilds the message as multipart/mixed containing a
|
||||||
|
multipart/alternative (text/plain + styled text/html) and the original
|
||||||
|
audio attachment, and delivers it via SMTP on localhost:25.
|
||||||
|
|
||||||
|
Fail-safe: any error at all -> the ORIGINAL message is relayed unchanged, so a
|
||||||
|
voicemail notification is never lost. All diagnostics go to LOG_PATH.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import email
|
||||||
|
import email.policy
|
||||||
|
import email.utils
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import smtplib
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import traceback
|
||||||
|
from datetime import datetime
|
||||||
|
from email.message import EmailMessage
|
||||||
|
from html import escape
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ settings
|
||||||
|
VENV_PY = "/opt/vm-transcribe/venv/bin/python3"
|
||||||
|
MODEL_SIZE = os.environ.get("VM_WHISPER_MODEL", "base.en")
|
||||||
|
MODEL_CACHE = os.environ.get("VM_MODEL_CACHE", "/opt/vm-transcribe/models")
|
||||||
|
SMTP_HOST = "localhost"
|
||||||
|
SMTP_PORT = 25
|
||||||
|
LOG_PATH = os.environ.get("VM_LOG", "/var/log/asterisk/vm_mailcmd.log")
|
||||||
|
MAX_SECONDS = 900 # ignore absurdly long recordings
|
||||||
|
BRAND = "Voicemail"
|
||||||
|
ACCENT = "#2f6fed"
|
||||||
|
|
||||||
|
|
||||||
|
def log(msg):
|
||||||
|
try:
|
||||||
|
with open(LOG_PATH, "a") as fh:
|
||||||
|
fh.write("%s %s\n" % (datetime.now().isoformat(timespec="seconds"), msg))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------- transcription
|
||||||
|
def transcribe(wav_path):
|
||||||
|
"""Return transcript text (may be '')."""
|
||||||
|
from faster_whisper import WhisperModel
|
||||||
|
|
||||||
|
model = WhisperModel(
|
||||||
|
MODEL_SIZE,
|
||||||
|
device="cpu",
|
||||||
|
compute_type="int8",
|
||||||
|
cpu_threads=max(1, (os.cpu_count() or 2) - 1),
|
||||||
|
download_root=MODEL_CACHE,
|
||||||
|
)
|
||||||
|
segments, info = model.transcribe(
|
||||||
|
wav_path,
|
||||||
|
beam_size=1,
|
||||||
|
vad_filter=True,
|
||||||
|
vad_parameters=dict(min_silence_duration_ms=500),
|
||||||
|
condition_on_previous_text=False,
|
||||||
|
)
|
||||||
|
parts = []
|
||||||
|
for seg in segments:
|
||||||
|
if seg.start > MAX_SECONDS:
|
||||||
|
break
|
||||||
|
parts.append(seg.text.strip())
|
||||||
|
text = " ".join(p for p in parts if p).strip()
|
||||||
|
text = re.sub(r"\s+", " ", text)
|
||||||
|
log("transcribed %.1fs audio, lang=%s, %d chars"
|
||||||
|
% (getattr(info, "duration", 0.0), getattr(info, "language", "?"), len(text)))
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------- summarise
|
||||||
|
_STOP = set("""a an and are as at be been but by for from had has have he her his i if in into is it
|
||||||
|
its me my not of on or our she so than that the their them then there these they this to was we were
|
||||||
|
what when which who will with would you your um uh yeah okay ok just like know really got get""".split())
|
||||||
|
|
||||||
|
_PHONE_RE = re.compile(
|
||||||
|
r"(?:(?:\+|00)\d{1,3}[ .\-]?)?(?:\(?\d{2,5}\)?[ .\-]?){2,5}\d{2,4}")
|
||||||
|
_SPOKEN_DIGITS = {
|
||||||
|
"zero": "0", "oh": "0", "one": "1", "two": "2", "three": "3", "four": "4",
|
||||||
|
"five": "5", "six": "6", "seven": "7", "eight": "8", "nine": "9",
|
||||||
|
"double": "", "triple": "",
|
||||||
|
}
|
||||||
|
|
||||||
|
_INTENTS = [
|
||||||
|
("Call back requested", r"\b(call (me|us|him|her|them)? ?back|give (me|us) a (call|ring|bell)|"
|
||||||
|
r"ring (me|us) back|get back to (me|us)|reach (me|us))\b"),
|
||||||
|
("Urgent", r"\b(urgent|asap|as soon as possible|emergency|immediately|straight away|"
|
||||||
|
r"right away|critical)\b"),
|
||||||
|
("Appointment", r"\b(appointment|meeting|schedule|reschedule|booking|book(ed)? (you )?in|"
|
||||||
|
r"confirm(ing)? (the|your)? ?(time|date|slot))\b"),
|
||||||
|
("Payment / invoice", r"\b(invoice|payment|pay(ing|ment)? ?(due|late)?|billing|account balance|"
|
||||||
|
r"overdue|quote|estimate)\b"),
|
||||||
|
("Delivery", r"\b(deliver(y|ies)?|parcel|package|courier|dispatch|shipment)\b"),
|
||||||
|
("Complaint / issue", r"\b(complaint|complain|problem|issue|not working|broken|fault|unhappy|"
|
||||||
|
r"disappointed)\b"),
|
||||||
|
("Cancellation", r"\b(cancel(l(ed|ing))?|can't make it|cannot make it|postpone)\b"),
|
||||||
|
("Sales / marketing", r"\b(special offer|promotion|no obligation|free quote|marketing|"
|
||||||
|
r"we noticed your website|SEO)\b"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _split_sentences(text):
|
||||||
|
parts = re.split(r"(?<=[.!?])\s+", text)
|
||||||
|
out = []
|
||||||
|
for p in parts:
|
||||||
|
p = p.strip()
|
||||||
|
if not p:
|
||||||
|
continue
|
||||||
|
# very long run-ons with no punctuation: chop on discourse markers
|
||||||
|
if len(p) > 220:
|
||||||
|
out.extend(s.strip() for s in re.split(r"\s+(?:and then|but|so|however)\s+", p) if s.strip())
|
||||||
|
else:
|
||||||
|
out.append(p)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def find_numbers(text):
|
||||||
|
"""Callback numbers, both digit-written and spoken-out."""
|
||||||
|
found = []
|
||||||
|
for m in _PHONE_RE.finditer(text):
|
||||||
|
cand = m.group(0).strip(" .-")
|
||||||
|
digits = re.sub(r"\D", "", cand)
|
||||||
|
if 7 <= len(digits) <= 15:
|
||||||
|
found.append(cand)
|
||||||
|
# spoken digit runs: "oh seven nine one ..."
|
||||||
|
words = re.findall(r"[a-z]+", text.lower())
|
||||||
|
run, runs = [], []
|
||||||
|
for w in words:
|
||||||
|
if w in _SPOKEN_DIGITS:
|
||||||
|
run.append(_SPOKEN_DIGITS[w])
|
||||||
|
else:
|
||||||
|
if len(run) >= 7:
|
||||||
|
runs.append("".join(run))
|
||||||
|
run = []
|
||||||
|
if len(run) >= 7:
|
||||||
|
runs.append("".join(run))
|
||||||
|
found.extend(runs)
|
||||||
|
seen, uniq = set(), []
|
||||||
|
for f in found:
|
||||||
|
k = re.sub(r"\D", "", f)
|
||||||
|
if k and k not in seen:
|
||||||
|
seen.add(k)
|
||||||
|
uniq.append(f)
|
||||||
|
return uniq[:3]
|
||||||
|
|
||||||
|
|
||||||
|
def detect_intents(text):
|
||||||
|
low = text.lower()
|
||||||
|
return [label for label, pat in _INTENTS if re.search(pat, low)]
|
||||||
|
|
||||||
|
|
||||||
|
def summarise(text, max_sentences=3):
|
||||||
|
"""Frequency-scored extractive summary, original order preserved."""
|
||||||
|
sents = _split_sentences(text)
|
||||||
|
if not sents:
|
||||||
|
return ""
|
||||||
|
if len(sents) <= max_sentences:
|
||||||
|
return " ".join(sents)
|
||||||
|
|
||||||
|
freq = {}
|
||||||
|
for w in re.findall(r"[a-z']+", text.lower()):
|
||||||
|
if w in _STOP or len(w) < 3:
|
||||||
|
continue
|
||||||
|
freq[w] = freq.get(w, 0) + 1
|
||||||
|
if not freq:
|
||||||
|
return " ".join(sents[:max_sentences])
|
||||||
|
top = max(freq.values())
|
||||||
|
|
||||||
|
scored = []
|
||||||
|
for i, s in enumerate(sents):
|
||||||
|
words = [w for w in re.findall(r"[a-z']+", s.lower()) if w not in _STOP and len(w) >= 3]
|
||||||
|
if not words:
|
||||||
|
score = 0.0
|
||||||
|
else:
|
||||||
|
score = sum(freq.get(w, 0) / top for w in words) / (len(words) ** 0.5)
|
||||||
|
if i == 0:
|
||||||
|
score *= 1.35 # openers carry the reason for the call
|
||||||
|
if re.search(r"\d", s):
|
||||||
|
score *= 1.15 # numbers/dates matter
|
||||||
|
scored.append((score, i, s))
|
||||||
|
|
||||||
|
keep = sorted(sorted(scored, reverse=True)[:max_sentences], key=lambda t: t[1])
|
||||||
|
return " ".join(s for _, _, s in keep)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------- MIME output
|
||||||
|
def header_val(msg, name, default=""):
|
||||||
|
v = msg.get(name)
|
||||||
|
return str(v) if v else default
|
||||||
|
|
||||||
|
|
||||||
|
def parse_vm_fields(body_text, subject):
|
||||||
|
"""Best-effort scrape of the Asterisk notification body for context."""
|
||||||
|
f = {}
|
||||||
|
m = re.search(r"in mailbox (\S+?)[ ,]", body_text)
|
||||||
|
if m:
|
||||||
|
f["mailbox"] = m.group(1)
|
||||||
|
m = re.search(r"from (.+?), on (.+?),? so you might", body_text, re.S)
|
||||||
|
if m:
|
||||||
|
f["from"] = m.group(1).strip()
|
||||||
|
f["date"] = m.group(2).strip().rstrip(",").replace("\n", " ")
|
||||||
|
else:
|
||||||
|
m = re.search(r"from (.+?), on (.+?),", body_text)
|
||||||
|
if m:
|
||||||
|
f["from"] = m.group(1).strip()
|
||||||
|
f["date"] = m.group(2).strip()
|
||||||
|
m = re.search(r"a (\d+:\d+) long message", body_text)
|
||||||
|
if m:
|
||||||
|
f["duration"] = m.group(1)
|
||||||
|
m = re.search(r"number (\d+)", body_text)
|
||||||
|
if m:
|
||||||
|
f["msgnum"] = m.group(1)
|
||||||
|
m = re.search(r"mailbox (\S+)", subject)
|
||||||
|
if m and "mailbox" not in f:
|
||||||
|
f["mailbox"] = m.group(1)
|
||||||
|
return f
|
||||||
|
|
||||||
|
|
||||||
|
def build_plain(fields, summary, intents, numbers, transcript):
|
||||||
|
L = []
|
||||||
|
L.append("NEW VOICEMAIL")
|
||||||
|
L.append("=" * 40)
|
||||||
|
meta = [("From", fields.get("from")), ("Mailbox", fields.get("mailbox")),
|
||||||
|
("Received", fields.get("date")), ("Duration", fields.get("duration")),
|
||||||
|
("Message", fields.get("msgnum"))]
|
||||||
|
for k, v in meta:
|
||||||
|
if v:
|
||||||
|
L.append("%-10s %s" % (k + ":", v))
|
||||||
|
L.append("")
|
||||||
|
L.append("SUMMARY")
|
||||||
|
L.append("-" * 40)
|
||||||
|
L.append(summary or "(no speech detected in this recording)")
|
||||||
|
if intents:
|
||||||
|
L.append("")
|
||||||
|
L.append("Tags: " + ", ".join(intents))
|
||||||
|
if numbers:
|
||||||
|
L.append("Callback number(s) heard: " + ", ".join(numbers))
|
||||||
|
L.append("")
|
||||||
|
L.append("FULL TRANSCRIPT")
|
||||||
|
L.append("-" * 40)
|
||||||
|
L.append(transcript or "(no speech detected)")
|
||||||
|
L.append("")
|
||||||
|
L.append("The original recording is attached.")
|
||||||
|
return "\n".join(L)
|
||||||
|
|
||||||
|
|
||||||
|
def build_html(fields, summary, intents, numbers, transcript):
|
||||||
|
def row(label, value):
|
||||||
|
if not value:
|
||||||
|
return ""
|
||||||
|
return (
|
||||||
|
'<tr>'
|
||||||
|
'<td style="padding:4px 14px 4px 0;font:600 12px/1.5 -apple-system,Segoe UI,Roboto,'
|
||||||
|
'Helvetica,Arial,sans-serif;color:#8a94a6;text-transform:uppercase;letter-spacing:.5px;'
|
||||||
|
'white-space:nowrap;vertical-align:top;">%s</td>'
|
||||||
|
'<td style="padding:4px 0;font:400 14px/1.5 -apple-system,Segoe UI,Roboto,Helvetica,'
|
||||||
|
'Arial,sans-serif;color:#1c2430;">%s</td></tr>' % (escape(label), escape(value))
|
||||||
|
)
|
||||||
|
|
||||||
|
chips = "".join(
|
||||||
|
'<span style="display:inline-block;margin:0 6px 6px 0;padding:4px 11px;border-radius:999px;'
|
||||||
|
'background:#eef3ff;color:%s;font:600 12px/1.4 -apple-system,Segoe UI,Roboto,Helvetica,'
|
||||||
|
'Arial,sans-serif;">%s</span>' % (ACCENT, escape(t)) for t in intents)
|
||||||
|
|
||||||
|
numbers_html = ""
|
||||||
|
if numbers:
|
||||||
|
links = " ".join(
|
||||||
|
'<a href="tel:%s" style="color:%s;text-decoration:none;font-weight:600;">%s</a>'
|
||||||
|
% (escape(re.sub(r"[^\d+]", "", n)), ACCENT, escape(n)) for n in numbers)
|
||||||
|
numbers_html = (
|
||||||
|
'<div style="margin-top:14px;padding:12px 14px;background:#f6f8fc;border-left:3px solid %s;'
|
||||||
|
'border-radius:4px;font:400 14px/1.5 -apple-system,Segoe UI,Roboto,Helvetica,Arial,'
|
||||||
|
'sans-serif;color:#1c2430;">📞 Callback number heard: %s</div>' % (ACCENT, links))
|
||||||
|
|
||||||
|
transcript_html = escape(transcript or "(no speech detected)").replace("\n", "<br>")
|
||||||
|
summary_html = escape(summary or "(no speech detected in this recording)")
|
||||||
|
|
||||||
|
return """<!DOCTYPE html>
|
||||||
|
<html><head><meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width,initial-scale=1"></head>
|
||||||
|
<body style="margin:0;padding:24px 12px;background:#eef1f6;">
|
||||||
|
<table role="presentation" width="100%%" cellpadding="0" cellspacing="0" style="border-collapse:collapse;">
|
||||||
|
<tr><td align="center">
|
||||||
|
<table role="presentation" width="600" cellpadding="0" cellspacing="0"
|
||||||
|
style="width:600px;max-width:100%%;border-collapse:collapse;background:#ffffff;
|
||||||
|
border-radius:12px;overflow:hidden;box-shadow:0 2px 10px rgba(20,30,50,.08);">
|
||||||
|
|
||||||
|
<tr><td style="background:linear-gradient(135deg,%(accent)s,#1b4bb8);padding:22px 28px;">
|
||||||
|
<div style="font:700 19px/1.3 -apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif;color:#fff;">
|
||||||
|
✉️ New %(brand)s</div>
|
||||||
|
<div style="font:400 13px/1.5 -apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif;
|
||||||
|
color:#d9e3ff;margin-top:3px;">Transcribed and summarised automatically</div>
|
||||||
|
</td></tr>
|
||||||
|
|
||||||
|
<tr><td style="padding:22px 28px 6px 28px;">
|
||||||
|
<table role="presentation" cellpadding="0" cellspacing="0" style="border-collapse:collapse;">
|
||||||
|
%(rows)s
|
||||||
|
</table>
|
||||||
|
</td></tr>
|
||||||
|
|
||||||
|
<tr><td style="padding:16px 28px 0 28px;">
|
||||||
|
<div style="font:700 12px/1.5 -apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif;
|
||||||
|
color:#8a94a6;text-transform:uppercase;letter-spacing:.6px;">Summary</div>
|
||||||
|
<div style="margin-top:8px;padding:16px 18px;background:#f9fbff;border:1px solid #e3e9f5;
|
||||||
|
border-radius:8px;font:400 15px/1.6 -apple-system,Segoe UI,Roboto,Helvetica,Arial,
|
||||||
|
sans-serif;color:#101828;">%(summary)s</div>
|
||||||
|
%(chipwrap)s
|
||||||
|
%(numbers)s
|
||||||
|
</td></tr>
|
||||||
|
|
||||||
|
<tr><td style="padding:22px 28px 0 28px;">
|
||||||
|
<div style="font:700 12px/1.5 -apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif;
|
||||||
|
color:#8a94a6;text-transform:uppercase;letter-spacing:.6px;">Full transcript</div>
|
||||||
|
<div style="margin-top:8px;padding:16px 18px;background:#ffffff;border:1px solid #e8ecf3;
|
||||||
|
border-radius:8px;font:400 14px/1.7 -apple-system,Segoe UI,Roboto,Helvetica,Arial,
|
||||||
|
sans-serif;color:#39424e;">%(transcript)s</div>
|
||||||
|
</td></tr>
|
||||||
|
|
||||||
|
<tr><td style="padding:20px 28px 26px 28px;">
|
||||||
|
<div style="padding:12px 14px;background:#f3f5f9;border-radius:8px;
|
||||||
|
font:400 13px/1.5 -apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif;
|
||||||
|
color:#5b6675;">🎧 The original recording is attached to this email.</div>
|
||||||
|
</td></tr>
|
||||||
|
|
||||||
|
<tr><td style="padding:14px 28px;background:#fafbfd;border-top:1px solid #eceff5;
|
||||||
|
font:400 11px/1.5 -apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif;
|
||||||
|
color:#98a1b0;">
|
||||||
|
Machine transcription may contain errors — listen to the recording if in doubt.
|
||||||
|
</td></tr>
|
||||||
|
|
||||||
|
</table>
|
||||||
|
</td></tr></table>
|
||||||
|
</body></html>""" % {
|
||||||
|
"accent": ACCENT,
|
||||||
|
"brand": escape(BRAND),
|
||||||
|
"rows": (row("From", fields.get("from")) + row("Mailbox", fields.get("mailbox")) +
|
||||||
|
row("Received", fields.get("date")) + row("Duration", fields.get("duration")) +
|
||||||
|
row("Message", fields.get("msgnum"))),
|
||||||
|
"summary": summary_html,
|
||||||
|
"chipwrap": ('<div style="margin-top:12px;">%s</div>' % chips) if chips else "",
|
||||||
|
"numbers": numbers_html,
|
||||||
|
"transcript": transcript_html,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------- main
|
||||||
|
def relay(raw_bytes, envelope_from, rcpts):
|
||||||
|
dry = os.environ.get("VM_DRYRUN")
|
||||||
|
if dry:
|
||||||
|
with open(dry, "wb") as fh:
|
||||||
|
fh.write(raw_bytes if isinstance(raw_bytes, bytes) else raw_bytes.encode())
|
||||||
|
log("DRYRUN wrote %s (%d bytes) for %s" % (dry, len(raw_bytes), rcpts))
|
||||||
|
return
|
||||||
|
with smtplib.SMTP(SMTP_HOST, SMTP_PORT, timeout=30) as s:
|
||||||
|
s.sendmail(envelope_from, rcpts, raw_bytes)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
raw = sys.stdin.buffer.read()
|
||||||
|
orig = email.message_from_bytes(raw, policy=email.policy.default)
|
||||||
|
|
||||||
|
rcpts = []
|
||||||
|
for h in ("To", "Cc", "Bcc"):
|
||||||
|
for addr in email.utils.getaddresses(orig.get_all(h, [])):
|
||||||
|
if addr[1]:
|
||||||
|
rcpts.append(addr[1])
|
||||||
|
env_from = (email.utils.parseaddr(header_val(orig, "From"))[1]
|
||||||
|
or "voicemail@localhost")
|
||||||
|
if not rcpts:
|
||||||
|
log("no recipients found; relaying original")
|
||||||
|
relay(raw, env_from, rcpts or [env_from])
|
||||||
|
return
|
||||||
|
|
||||||
|
try:
|
||||||
|
# ---- pull out audio + original text body
|
||||||
|
audio_part, body_text = None, ""
|
||||||
|
for part in orig.walk():
|
||||||
|
ctype = part.get_content_type()
|
||||||
|
if part.get_content_maintype() == "audio" or (
|
||||||
|
part.get_filename() or "").lower().endswith(
|
||||||
|
(".wav", ".gsm", ".mp3", ".ogg", ".WAV")):
|
||||||
|
if audio_part is None:
|
||||||
|
audio_part = part
|
||||||
|
elif ctype == "text/plain" and not body_text:
|
||||||
|
try:
|
||||||
|
body_text = part.get_content()
|
||||||
|
except Exception:
|
||||||
|
body_text = part.get_payload(decode=True).decode("utf-8", "replace")
|
||||||
|
|
||||||
|
if audio_part is None:
|
||||||
|
log("no audio attachment; relaying original")
|
||||||
|
relay(raw, env_from, rcpts)
|
||||||
|
return
|
||||||
|
|
||||||
|
audio_bytes = audio_part.get_payload(decode=True)
|
||||||
|
fname = audio_part.get_filename() or "voicemail.wav"
|
||||||
|
suffix = os.path.splitext(fname)[1] or ".wav"
|
||||||
|
|
||||||
|
with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tf:
|
||||||
|
tf.write(audio_bytes)
|
||||||
|
tmp_audio = tf.name
|
||||||
|
|
||||||
|
try:
|
||||||
|
src = tmp_audio
|
||||||
|
if suffix.lower() not in (".wav", ".mp3", ".ogg", ".flac", ".m4a"):
|
||||||
|
conv = tmp_audio + ".wav"
|
||||||
|
subprocess.run(["/usr/bin/sox", tmp_audio, "-r", "16000", "-c", "1", conv],
|
||||||
|
check=True, capture_output=True)
|
||||||
|
src = conv
|
||||||
|
transcript = transcribe(src)
|
||||||
|
finally:
|
||||||
|
for p in (tmp_audio, tmp_audio + ".wav"):
|
||||||
|
try:
|
||||||
|
os.unlink(p)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
fields = parse_vm_fields(body_text, header_val(orig, "Subject"))
|
||||||
|
|
||||||
|
# ---- caller-ID -> contact name (best-effort)
|
||||||
|
try:
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
import vm_contacts
|
||||||
|
who = vm_contacts.resolve(fields.get("from", ""), log=log)
|
||||||
|
if who:
|
||||||
|
fields["contact"] = who
|
||||||
|
num = vm_contacts.extract_number(fields.get("from", ""))
|
||||||
|
fields["from"] = "%s (%s)" % (who, num) if num else who
|
||||||
|
except Exception:
|
||||||
|
log("contact lookup failed (non-fatal):\n" + traceback.format_exc())
|
||||||
|
|
||||||
|
summary = summarise(transcript)
|
||||||
|
intents = detect_intents(transcript)
|
||||||
|
numbers = find_numbers(transcript)
|
||||||
|
|
||||||
|
# ---- assemble multipart/mixed > multipart/alternative + attachment
|
||||||
|
out = EmailMessage()
|
||||||
|
for h in ("From", "To", "Cc", "Reply-To", "Date", "Message-ID",
|
||||||
|
"X-Asterisk-CallerID", "X-Asterisk-VM-Mailbox"):
|
||||||
|
if orig.get(h):
|
||||||
|
out[h] = orig[h]
|
||||||
|
if not out.get("From"):
|
||||||
|
out["From"] = env_from
|
||||||
|
|
||||||
|
subj_bits = []
|
||||||
|
if fields.get("from"):
|
||||||
|
subj_bits.append(fields["from"])
|
||||||
|
if fields.get("mailbox"):
|
||||||
|
subj_bits.append("mbox %s" % fields["mailbox"])
|
||||||
|
gist = (summary or transcript or "no speech detected").strip()
|
||||||
|
if len(gist) > 90:
|
||||||
|
gist = gist[:87].rsplit(" ", 1)[0] + "..."
|
||||||
|
out["Subject"] = "Voicemail%s: %s" % (
|
||||||
|
(" from " + subj_bits[0]) if subj_bits else "", gist)
|
||||||
|
out["X-Voicemail-Transcribed"] = "faster-whisper/%s" % MODEL_SIZE
|
||||||
|
if intents:
|
||||||
|
out["X-Voicemail-Tags"] = ", ".join(intents)
|
||||||
|
|
||||||
|
out.set_content(build_plain(fields, summary, intents, numbers, transcript))
|
||||||
|
out.add_alternative(build_html(fields, summary, intents, numbers, transcript),
|
||||||
|
subtype="html")
|
||||||
|
|
||||||
|
maintype, _, subtype = (audio_part.get_content_type() or "audio/x-wav").partition("/")
|
||||||
|
out.add_attachment(audio_bytes, maintype=maintype or "audio",
|
||||||
|
subtype=subtype or "x-wav", filename=fname)
|
||||||
|
|
||||||
|
relay(out.as_bytes(), env_from, rcpts)
|
||||||
|
log("sent enriched notification to %s (%d char transcript, tags=%s)"
|
||||||
|
% (",".join(rcpts), len(transcript), intents))
|
||||||
|
|
||||||
|
# ---- persist to the SQLite store for the web app (best-effort)
|
||||||
|
try:
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
import vm_store
|
||||||
|
con = vm_store.connect()
|
||||||
|
origtime = None
|
||||||
|
d = orig.get("Date")
|
||||||
|
if d:
|
||||||
|
try:
|
||||||
|
origtime = int(email.utils.mktime_tz(email.utils.parsedate_tz(str(d))))
|
||||||
|
except Exception:
|
||||||
|
origtime = None
|
||||||
|
dur = None
|
||||||
|
if fields.get("duration") and ":" in fields["duration"]:
|
||||||
|
mm, ss = fields["duration"].split(":")[:2]
|
||||||
|
try:
|
||||||
|
dur = int(mm) * 60 + int(ss)
|
||||||
|
except ValueError:
|
||||||
|
dur = None
|
||||||
|
vm_store.add_message(
|
||||||
|
con, fields.get("mailbox") or "unknown",
|
||||||
|
callerid=fields.get("from"), contact_name=fields.get("contact"),
|
||||||
|
origtime=origtime, duration=dur, transcript=transcript,
|
||||||
|
summary=summary, intents=intents, numbers=numbers,
|
||||||
|
audio_bytes=audio_bytes,
|
||||||
|
audio_ext=(os.path.splitext(fname)[1].lstrip(".") or "wav"))
|
||||||
|
con.close()
|
||||||
|
log("stored message for mailbox %s in the web-app database"
|
||||||
|
% fields.get("mailbox"))
|
||||||
|
except Exception:
|
||||||
|
log("db store failed (non-fatal):\n" + traceback.format_exc())
|
||||||
|
|
||||||
|
# ---- Telegram: strictly best-effort, email is already delivered
|
||||||
|
try:
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
import vm_telegram
|
||||||
|
vm_telegram.notify(
|
||||||
|
fields.get("mailbox"), fields, summary, intents, numbers,
|
||||||
|
transcript, audio_bytes=audio_bytes, audio_name=fname, log=log)
|
||||||
|
except Exception:
|
||||||
|
log("telegram notify failed (email was sent OK):\n" + traceback.format_exc())
|
||||||
|
|
||||||
|
except Exception:
|
||||||
|
log("FAILED, relaying original:\n" + traceback.format_exc())
|
||||||
|
try:
|
||||||
|
relay(raw, env_from, rcpts)
|
||||||
|
except Exception:
|
||||||
|
log("relay of original ALSO failed:\n" + traceback.format_exc())
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
158
src/vm_store.py
Normal file
158
src/vm_store.py
Normal file
@ -0,0 +1,158 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
SQLite store for voicemail transcripts, shared by the mailcmd hook and the
|
||||||
|
web app.
|
||||||
|
|
||||||
|
Design notes:
|
||||||
|
* WAL mode, because the mailcmd process writes while the web app reads.
|
||||||
|
* The audio is copied into a content-addressed store rather than referencing
|
||||||
|
the Asterisk spool, since Asterisk renumbers msgNNNN files whenever a
|
||||||
|
message is deleted - a stored path would silently point at the wrong
|
||||||
|
recording. The spool path is kept only as a hint for delete-on-disk.
|
||||||
|
* Every write is idempotent on (mailbox, origtime, callerid) so a re-run of
|
||||||
|
the importer cannot duplicate rows.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sqlite3
|
||||||
|
import time
|
||||||
|
|
||||||
|
DB_PATH = os.environ.get("VM_DB", "/var/lib/vm-transcribe/voicemail.db")
|
||||||
|
AUDIO_DIR = os.environ.get("VM_AUDIO_DIR", "/var/lib/vm-transcribe/audio")
|
||||||
|
|
||||||
|
SCHEMA = """
|
||||||
|
CREATE TABLE IF NOT EXISTS messages (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
mailbox TEXT NOT NULL,
|
||||||
|
context TEXT DEFAULT 'default',
|
||||||
|
folder TEXT DEFAULT 'INBOX',
|
||||||
|
callerid TEXT,
|
||||||
|
contact_name TEXT,
|
||||||
|
origtime INTEGER,
|
||||||
|
duration INTEGER,
|
||||||
|
transcript TEXT,
|
||||||
|
summary TEXT,
|
||||||
|
intents TEXT,
|
||||||
|
numbers TEXT,
|
||||||
|
audio_sha TEXT,
|
||||||
|
audio_ext TEXT DEFAULT 'wav',
|
||||||
|
spool_path TEXT,
|
||||||
|
is_read INTEGER DEFAULT 0,
|
||||||
|
created_at INTEGER,
|
||||||
|
UNIQUE (mailbox, origtime, callerid)
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_msg_mailbox ON messages (mailbox, origtime DESC);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS settings (
|
||||||
|
mailbox TEXT NOT NULL,
|
||||||
|
key TEXT NOT NULL,
|
||||||
|
value TEXT,
|
||||||
|
PRIMARY KEY (mailbox, key)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS sessions (
|
||||||
|
token TEXT PRIMARY KEY,
|
||||||
|
mailbox TEXT NOT NULL,
|
||||||
|
created_at INTEGER,
|
||||||
|
expires_at INTEGER
|
||||||
|
);
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Settings a mailbox user is allowed to change, with defaults.
|
||||||
|
USER_SETTINGS = {
|
||||||
|
"email_enabled": ("yes", "Send an email notification"),
|
||||||
|
"email_address": ("", "Override the notification email address"),
|
||||||
|
"attach_audio": ("yes", "Attach the recording to the email"),
|
||||||
|
"telegram_enabled": ("no", "Send a Telegram DM"),
|
||||||
|
"telegram_chat_id": ("", "Telegram chat ID (message the bot first)"),
|
||||||
|
"telegram_audio": ("yes", "Include the recording as a Telegram voice note"),
|
||||||
|
"telegram_transcript":("yes", "Send the full transcript as a follow-up"),
|
||||||
|
"transcribe": ("yes", "Transcribe recordings to text"),
|
||||||
|
"summarise": ("yes", "Include an automatic summary"),
|
||||||
|
"contact_lookup": ("yes", "Resolve caller ID against contacts"),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def connect(path=None):
|
||||||
|
p = os.path.abspath(path or DB_PATH)
|
||||||
|
os.makedirs(os.path.dirname(p), exist_ok=True)
|
||||||
|
con = sqlite3.connect(p, timeout=20)
|
||||||
|
con.row_factory = sqlite3.Row
|
||||||
|
con.execute("PRAGMA journal_mode=WAL")
|
||||||
|
con.execute("PRAGMA busy_timeout=10000")
|
||||||
|
con.executescript(SCHEMA)
|
||||||
|
return con
|
||||||
|
|
||||||
|
|
||||||
|
def store_audio(audio_bytes, ext="wav", audio_dir=None):
|
||||||
|
"""Content-addressed write. Returns the sha256 hex digest."""
|
||||||
|
d = audio_dir or AUDIO_DIR
|
||||||
|
sha = hashlib.sha256(audio_bytes).hexdigest()
|
||||||
|
sub = os.path.join(d, sha[:2])
|
||||||
|
os.makedirs(sub, exist_ok=True)
|
||||||
|
dest = os.path.join(sub, "%s.%s" % (sha, ext))
|
||||||
|
if not os.path.exists(dest):
|
||||||
|
tmp = dest + ".tmp"
|
||||||
|
with open(tmp, "wb") as fh:
|
||||||
|
fh.write(audio_bytes)
|
||||||
|
os.replace(tmp, dest)
|
||||||
|
return sha
|
||||||
|
|
||||||
|
|
||||||
|
def audio_path(sha, ext="wav", audio_dir=None):
|
||||||
|
d = audio_dir or AUDIO_DIR
|
||||||
|
return os.path.join(d, sha[:2], "%s.%s" % (sha, ext))
|
||||||
|
|
||||||
|
|
||||||
|
def add_message(con, mailbox, callerid=None, contact_name=None, origtime=None,
|
||||||
|
duration=None, transcript="", summary="", intents=None,
|
||||||
|
numbers=None, audio_bytes=None, audio_ext="wav",
|
||||||
|
spool_path=None, context="default", folder="INBOX"):
|
||||||
|
sha = store_audio(audio_bytes, audio_ext) if audio_bytes else None
|
||||||
|
now = int(time.time())
|
||||||
|
cur = con.execute(
|
||||||
|
"""INSERT OR IGNORE INTO messages
|
||||||
|
(mailbox, context, folder, callerid, contact_name, origtime, duration,
|
||||||
|
transcript, summary, intents, numbers, audio_sha, audio_ext,
|
||||||
|
spool_path, created_at)
|
||||||
|
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)""",
|
||||||
|
(str(mailbox), context, folder, callerid, contact_name,
|
||||||
|
int(origtime or now), duration, transcript, summary,
|
||||||
|
json.dumps(intents or []), json.dumps(numbers or []),
|
||||||
|
sha, audio_ext, spool_path, now))
|
||||||
|
con.commit()
|
||||||
|
return cur.lastrowid
|
||||||
|
|
||||||
|
|
||||||
|
def get_setting(con, mailbox, key, default=None):
|
||||||
|
row = con.execute("SELECT value FROM settings WHERE mailbox=? AND key=?",
|
||||||
|
(str(mailbox), key)).fetchone()
|
||||||
|
if row is not None:
|
||||||
|
return row["value"]
|
||||||
|
if default is not None:
|
||||||
|
return default
|
||||||
|
return USER_SETTINGS.get(key, ("", ""))[0]
|
||||||
|
|
||||||
|
|
||||||
|
def get_settings(con, mailbox):
|
||||||
|
out = {k: v[0] for k, v in USER_SETTINGS.items()}
|
||||||
|
for r in con.execute("SELECT key, value FROM settings WHERE mailbox=?",
|
||||||
|
(str(mailbox),)):
|
||||||
|
if r["key"] in USER_SETTINGS:
|
||||||
|
out[r["key"]] = r["value"]
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def set_setting(con, mailbox, key, value):
|
||||||
|
if key not in USER_SETTINGS:
|
||||||
|
raise KeyError("unknown setting %r" % key)
|
||||||
|
con.execute("""INSERT INTO settings (mailbox, key, value) VALUES (?,?,?)
|
||||||
|
ON CONFLICT(mailbox, key) DO UPDATE SET value=excluded.value""",
|
||||||
|
(str(mailbox), key, value))
|
||||||
|
con.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def truthy(v):
|
||||||
|
return str(v).strip().lower() in ("1", "yes", "true", "on")
|
||||||
247
src/vm_telegram.py
Normal file
247
src/vm_telegram.py
Normal file
@ -0,0 +1,247 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Telegram delivery for Asterisk voicemail notifications.
|
||||||
|
|
||||||
|
Routing is driven by /opt/vm-transcribe/telegram.conf, re-read on every call so
|
||||||
|
edits take effect without restarting anything.
|
||||||
|
|
||||||
|
Delivery is strictly best-effort: the email has already been sent by the time
|
||||||
|
this runs, so every failure is logged and swallowed rather than raised.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import configparser
|
||||||
|
import json
|
||||||
|
import mimetypes
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import tempfile
|
||||||
|
import urllib.parse
|
||||||
|
import urllib.request
|
||||||
|
from html import escape
|
||||||
|
|
||||||
|
CONF_PATH = os.environ.get("VM_TG_CONF", "/opt/vm-transcribe/telegram.conf")
|
||||||
|
API = "https://api.telegram.org/bot%s/%s"
|
||||||
|
|
||||||
|
# Telegram hard limits
|
||||||
|
CAPTION_LIMIT = 1024
|
||||||
|
MESSAGE_LIMIT = 4096
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------- config load
|
||||||
|
class Route:
|
||||||
|
__slots__ = ("chat_ids", "send_audio", "send_transcript", "timeout", "token")
|
||||||
|
|
||||||
|
def __init__(self, token, chat_ids, send_audio, send_transcript, timeout):
|
||||||
|
self.token = token
|
||||||
|
self.chat_ids = chat_ids
|
||||||
|
self.send_audio = send_audio
|
||||||
|
self.send_transcript = send_transcript
|
||||||
|
self.timeout = timeout
|
||||||
|
|
||||||
|
|
||||||
|
def _split_ids(raw):
|
||||||
|
return [c.strip() for c in (raw or "").replace(";", ",").split(",") if c.strip()]
|
||||||
|
|
||||||
|
|
||||||
|
def load_route(mailbox):
|
||||||
|
"""Return a Route for this mailbox, or None if Telegram is off/unmapped."""
|
||||||
|
if not os.path.exists(CONF_PATH):
|
||||||
|
return None
|
||||||
|
|
||||||
|
cp = configparser.ConfigParser(inline_comment_prefixes=("#", ";"))
|
||||||
|
cp.read(CONF_PATH)
|
||||||
|
if not cp.has_section("telegram"):
|
||||||
|
return None
|
||||||
|
|
||||||
|
g = cp["telegram"]
|
||||||
|
if not g.getboolean("enabled", fallback=False):
|
||||||
|
return None
|
||||||
|
|
||||||
|
token = (g.get("token", "") or "").strip()
|
||||||
|
if not token:
|
||||||
|
return None
|
||||||
|
|
||||||
|
sect = "mailbox:%s" % mailbox if mailbox else None
|
||||||
|
m = cp[sect] if (sect and cp.has_section(sect)) else None
|
||||||
|
|
||||||
|
def val(key, fallback):
|
||||||
|
if m is not None and key in m:
|
||||||
|
return m[key]
|
||||||
|
return g.get(key, fallback)
|
||||||
|
|
||||||
|
def boolval(key, fallback):
|
||||||
|
if m is not None and key in m:
|
||||||
|
return m.getboolean(key, fallback=fallback)
|
||||||
|
return g.getboolean(key, fallback=fallback)
|
||||||
|
|
||||||
|
chat_ids = _split_ids(val("chat_id", "")) if m is not None else []
|
||||||
|
if not chat_ids:
|
||||||
|
chat_ids = _split_ids(g.get("default_chat_id", ""))
|
||||||
|
if not chat_ids:
|
||||||
|
return None
|
||||||
|
|
||||||
|
return Route(
|
||||||
|
token=token,
|
||||||
|
chat_ids=chat_ids,
|
||||||
|
send_audio=boolval("send_audio", True),
|
||||||
|
send_transcript=boolval("send_transcript", True),
|
||||||
|
timeout=int(val("timeout", "20") or 20),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ HTTP bits
|
||||||
|
def _multipart(fields, files):
|
||||||
|
"""Build a multipart/form-data body. files = [(name, filename, bytes, ctype)]"""
|
||||||
|
boundary = "----vmtg%s" % os.urandom(12).hex()
|
||||||
|
out = bytearray()
|
||||||
|
for k, v in fields.items():
|
||||||
|
out += b"--%s\r\n" % boundary.encode()
|
||||||
|
out += b'Content-Disposition: form-data; name="%s"\r\n\r\n' % k.encode()
|
||||||
|
out += str(v).encode() + b"\r\n"
|
||||||
|
for name, fname, data, ctype in files:
|
||||||
|
out += b"--%s\r\n" % boundary.encode()
|
||||||
|
out += (b'Content-Disposition: form-data; name="%s"; filename="%s"\r\n'
|
||||||
|
% (name.encode(), fname.encode()))
|
||||||
|
out += b"Content-Type: %s\r\n\r\n" % ctype.encode()
|
||||||
|
out += data + b"\r\n"
|
||||||
|
out += b"--%s--\r\n" % boundary.encode()
|
||||||
|
return bytes(out), "multipart/form-data; boundary=%s" % boundary
|
||||||
|
|
||||||
|
|
||||||
|
def _call(route, method, fields, files=None, log=print):
|
||||||
|
url = API % (route.token, method)
|
||||||
|
try:
|
||||||
|
if files:
|
||||||
|
body, ctype = _multipart(fields, files)
|
||||||
|
else:
|
||||||
|
body = urllib.parse.urlencode(fields).encode()
|
||||||
|
ctype = "application/x-www-form-urlencoded"
|
||||||
|
req = urllib.request.Request(url, data=body, headers={"Content-Type": ctype})
|
||||||
|
with urllib.request.urlopen(req, timeout=route.timeout) as r:
|
||||||
|
resp = json.loads(r.read().decode("utf-8", "replace"))
|
||||||
|
if not resp.get("ok"):
|
||||||
|
log("telegram %s failed: %s" % (method, resp.get("description")))
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
except Exception as e:
|
||||||
|
log("telegram %s error: %s" % (method, e))
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------- formatting
|
||||||
|
def _clip(s, limit):
|
||||||
|
if len(s) <= limit:
|
||||||
|
return s
|
||||||
|
return s[: limit - 20].rsplit(" ", 1)[0] + "\n\n[...truncated]"
|
||||||
|
|
||||||
|
|
||||||
|
def build_caption(fields, summary, intents, numbers):
|
||||||
|
L = ["\U0001F4E7 <b>New voicemail</b>"]
|
||||||
|
if fields.get("from"):
|
||||||
|
L.append("\U0001F464 <b>From:</b> %s" % escape(fields["from"]))
|
||||||
|
meta = []
|
||||||
|
if fields.get("mailbox"):
|
||||||
|
meta.append("mailbox %s" % escape(fields["mailbox"]))
|
||||||
|
if fields.get("duration"):
|
||||||
|
meta.append(escape(fields["duration"]))
|
||||||
|
if fields.get("date"):
|
||||||
|
meta.append(escape(fields["date"]))
|
||||||
|
if meta:
|
||||||
|
L.append("\U0001F553 %s" % " \u00b7 ".join(meta))
|
||||||
|
L.append("")
|
||||||
|
L.append("<b>Summary</b>")
|
||||||
|
L.append(escape(summary or "(no speech detected in this recording)"))
|
||||||
|
if intents:
|
||||||
|
L.append("")
|
||||||
|
L.append("\U0001F3F7 " + " ".join("#" + re.sub(r"[^A-Za-z]+", "", t) for t in intents))
|
||||||
|
if numbers:
|
||||||
|
pretty = ", ".join(
|
||||||
|
'<a href="tel:%s">%s</a>' % (escape(re.sub(r"[^\d+]", "", n)), escape(n))
|
||||||
|
for n in numbers)
|
||||||
|
L.append("\U0001F4DE Callback: %s" % pretty)
|
||||||
|
return _clip("\n".join(L), CAPTION_LIMIT)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- audio prep
|
||||||
|
def to_voice_ogg(audio_bytes, suffix, log=print):
|
||||||
|
"""Transcode to ogg/opus for a native Telegram voice note. None on failure."""
|
||||||
|
ff = shutil.which("ffmpeg") or "/usr/bin/ffmpeg"
|
||||||
|
if not os.path.exists(ff):
|
||||||
|
log("ffmpeg not found; sending original audio as a document")
|
||||||
|
return None
|
||||||
|
src = dst = None
|
||||||
|
try:
|
||||||
|
with tempfile.NamedTemporaryFile(suffix=suffix or ".wav", delete=False) as tf:
|
||||||
|
tf.write(audio_bytes)
|
||||||
|
src = tf.name
|
||||||
|
dst = src + ".ogg"
|
||||||
|
subprocess.run(
|
||||||
|
[ff, "-v", "error", "-y", "-i", src,
|
||||||
|
"-c:a", "libopus", "-b:a", "24k", "-ar", "48000", "-ac", "1",
|
||||||
|
"-application", "voip", dst],
|
||||||
|
check=True, capture_output=True, timeout=120)
|
||||||
|
with open(dst, "rb") as fh:
|
||||||
|
return fh.read()
|
||||||
|
except Exception as e:
|
||||||
|
log("opus transcode failed (%s); falling back to raw audio" % e)
|
||||||
|
return None
|
||||||
|
finally:
|
||||||
|
for p in (src, dst):
|
||||||
|
if p:
|
||||||
|
try:
|
||||||
|
os.unlink(p)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------- entrypoint
|
||||||
|
def notify(mailbox, fields, summary, intents, numbers, transcript,
|
||||||
|
audio_bytes=None, audio_name="voicemail.wav", log=print):
|
||||||
|
"""Best-effort Telegram delivery. Returns number of chats notified."""
|
||||||
|
try:
|
||||||
|
route = load_route(mailbox)
|
||||||
|
except Exception as e:
|
||||||
|
log("telegram config error: %s" % e)
|
||||||
|
return 0
|
||||||
|
if route is None:
|
||||||
|
return 0
|
||||||
|
|
||||||
|
caption = build_caption(fields, summary, intents, numbers)
|
||||||
|
suffix = os.path.splitext(audio_name)[1] or ".wav"
|
||||||
|
|
||||||
|
voice = None
|
||||||
|
if route.send_audio and audio_bytes:
|
||||||
|
voice = to_voice_ogg(audio_bytes, suffix, log=log)
|
||||||
|
|
||||||
|
sent = 0
|
||||||
|
for chat_id in route.chat_ids:
|
||||||
|
ok = False
|
||||||
|
if route.send_audio and audio_bytes:
|
||||||
|
if voice is not None:
|
||||||
|
ok = _call(route, "sendVoice",
|
||||||
|
{"chat_id": chat_id, "caption": caption, "parse_mode": "HTML"},
|
||||||
|
files=[("voice", "voicemail.ogg", voice, "audio/ogg")], log=log)
|
||||||
|
if not ok:
|
||||||
|
ctype = mimetypes.guess_type(audio_name)[0] or "audio/wav"
|
||||||
|
ok = _call(route, "sendDocument",
|
||||||
|
{"chat_id": chat_id, "caption": caption, "parse_mode": "HTML"},
|
||||||
|
files=[("document", audio_name, audio_bytes, ctype)], log=log)
|
||||||
|
if not ok:
|
||||||
|
ok = _call(route, "sendMessage",
|
||||||
|
{"chat_id": chat_id, "text": caption, "parse_mode": "HTML",
|
||||||
|
"disable_web_page_preview": "true"}, log=log)
|
||||||
|
|
||||||
|
if ok and route.send_transcript and transcript and transcript.strip() != (summary or "").strip():
|
||||||
|
body = "\U0001F4DD <b>Full transcript</b>\n\n" + escape(transcript)
|
||||||
|
_call(route, "sendMessage",
|
||||||
|
{"chat_id": chat_id, "text": _clip(body, MESSAGE_LIMIT),
|
||||||
|
"parse_mode": "HTML", "disable_web_page_preview": "true"}, log=log)
|
||||||
|
|
||||||
|
if ok:
|
||||||
|
sent += 1
|
||||||
|
|
||||||
|
log("telegram: notified %d/%d chat(s) for mailbox %s"
|
||||||
|
% (sent, len(route.chat_ids), mailbox))
|
||||||
|
return sent
|
||||||
89
src/vm_tg_setup.py
Normal file
89
src/vm_tg_setup.py
Normal file
@ -0,0 +1,89 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Helper: discover Telegram chat IDs, and send a test voicemail notification.
|
||||||
|
|
||||||
|
# 1. who has messaged the bot?
|
||||||
|
/opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py ids
|
||||||
|
|
||||||
|
# 2. verify a route end-to-end (uses telegram.conf, real audio optional)
|
||||||
|
/opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py test 1001
|
||||||
|
"""
|
||||||
|
import configparser
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
import vm_telegram as T
|
||||||
|
|
||||||
|
|
||||||
|
def token():
|
||||||
|
cp = configparser.ConfigParser(inline_comment_prefixes=("#", ";"))
|
||||||
|
cp.read(T.CONF_PATH)
|
||||||
|
tok = (cp.get("telegram", "token", fallback="") or "").strip()
|
||||||
|
if not tok:
|
||||||
|
sys.exit("No token set in %s" % T.CONF_PATH)
|
||||||
|
return tok
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_ids():
|
||||||
|
tok = token()
|
||||||
|
url = T.API % (tok, "getUpdates")
|
||||||
|
with urllib.request.urlopen(url, timeout=20) as r:
|
||||||
|
d = json.loads(r.read().decode())
|
||||||
|
if not d.get("ok"):
|
||||||
|
sys.exit("API error: %s" % d.get("description"))
|
||||||
|
seen = {}
|
||||||
|
for u in d.get("result", []):
|
||||||
|
msg = u.get("message") or u.get("channel_post") or {}
|
||||||
|
ch = msg.get("chat") or {}
|
||||||
|
if ch.get("id") is not None:
|
||||||
|
name = " ".join(x for x in (ch.get("title"), ch.get("first_name"),
|
||||||
|
ch.get("last_name"),
|
||||||
|
("@" + ch["username"]) if ch.get("username") else None) if x)
|
||||||
|
seen[ch["id"]] = "%s [%s]" % (name or "?", ch.get("type"))
|
||||||
|
if not seen:
|
||||||
|
print("No chats found. Send your bot a message first (or /start), then re-run.")
|
||||||
|
print("Note: getUpdates returns nothing if a webhook is set, and only ~24h of history.")
|
||||||
|
return
|
||||||
|
print("Chat IDs that have talked to this bot:")
|
||||||
|
for cid, who in seen.items():
|
||||||
|
print(" chat_id = %-16s %s" % (cid, who))
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_test(mailbox):
|
||||||
|
route = T.load_route(mailbox)
|
||||||
|
if route is None:
|
||||||
|
sys.exit("No route for mailbox %r (check enabled/token/chat_id in %s)"
|
||||||
|
% (mailbox, T.CONF_PATH))
|
||||||
|
print("Route: chats=%s audio=%s transcript=%s"
|
||||||
|
% (route.chat_ids, route.send_audio, route.send_transcript))
|
||||||
|
|
||||||
|
audio = None
|
||||||
|
for cand in ("/home/jp/asterisk-vm/test_vm.wav",):
|
||||||
|
if os.path.exists(cand):
|
||||||
|
audio = open(cand, "rb").read()
|
||||||
|
break
|
||||||
|
|
||||||
|
fields = {"from": "Test Caller <07700900123>", "mailbox": mailbox or "test",
|
||||||
|
"date": "now", "duration": "0:23", "msgnum": "1"}
|
||||||
|
n = T.notify(
|
||||||
|
mailbox, fields,
|
||||||
|
"This is a test of the voicemail Telegram delivery. If you can read this "
|
||||||
|
"and play the attached voice note, the route works.",
|
||||||
|
["Call back requested"], ["07700900123"],
|
||||||
|
"This is a test of the voicemail Telegram delivery. If you can read this "
|
||||||
|
"and play the attached voice note, the route works.",
|
||||||
|
audio_bytes=audio, audio_name="test.wav")
|
||||||
|
print("notified %d chat(s)" % n)
|
||||||
|
sys.exit(0 if n else 1)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
if len(sys.argv) < 2 or sys.argv[1] not in ("ids", "test"):
|
||||||
|
sys.exit(__doc__)
|
||||||
|
if sys.argv[1] == "ids":
|
||||||
|
cmd_ids()
|
||||||
|
else:
|
||||||
|
cmd_test(sys.argv[2] if len(sys.argv) > 2 else None)
|
||||||
427
src/vm_web.py
Normal file
427
src/vm_web.py
Normal file
@ -0,0 +1,427 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Voicemail portal - FastAPI app.
|
||||||
|
|
||||||
|
Mailbox users log in with their existing voicemail mailbox number + PIN from
|
||||||
|
/etc/asterisk/voicemail.conf, then can:
|
||||||
|
* list voicemails with transcript, summary and tags
|
||||||
|
* play or download the recording
|
||||||
|
* mark read / delete (DB row, stored audio, and the spool file if present)
|
||||||
|
* edit their own notification settings
|
||||||
|
|
||||||
|
Runs as the 'asterisk' user behind Apache. Sessions are signed cookies backed
|
||||||
|
by a DB table so logout/expiry is enforced server-side.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import secrets
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
|
||||||
|
from fastapi import Cookie, FastAPI, Form, HTTPException, Request, Response
|
||||||
|
from fastapi.responses import FileResponse, HTMLResponse, RedirectResponse
|
||||||
|
from html import escape
|
||||||
|
|
||||||
|
import vm_auth
|
||||||
|
import vm_store
|
||||||
|
|
||||||
|
SESSION_HOURS = int(os.environ.get("VM_SESSION_HOURS", "12"))
|
||||||
|
COOKIE = "vm_session"
|
||||||
|
BASE = os.environ.get("VM_BASE_PATH", "") # e.g. "/voicemail" if sub-pathed
|
||||||
|
# Cookies are Secure by default (the app is served over HTTPS). Set
|
||||||
|
# VM_INSECURE_COOKIE=1 only for local plain-HTTP testing.
|
||||||
|
SECURE_COOKIE = os.environ.get("VM_INSECURE_COOKIE", "") not in ("1", "yes", "true")
|
||||||
|
|
||||||
|
app = FastAPI(title="Voicemail Portal", docs_url=None, redoc_url=None,
|
||||||
|
openapi_url=None)
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------- sessions
|
||||||
|
def new_session(mailbox):
|
||||||
|
con = vm_store.connect()
|
||||||
|
tok = secrets.token_urlsafe(32)
|
||||||
|
now = int(time.time())
|
||||||
|
con.execute("INSERT INTO sessions (token, mailbox, created_at, expires_at)"
|
||||||
|
" VALUES (?,?,?,?)",
|
||||||
|
(tok, str(mailbox), now, now + SESSION_HOURS * 3600))
|
||||||
|
con.execute("DELETE FROM sessions WHERE expires_at < ?", (now,))
|
||||||
|
con.commit()
|
||||||
|
con.close()
|
||||||
|
return tok
|
||||||
|
|
||||||
|
|
||||||
|
def session_mailbox(token):
|
||||||
|
if not token:
|
||||||
|
return None
|
||||||
|
con = vm_store.connect()
|
||||||
|
r = con.execute("SELECT mailbox, expires_at FROM sessions WHERE token=?",
|
||||||
|
(token,)).fetchone()
|
||||||
|
con.close()
|
||||||
|
if not r or r["expires_at"] < time.time():
|
||||||
|
return None
|
||||||
|
return r["mailbox"]
|
||||||
|
|
||||||
|
|
||||||
|
def require(token):
|
||||||
|
mb = session_mailbox(token)
|
||||||
|
if not mb:
|
||||||
|
raise HTTPException(status_code=303, detail="login",
|
||||||
|
headers={"Location": BASE + "/login"})
|
||||||
|
return mb
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------- brute-force lockout
|
||||||
|
# Voicemail PINs are short (often 4 digits) and this app is internet-facing,
|
||||||
|
# so failed logins are throttled per (mailbox, source IP). In-process state is
|
||||||
|
# fine: the service is a single uvicorn worker.
|
||||||
|
MAX_FAILS = int(os.environ.get("VM_MAX_FAILS", "5"))
|
||||||
|
LOCK_MINUTES = int(os.environ.get("VM_LOCK_MINUTES", "15"))
|
||||||
|
_fails = {} # (mailbox, ip) -> [count, first_ts]
|
||||||
|
|
||||||
|
|
||||||
|
def _lock_key(mailbox, ip):
|
||||||
|
return (str(mailbox).strip(), ip)
|
||||||
|
|
||||||
|
|
||||||
|
def _lock_check(mailbox, ip):
|
||||||
|
"""Return remaining lock time in minutes, or 0 if not locked."""
|
||||||
|
e = _fails.get(_lock_key(mailbox, ip))
|
||||||
|
if not e or e[0] < MAX_FAILS:
|
||||||
|
return 0
|
||||||
|
elapsed = time.time() - e[1]
|
||||||
|
if elapsed > LOCK_MINUTES * 60:
|
||||||
|
_fails.pop(_lock_key(mailbox, ip), None)
|
||||||
|
return 0
|
||||||
|
return max(1, int((LOCK_MINUTES * 60 - elapsed) // 60) + 1)
|
||||||
|
|
||||||
|
|
||||||
|
def _lock_fail(mailbox, ip):
|
||||||
|
k = _lock_key(mailbox, ip)
|
||||||
|
e = _fails.get(k)
|
||||||
|
now = time.time()
|
||||||
|
if not e or now - e[1] > LOCK_MINUTES * 60:
|
||||||
|
_fails[k] = [1, now]
|
||||||
|
else:
|
||||||
|
e[0] += 1
|
||||||
|
|
||||||
|
|
||||||
|
def _lock_clear(mailbox, ip):
|
||||||
|
_fails.pop(_lock_key(mailbox, ip), None)
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------- layout
|
||||||
|
CSS = """
|
||||||
|
*{box-sizing:border-box}
|
||||||
|
body{margin:0;font:15px/1.55 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,
|
||||||
|
Helvetica,Arial,sans-serif;background:#eef1f6;color:#1c2430}
|
||||||
|
a{color:#2f6fed}
|
||||||
|
header{background:linear-gradient(135deg,#2f6fed,#1b4bb8);color:#fff;
|
||||||
|
padding:16px 22px;display:flex;align-items:center;gap:14px;flex-wrap:wrap}
|
||||||
|
header h1{margin:0;font-size:18px;font-weight:700}
|
||||||
|
header .sp{flex:1}
|
||||||
|
header a{color:#dce6ff;text-decoration:none;font-size:14px;font-weight:600}
|
||||||
|
header a:hover{color:#fff;text-decoration:underline}
|
||||||
|
.wrap{max-width:880px;margin:22px auto;padding:0 14px}
|
||||||
|
.card{background:#fff;border:1px solid #e3e9f5;border-radius:12px;
|
||||||
|
padding:18px 20px;margin-bottom:14px;box-shadow:0 1px 4px rgba(20,30,50,.05)}
|
||||||
|
.meta{font-size:13px;color:#6b7686;margin-bottom:6px}
|
||||||
|
.who{font-weight:700;font-size:16px}
|
||||||
|
.sum{background:#f9fbff;border:1px solid #e3e9f5;border-radius:8px;
|
||||||
|
padding:12px 14px;margin:10px 0}
|
||||||
|
.tag{display:inline-block;background:#eef3ff;color:#2f6fed;border-radius:999px;
|
||||||
|
padding:3px 10px;font-size:12px;font-weight:600;margin:0 5px 5px 0}
|
||||||
|
audio{width:100%;margin-top:10px}
|
||||||
|
details{margin-top:8px}
|
||||||
|
summary{cursor:pointer;color:#2f6fed;font-size:14px;font-weight:600}
|
||||||
|
.tr{white-space:pre-wrap;color:#39424e;font-size:14px;margin-top:8px;
|
||||||
|
background:#fafbfd;padding:12px;border-radius:8px;border:1px solid #eceff5}
|
||||||
|
.row{display:flex;gap:8px;margin-top:12px;flex-wrap:wrap}
|
||||||
|
button,.btn{font:600 14px/1 inherit;padding:9px 14px;border-radius:8px;
|
||||||
|
border:1px solid #d4dcea;background:#fff;color:#2b3442;cursor:pointer}
|
||||||
|
button:hover{background:#f4f7fd}
|
||||||
|
.danger{border-color:#f0c8c8;color:#b93a3a}
|
||||||
|
.danger:hover{background:#fdf3f3}
|
||||||
|
.primary{background:#2f6fed;border-color:#2f6fed;color:#fff}
|
||||||
|
.primary:hover{background:#2860d8}
|
||||||
|
input[type=text],input[type=password],input[type=email]{width:100%;padding:10px 12px;
|
||||||
|
border:1px solid #d4dcea;border-radius:8px;font:15px inherit;background:#fff}
|
||||||
|
label{display:block;margin:12px 0 5px;font-weight:600;font-size:14px}
|
||||||
|
.hint{font-size:12px;color:#7b8494;font-weight:400}
|
||||||
|
.empty{text-align:center;color:#7b8494;padding:40px 10px}
|
||||||
|
.unread{border-left:4px solid #2f6fed}
|
||||||
|
.err{background:#fdecec;border:1px solid #f5c2c2;color:#a32c2c;padding:10px 12px;
|
||||||
|
border-radius:8px;margin-bottom:12px;font-size:14px}
|
||||||
|
.ok{background:#eaf7ee;border:1px solid #bfe3ca;color:#22683a;padding:10px 12px;
|
||||||
|
border-radius:8px;margin-bottom:12px;font-size:14px}
|
||||||
|
table{width:100%;border-collapse:collapse}
|
||||||
|
td{padding:6px 0;vertical-align:top}
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def page(title, body, mailbox=None, name=None):
|
||||||
|
nav = ""
|
||||||
|
if mailbox:
|
||||||
|
nav = ('<a href="%s/">Messages</a><a href="%s/settings">Settings</a>'
|
||||||
|
'<a href="%s/logout">Log out</a>' % (BASE, BASE, BASE))
|
||||||
|
return HTMLResponse("""<!DOCTYPE html><html><head><meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width,initial-scale=1">
|
||||||
|
<title>%s</title><style>%s</style></head><body>
|
||||||
|
<header><h1>✉ Voicemail</h1>%s<span class="sp"></span>%s</header>
|
||||||
|
<div class="wrap">%s</div></body></html>""" % (
|
||||||
|
escape(title), CSS,
|
||||||
|
('<span style="font-size:14px;color:#dce6ff">%s · mailbox %s</span>'
|
||||||
|
% (escape(name or ""), escape(mailbox))) if mailbox else "",
|
||||||
|
nav, body))
|
||||||
|
|
||||||
|
|
||||||
|
def fmt_time(ts):
|
||||||
|
if not ts:
|
||||||
|
return ""
|
||||||
|
return time.strftime("%a %d %b %Y, %H:%M", time.localtime(ts))
|
||||||
|
|
||||||
|
|
||||||
|
def fmt_dur(sec):
|
||||||
|
if not sec:
|
||||||
|
return ""
|
||||||
|
return "%d:%02d" % (sec // 60, sec % 60)
|
||||||
|
|
||||||
|
|
||||||
|
# -------------------------------------------------------------------- routes
|
||||||
|
@app.get("/login", response_class=HTMLResponse)
|
||||||
|
def login_form(err: str = ""):
|
||||||
|
e = '<div class="err">%s</div>' % escape(err) if err else ""
|
||||||
|
return page("Log in", """%s<div class="card">
|
||||||
|
<form method="post" action="%s/login">
|
||||||
|
<label>Mailbox number</label>
|
||||||
|
<input type="text" name="mailbox" inputmode="numeric" autocomplete="username" autofocus>
|
||||||
|
<label>PIN <span class="hint">the same PIN you use on the phone</span></label>
|
||||||
|
<input type="password" name="pin" inputmode="numeric" autocomplete="current-password">
|
||||||
|
<div class="row"><button class="primary" type="submit">Log in</button></div>
|
||||||
|
</form></div>""" % (e, BASE))
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/login")
|
||||||
|
def do_login(request: Request, mailbox: str = Form(...), pin: str = Form(...)):
|
||||||
|
ip = request.client.host if request.client else "?"
|
||||||
|
locked = _lock_check(mailbox, ip)
|
||||||
|
if locked:
|
||||||
|
return RedirectResponse(
|
||||||
|
BASE + "/login?err=Too+many+failed+attempts.+Try+again+in+%d+minutes."
|
||||||
|
% locked, status_code=303)
|
||||||
|
|
||||||
|
info = vm_auth.check_login(mailbox, pin)
|
||||||
|
if not info:
|
||||||
|
_lock_fail(mailbox, ip)
|
||||||
|
time.sleep(1) # slow down brute force
|
||||||
|
return RedirectResponse(BASE + "/login?err=Incorrect+mailbox+or+PIN",
|
||||||
|
status_code=303)
|
||||||
|
_lock_clear(mailbox, ip)
|
||||||
|
tok = new_session(mailbox.strip())
|
||||||
|
r = RedirectResponse(BASE + "/", status_code=303)
|
||||||
|
r.set_cookie(COOKIE, tok, httponly=True, samesite="lax",
|
||||||
|
secure=SECURE_COOKIE, max_age=SESSION_HOURS * 3600,
|
||||||
|
path=BASE + "/")
|
||||||
|
return r
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/logout")
|
||||||
|
def logout(vm_session: str = Cookie(default=None)):
|
||||||
|
if vm_session:
|
||||||
|
con = vm_store.connect()
|
||||||
|
con.execute("DELETE FROM sessions WHERE token=?", (vm_session,))
|
||||||
|
con.commit()
|
||||||
|
con.close()
|
||||||
|
r = RedirectResponse(BASE + "/login", status_code=303)
|
||||||
|
r.delete_cookie(COOKIE, path=BASE + "/")
|
||||||
|
return r
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/", response_class=HTMLResponse)
|
||||||
|
def index(vm_session: str = Cookie(default=None), msg: str = ""):
|
||||||
|
mb = require(vm_session)
|
||||||
|
boxes = vm_auth.parse_mailboxes()
|
||||||
|
name = boxes.get(mb, {}).get("name", "")
|
||||||
|
con = vm_store.connect()
|
||||||
|
rows = con.execute(
|
||||||
|
"SELECT * FROM messages WHERE mailbox=? ORDER BY origtime DESC, id DESC",
|
||||||
|
(mb,)).fetchall()
|
||||||
|
con.close()
|
||||||
|
|
||||||
|
banner = '<div class="ok">%s</div>' % escape(msg) if msg else ""
|
||||||
|
if not rows:
|
||||||
|
return page("Messages", banner + '<div class="card empty">'
|
||||||
|
'No voicemails yet.<br><span class="hint">New messages appear '
|
||||||
|
'here automatically once transcribed.</span></div>',
|
||||||
|
mb, name)
|
||||||
|
|
||||||
|
out = [banner]
|
||||||
|
for r in rows:
|
||||||
|
tags = "".join('<span class="tag">%s</span>' % escape(t)
|
||||||
|
for t in json.loads(r["intents"] or "[]"))
|
||||||
|
nums = json.loads(r["numbers"] or "[]")
|
||||||
|
callback = ""
|
||||||
|
if nums:
|
||||||
|
callback = ' · '.join(
|
||||||
|
'<a href="tel:%s">%s</a>' % (escape("".join(
|
||||||
|
ch for ch in n if ch.isdigit() or ch == "+")), escape(n))
|
||||||
|
for n in nums)
|
||||||
|
callback = '<div class="meta">📞 Callback: %s</div>' % callback
|
||||||
|
|
||||||
|
audio = ""
|
||||||
|
if r["audio_sha"]:
|
||||||
|
audio = ('<audio controls preload="none" src="%s/audio/%d"></audio>'
|
||||||
|
% (BASE, r["id"]))
|
||||||
|
|
||||||
|
transcript = ""
|
||||||
|
if r["transcript"]:
|
||||||
|
transcript = ('<details><summary>Full transcript</summary>'
|
||||||
|
'<div class="tr">%s</div></details>'
|
||||||
|
% escape(r["transcript"]))
|
||||||
|
|
||||||
|
out.append("""<div class="card%s">
|
||||||
|
<div class="who">%s</div>
|
||||||
|
<div class="meta">%s%s</div>
|
||||||
|
<div class="sum">%s</div>
|
||||||
|
%s%s
|
||||||
|
%s
|
||||||
|
%s
|
||||||
|
<div class="row">
|
||||||
|
<form method="post" action="%s/read/%d"><button>%s</button></form>
|
||||||
|
<a class="btn" href="%s/audio/%d?dl=1">Download</a>
|
||||||
|
<form method="post" action="%s/delete/%d"
|
||||||
|
onsubmit="return confirm('Delete this voicemail permanently?')">
|
||||||
|
<button class="danger">Delete</button></form>
|
||||||
|
</div></div>""" % (
|
||||||
|
"" if r["is_read"] else " unread",
|
||||||
|
escape(r["contact_name"] or r["callerid"] or "Unknown caller"),
|
||||||
|
escape(fmt_time(r["origtime"])),
|
||||||
|
(" · " + escape(fmt_dur(r["duration"]))) if r["duration"] else "",
|
||||||
|
escape(r["summary"] or "(no speech detected)"),
|
||||||
|
('<div style="margin-top:8px">%s</div>' % tags) if tags else "",
|
||||||
|
callback, audio, transcript,
|
||||||
|
BASE, r["id"], "Mark unread" if r["is_read"] else "Mark read",
|
||||||
|
BASE, r["id"], BASE, r["id"]))
|
||||||
|
|
||||||
|
return page("Messages", "".join(out), mb, name)
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/audio/{msg_id}")
|
||||||
|
def audio(msg_id: int, dl: int = 0, vm_session: str = Cookie(default=None)):
|
||||||
|
mb = require(vm_session)
|
||||||
|
con = vm_store.connect()
|
||||||
|
r = con.execute("SELECT * FROM messages WHERE id=? AND mailbox=?",
|
||||||
|
(msg_id, mb)).fetchone()
|
||||||
|
con.close()
|
||||||
|
if not r or not r["audio_sha"]:
|
||||||
|
raise HTTPException(404, "not found")
|
||||||
|
p = vm_store.audio_path(r["audio_sha"], r["audio_ext"] or "wav")
|
||||||
|
if not os.path.exists(p):
|
||||||
|
raise HTTPException(404, "recording missing")
|
||||||
|
fname = "voicemail-%s-%d.%s" % (mb, msg_id, r["audio_ext"] or "wav")
|
||||||
|
return FileResponse(
|
||||||
|
p, media_type="audio/wav",
|
||||||
|
filename=fname if dl else None,
|
||||||
|
headers={} if dl else {"Content-Disposition": 'inline; filename="%s"' % fname})
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/read/{msg_id}")
|
||||||
|
def toggle_read(msg_id: int, vm_session: str = Cookie(default=None)):
|
||||||
|
mb = require(vm_session)
|
||||||
|
con = vm_store.connect()
|
||||||
|
con.execute("UPDATE messages SET is_read = 1 - is_read"
|
||||||
|
" WHERE id=? AND mailbox=?", (msg_id, mb))
|
||||||
|
con.commit()
|
||||||
|
con.close()
|
||||||
|
return RedirectResponse(BASE + "/", status_code=303)
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/delete/{msg_id}")
|
||||||
|
def delete(msg_id: int, vm_session: str = Cookie(default=None)):
|
||||||
|
mb = require(vm_session)
|
||||||
|
con = vm_store.connect()
|
||||||
|
r = con.execute("SELECT * FROM messages WHERE id=? AND mailbox=?",
|
||||||
|
(msg_id, mb)).fetchone()
|
||||||
|
if not r:
|
||||||
|
con.close()
|
||||||
|
raise HTTPException(404, "not found")
|
||||||
|
|
||||||
|
# remove the stored audio only if no other row references it
|
||||||
|
if r["audio_sha"]:
|
||||||
|
others = con.execute("SELECT COUNT(*) c FROM messages"
|
||||||
|
" WHERE audio_sha=? AND id<>?",
|
||||||
|
(r["audio_sha"], msg_id)).fetchone()["c"]
|
||||||
|
if not others:
|
||||||
|
try:
|
||||||
|
os.unlink(vm_store.audio_path(r["audio_sha"], r["audio_ext"] or "wav"))
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
# and the spool copy, when we know where it was
|
||||||
|
if r["spool_path"]:
|
||||||
|
base = os.path.splitext(r["spool_path"])[0]
|
||||||
|
for ext in (".wav", ".WAV", ".gsm", ".txt", ".wav49"):
|
||||||
|
try:
|
||||||
|
os.unlink(base + ext)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
con.execute("DELETE FROM messages WHERE id=? AND mailbox=?", (msg_id, mb))
|
||||||
|
con.commit()
|
||||||
|
con.close()
|
||||||
|
return RedirectResponse(BASE + "/?msg=Voicemail+deleted", status_code=303)
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/settings", response_class=HTMLResponse)
|
||||||
|
def settings_form(vm_session: str = Cookie(default=None), msg: str = ""):
|
||||||
|
mb = require(vm_session)
|
||||||
|
boxes = vm_auth.parse_mailboxes()
|
||||||
|
info = boxes.get(mb, {})
|
||||||
|
con = vm_store.connect()
|
||||||
|
cur = vm_store.get_settings(con, mb)
|
||||||
|
con.close()
|
||||||
|
|
||||||
|
rows = []
|
||||||
|
for key, (default, label) in vm_store.USER_SETTINGS.items():
|
||||||
|
val = cur.get(key, default)
|
||||||
|
if default in ("yes", "no"):
|
||||||
|
checked = " checked" if vm_store.truthy(val) else ""
|
||||||
|
rows.append('<tr><td><label style="font-weight:400;margin:0">'
|
||||||
|
'<input type="checkbox" name="%s" value="yes"%s> %s'
|
||||||
|
'</label></td></tr>' % (key, checked, escape(label)))
|
||||||
|
else:
|
||||||
|
rows.append('<tr><td><label>%s</label>'
|
||||||
|
'<input type="text" name="%s" value="%s"></td></tr>'
|
||||||
|
% (escape(label), key, escape(val or "")))
|
||||||
|
|
||||||
|
banner = '<div class="ok">%s</div>' % escape(msg) if msg else ""
|
||||||
|
return page("Settings", """%s<div class="card">
|
||||||
|
<div class="meta">Notification settings for <b>%s</b> (mailbox %s).
|
||||||
|
Phone PIN changes must still be made on the phone or by your administrator.</div>
|
||||||
|
<form method="post" action="%s/settings"><table>%s</table>
|
||||||
|
<div class="row"><button class="primary" type="submit">Save settings</button></div>
|
||||||
|
</form></div>""" % (banner, escape(info.get("name", "")), escape(mb), BASE,
|
||||||
|
"".join(rows)), mb, info.get("name", ""))
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/settings")
|
||||||
|
async def save_settings(request: Request, vm_session: str = Cookie(default=None)):
|
||||||
|
mb = require(vm_session)
|
||||||
|
form = await request.form()
|
||||||
|
con = vm_store.connect()
|
||||||
|
for key, (default, _label) in vm_store.USER_SETTINGS.items():
|
||||||
|
if default in ("yes", "no"):
|
||||||
|
vm_store.set_setting(con, mb, key, "yes" if form.get(key) else "no")
|
||||||
|
else:
|
||||||
|
vm_store.set_setting(con, mb, key, (form.get(key) or "").strip())
|
||||||
|
con.close()
|
||||||
|
return RedirectResponse(BASE + "/settings?msg=Settings+saved", status_code=303)
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/healthz")
|
||||||
|
def healthz():
|
||||||
|
con = vm_store.connect()
|
||||||
|
n = con.execute("SELECT COUNT(*) c FROM messages").fetchone()["c"]
|
||||||
|
con.close()
|
||||||
|
return {"ok": True, "messages": n}
|
||||||
43
systemd/vm-portal.service
Normal file
43
systemd/vm-portal.service
Normal file
@ -0,0 +1,43 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Voicemail portal (transcripts, playback, per-mailbox settings)
|
||||||
|
Documentation=file:/opt/vm-transcribe/vm_web.py
|
||||||
|
After=network.target
|
||||||
|
Wants=network.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
# Runs as asterisk so it can read voicemail.conf and the spool.
|
||||||
|
User=asterisk
|
||||||
|
Group=asterisk
|
||||||
|
WorkingDirectory=/opt/vm-transcribe
|
||||||
|
|
||||||
|
Environment=VM_DB=/var/lib/vm-transcribe/voicemail.db
|
||||||
|
Environment=VM_AUDIO_DIR=/var/lib/vm-transcribe/audio
|
||||||
|
Environment=VM_ASTERISK_CONF=/etc/asterisk/voicemail.conf
|
||||||
|
Environment=VM_SESSION_HOURS=12
|
||||||
|
Environment=PYTHONUNBUFFERED=1
|
||||||
|
|
||||||
|
ExecStart=/opt/vm-transcribe/venv/bin/python -m uvicorn vm_web:app \
|
||||||
|
--host 127.0.0.1 --port 8099 \
|
||||||
|
--proxy-headers --forwarded-allow-ips 127.0.0.1 \
|
||||||
|
--log-level info
|
||||||
|
|
||||||
|
Restart=on-failure
|
||||||
|
RestartSec=3
|
||||||
|
|
||||||
|
# --- hardening ---------------------------------------------------------
|
||||||
|
NoNewPrivileges=yes
|
||||||
|
PrivateTmp=yes
|
||||||
|
ProtectSystem=full
|
||||||
|
ProtectHome=yes
|
||||||
|
ProtectKernelTunables=yes
|
||||||
|
ProtectControlGroups=yes
|
||||||
|
RestrictSUIDSGID=yes
|
||||||
|
# Only these paths need to be writable.
|
||||||
|
ReadWritePaths=/var/lib/vm-transcribe /var/log/asterisk
|
||||||
|
# Loopback only; Apache is the only client.
|
||||||
|
IPAddressAllow=localhost
|
||||||
|
IPAddressDeny=any
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
20
tests/make_test_mail.py
Normal file
20
tests/make_test_mail.py
Normal file
@ -0,0 +1,20 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Build a fake Asterisk voicemail notification (as sendmail -t would get it)."""
|
||||||
|
import sys
|
||||||
|
from email.message import EmailMessage
|
||||||
|
|
||||||
|
wav = sys.argv[1]
|
||||||
|
to = sys.argv[2]
|
||||||
|
|
||||||
|
m = EmailMessage()
|
||||||
|
m["From"] = "voicemail@txt3.net"
|
||||||
|
m["To"] = to
|
||||||
|
m["Subject"] = "New message 3 in mailbox 1001"
|
||||||
|
m.set_content(
|
||||||
|
"Dear Jamie:\n\n\tjust wanted to let you know you were just left a 0:37 long message "
|
||||||
|
"(number 3)\nin mailbox 1001 from Dave Roberts <07941223856>, on Thu, 13 Aug 2026 "
|
||||||
|
"07:12:00, so you might\nwant to check it when you get a chance. Thanks!\n\n"
|
||||||
|
"\t\t\t\t--Asterisk\n")
|
||||||
|
with open(wav, "rb") as fh:
|
||||||
|
m.add_attachment(fh.read(), maintype="audio", subtype="x-wav", filename="msg0003.wav")
|
||||||
|
sys.stdout.buffer.write(m.as_bytes())
|
||||||
113
tests/test_contacts.py
Normal file
113
tests/test_contacts.py
Normal file
@ -0,0 +1,113 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Offline tests for vm_contacts: vCard/CSV parsing, digit matching, cache."""
|
||||||
|
import os, sys, tempfile, json
|
||||||
|
sys.path.insert(0, "/home/jp/asterisk-vm")
|
||||||
|
|
||||||
|
d = tempfile.mkdtemp()
|
||||||
|
|
||||||
|
VCF = """BEGIN:VCARD
|
||||||
|
VERSION:3.0
|
||||||
|
FN:Dave Roberts
|
||||||
|
TEL;TYPE=CELL:+44 7941 223856
|
||||||
|
END:VCARD
|
||||||
|
BEGIN:VCARD
|
||||||
|
VERSION:3.0
|
||||||
|
FN:Alice Smith
|
||||||
|
TEL;TYPE=WORK:020 7946 0018
|
||||||
|
TEL;TYPE=CELL:07700 900123
|
||||||
|
END:VCARD
|
||||||
|
BEGIN:VCARD
|
||||||
|
VERSION:3.0
|
||||||
|
FN:No Number Person
|
||||||
|
END:VCARD
|
||||||
|
"""
|
||||||
|
vcf = os.path.join(d, "c.vcf"); open(vcf, "w").write(VCF)
|
||||||
|
|
||||||
|
CSV = """Name,Given Name,Family Name,Phone 1 - Type,Phone 1 - Value
|
||||||
|
Bob Jones,Bob,Jones,Mobile,+1 (555) 010-9876
|
||||||
|
Carol White,Carol,White,Mobile,07123 456789 ::: 02012345678
|
||||||
|
"""
|
||||||
|
csvp = os.path.join(d, "c.csv"); open(csvp, "w").write(CSV)
|
||||||
|
|
||||||
|
cache = os.path.join(d, "cache.json")
|
||||||
|
|
||||||
|
def write_conf(path_val, backends="file"):
|
||||||
|
c = os.path.join(d, "contacts.conf")
|
||||||
|
open(c, "w").write(f"""[contacts]
|
||||||
|
enabled = yes
|
||||||
|
backends = {backends}
|
||||||
|
cache_path = {cache}
|
||||||
|
cache_ttl = 86400
|
||||||
|
match_digits = 9
|
||||||
|
|
||||||
|
[file]
|
||||||
|
path = {path_val}
|
||||||
|
""")
|
||||||
|
os.environ["VM_CONTACTS_CONF"] = c
|
||||||
|
return c
|
||||||
|
|
||||||
|
import importlib
|
||||||
|
import vm_contacts as C
|
||||||
|
|
||||||
|
def fresh(path_val, backends="file"):
|
||||||
|
write_conf(path_val, backends)
|
||||||
|
if os.path.exists(cache): os.unlink(cache)
|
||||||
|
importlib.reload(C)
|
||||||
|
return C
|
||||||
|
|
||||||
|
print("== extract_number")
|
||||||
|
for s in ['Dave Roberts <07941223856>', '"Alice" <+447941223856>', '07700900123', 'unknown', '']:
|
||||||
|
print(" %-30r -> %r" % (s, C.extract_number(s)))
|
||||||
|
|
||||||
|
print("\n== vCard lookup, various formats of the SAME number")
|
||||||
|
c = fresh(vcf)
|
||||||
|
for s in ["<07941223856>", "<+447941223856>", "<447941223856>", "Dave <7941223856>"]:
|
||||||
|
print(" %-24s -> %r" % (s, c.resolve(s, log=lambda m: None)))
|
||||||
|
|
||||||
|
print("\n== vCard: second number on a multi-TEL contact")
|
||||||
|
c = fresh(vcf)
|
||||||
|
print(" Alice work 02079460018 ->", c.resolve("<02079460018>", log=lambda m: None))
|
||||||
|
c = fresh(vcf)
|
||||||
|
print(" Alice cell 07700900123 ->", c.resolve("<07700900123>", log=lambda m: None))
|
||||||
|
|
||||||
|
print("\n== unknown number -> None (and cached as a miss)")
|
||||||
|
c = fresh(vcf)
|
||||||
|
print(" ->", c.resolve("<07999999999>", log=lambda m: None))
|
||||||
|
print(" cache contents:", json.load(open(cache)))
|
||||||
|
|
||||||
|
print("\n== CSV backend (Google CSV export format, ::: multi-value)")
|
||||||
|
c = fresh(csvp)
|
||||||
|
print(" Bob 5550109876 ->", c.resolve("<+15550109876>", log=lambda m: None))
|
||||||
|
c = fresh(csvp)
|
||||||
|
print(" Carol 07123456789 ->", c.resolve("<07123456789>", log=lambda m: None))
|
||||||
|
c = fresh(csvp)
|
||||||
|
print(" Carol 2nd num 02012345678 ->", c.resolve("<02012345678>", log=lambda m: None))
|
||||||
|
|
||||||
|
print("\n== disabled / missing file / no number")
|
||||||
|
c = fresh(os.path.join(d, "nope.vcf"))
|
||||||
|
print(" missing file ->", c.resolve("<07941223856>", log=lambda m: None))
|
||||||
|
open(os.environ["VM_CONTACTS_CONF"], "a").write("\n")
|
||||||
|
w = os.path.join(d, "off.conf")
|
||||||
|
open(w, "w").write("[contacts]\nenabled = no\nbackends = file\n")
|
||||||
|
os.environ["VM_CONTACTS_CONF"] = w; importlib.reload(C)
|
||||||
|
print(" disabled ->", C.resolve("<07941223856>", log=lambda m: None))
|
||||||
|
c = fresh(vcf)
|
||||||
|
print(" empty callerid ->", c.resolve("", log=lambda m: None))
|
||||||
|
|
||||||
|
print("\n== carddav pointed at google must refuse app-password auth")
|
||||||
|
w2 = os.path.join(d, "cd.conf")
|
||||||
|
open(w2, "w").write(f"""[contacts]
|
||||||
|
enabled = yes
|
||||||
|
backends = carddav
|
||||||
|
cache_path = {os.path.join(d,'c2.json')}
|
||||||
|
match_digits = 9
|
||||||
|
|
||||||
|
[carddav]
|
||||||
|
url = https://www.google.com/carddav/v1/principals/me/lists/default/
|
||||||
|
username = me@gmail.com
|
||||||
|
app_password = abcdefghijklmnop
|
||||||
|
""")
|
||||||
|
os.environ["VM_CONTACTS_CONF"] = w2; importlib.reload(C)
|
||||||
|
msgs = []
|
||||||
|
print(" ->", C.resolve("<07941223856>", log=msgs.append))
|
||||||
|
for m in msgs: print(" [log]", m)
|
||||||
67
tests/test_telegram.py
Normal file
67
tests/test_telegram.py
Normal file
@ -0,0 +1,67 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Offline tests for vm_telegram: config routing + caption building + opus."""
|
||||||
|
import os, sys, tempfile
|
||||||
|
sys.path.insert(0, "/home/jp/asterisk-vm")
|
||||||
|
|
||||||
|
CONF = """
|
||||||
|
[telegram]
|
||||||
|
enabled = yes
|
||||||
|
token = 111:AAA
|
||||||
|
default_chat_id = 999
|
||||||
|
send_audio = yes
|
||||||
|
send_transcript = yes
|
||||||
|
timeout = 20
|
||||||
|
|
||||||
|
[mailbox:1001]
|
||||||
|
chat_id = 123456789
|
||||||
|
|
||||||
|
[mailbox:1002]
|
||||||
|
chat_id = 5551, 5552
|
||||||
|
send_transcript = no
|
||||||
|
|
||||||
|
[mailbox:1003]
|
||||||
|
chat_id = -1001234567890
|
||||||
|
send_audio = no
|
||||||
|
"""
|
||||||
|
tf = tempfile.NamedTemporaryFile("w", suffix=".conf", delete=False)
|
||||||
|
tf.write(CONF); tf.close()
|
||||||
|
os.environ["VM_TG_CONF"] = tf.name
|
||||||
|
import vm_telegram as T
|
||||||
|
|
||||||
|
print("== routing")
|
||||||
|
for mb in ("1001", "1002", "1003", "1099", None):
|
||||||
|
r = T.load_route(mb)
|
||||||
|
if r is None:
|
||||||
|
print(" mailbox %-5s -> no route" % mb)
|
||||||
|
else:
|
||||||
|
print(" mailbox %-5s -> chats=%s audio=%s transcript=%s"
|
||||||
|
% (mb, r.chat_ids, r.send_audio, r.send_transcript))
|
||||||
|
|
||||||
|
print("\n== disabled master switch")
|
||||||
|
open(tf.name, "w").write(CONF.replace("enabled = yes", "enabled = no"))
|
||||||
|
print(" ->", T.load_route("1001"))
|
||||||
|
print("== missing token")
|
||||||
|
open(tf.name, "w").write(CONF.replace("token = 111:AAA", "token ="))
|
||||||
|
print(" ->", T.load_route("1001"))
|
||||||
|
open(tf.name, "w").write(CONF)
|
||||||
|
|
||||||
|
print("\n== caption")
|
||||||
|
fields = {"from": "Dave Roberts <07941223856>", "mailbox": "1001",
|
||||||
|
"date": "Thu, 13 Aug 2026 07:12:00", "duration": "0:37", "msgnum": "3"}
|
||||||
|
cap = T.build_caption(fields,
|
||||||
|
"Hi, this is Dave Roberts calling from Meridian Plumbing about the invoice. "
|
||||||
|
"Could you please call me back as soon as possible on 07941-223856.",
|
||||||
|
["Call back requested", "Urgent", "Payment / invoice"], ["07941-223856"])
|
||||||
|
print(cap)
|
||||||
|
print(" [caption length %d / %d]" % (len(cap), T.CAPTION_LIMIT))
|
||||||
|
|
||||||
|
print("\n== caption clipping (5000-char summary)")
|
||||||
|
big = T.build_caption(fields, "word " * 1000, [], [])
|
||||||
|
print(" length %d (limit %d) ok=%s" % (len(big), T.CAPTION_LIMIT, len(big) <= T.CAPTION_LIMIT))
|
||||||
|
|
||||||
|
print("\n== opus transcode")
|
||||||
|
wav = open("/home/jp/asterisk-vm/test_vm.wav", "rb").read()
|
||||||
|
ogg = T.to_voice_ogg(wav, ".wav")
|
||||||
|
print(" wav %d bytes -> ogg %s bytes" % (len(wav), len(ogg) if ogg else None))
|
||||||
|
print(" magic:", ogg[:4] if ogg else None)
|
||||||
|
os.unlink(tf.name)
|
||||||
Reference in New Issue
Block a user