Initial build: Asterisk voicemail transcription + portal
- mailcmd replacement (vm_mailcmd.py): faster-whisper transcription (CPU int8), extractive summary + intent tags + spoken-digit number extraction, multipart/alternative HTML email, fail-safe relay of original message - Telegram DM delivery (vm_telegram.py) with per-mailbox routing - Caller-ID -> name (vm_contacts.py): file / google / carddav backends - SQLite store (vm_store.py) with content-addressed audio - FastAPI portal (vm_web.py): PIN login, list/play/delete, per-user settings, zero JS, loopback-only behind Apache TLS - Backfill importer (vm_import.py) for existing spool recordings - systemd unit, Apache vhost + certbot TLS, install.sh - Docs: INSTALL, CONFIGURATION, ARCHITECTURE, OPERATIONS, SECURITY, TESTING Verified end-to-end on mail.txt3.net: 157 historical messages backfilled, live voicemail -> transcribed -> stored -> visible at https://vm.txt3.net.
This commit is contained in:
34
.gitignore
vendored
Normal file
34
.gitignore
vendored
Normal file
@ -0,0 +1,34 @@
|
||||
# Local config with real credentials — never commit these.
|
||||
config/telegram.conf
|
||||
config/contacts.conf
|
||||
*.conf.local
|
||||
|
||||
# Runtime data
|
||||
*.db
|
||||
*.db-wal
|
||||
*.db-shm
|
||||
audio/
|
||||
models/
|
||||
*.log
|
||||
|
||||
# Test artefacts
|
||||
*.eml
|
||||
*.wav
|
||||
*.mp3
|
||||
*.ogg
|
||||
preview.html
|
||||
ui_preview.html
|
||||
tstore.db*
|
||||
taudio/
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
venv/
|
||||
.venv/
|
||||
|
||||
# Editor / OS
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
.DS_Store
|
||||
29
CHANGELOG.md
Normal file
29
CHANGELOG.md
Normal file
@ -0,0 +1,29 @@
|
||||
# Changelog
|
||||
|
||||
## 1.0.0 — 2026-08-13
|
||||
|
||||
Initial build on mail.txt3.net (Debian 12, Asterisk 20, Apache 2.4, Postfix).
|
||||
|
||||
- **Transcription**: per-voicemail `mailcmd` (`vm_mailcmd.py`) transcribes the
|
||||
attached recording with faster-whisper `base.en` (CPU, int8, VAD-filtered).
|
||||
- **Summarisation**: local extractive summariser with intent tags and spoken-
|
||||
digit callback-number extraction. No LLM, by choice.
|
||||
- **Email**: rebuilt `multipart/alternative` notification (plain + styled HTML)
|
||||
with the recording attached; relays the original Asterisk message unchanged on
|
||||
any error (fail-safe).
|
||||
- **Telegram** (optional): per-mailbox routed voice-note DM with summary caption
|
||||
and transcript follow-up; degrades sendVoice → sendDocument → sendMessage.
|
||||
- **Contacts** (optional): caller-ID → name via `file` (vCard/CSV export),
|
||||
`google` (People API OAuth), or `carddav` (app password; Nextcloud/Fastmail/
|
||||
iCloud). Last-9-digit matching. Note: Google app passwords do not work.
|
||||
- **Web portal**: FastAPI app at `https://vm.txt3.net` — PIN login (mailbox +
|
||||
voicemail PIN from `voicemail.conf`), list, play, download, delete, per-user
|
||||
settings. Zero JavaScript, `script-src 'none'` CSP, loopback-only backend
|
||||
behind an Apache TLS reverse proxy. Brute-force lockout per (mailbox, IP).
|
||||
- **Storage**: SQLite store with content-addressed audio (decoupled from
|
||||
Asterisk's renumbering) and DB-backed sessions.
|
||||
- **Backfill**: `vm_import.py` imports and transcribes existing spool recordings;
|
||||
idempotent on `(mailbox, origtime, callerid)`. Backfilled 157 historical
|
||||
messages on first run (3m40s, 0 failures).
|
||||
- **Packaging**: venv at `/opt/vm-transcribe`, systemd `vm-portal.service`,
|
||||
Apache vhost + certbot TLS, `install.sh`.
|
||||
21
LICENSE
Normal file
21
LICENSE
Normal file
@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 jp
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
162
README.md
Normal file
162
README.md
Normal file
@ -0,0 +1,162 @@
|
||||
# Asterisk Voicemail Transcription & Portal
|
||||
|
||||
Turns Asterisk voicemail into something you can actually read, search and
|
||||
manage:
|
||||
|
||||
- **Transcribes** each recording locally with [faster-whisper](https://github.com/SYSTRAN/faster-whisper) (CPU, no cloud, no API key)
|
||||
- **Summarises** it, tags intent (callback requested, urgent, invoice…) and
|
||||
extracts callback numbers — including spoken-out digits
|
||||
- **Emails** a graphically designed `multipart/alternative` notification
|
||||
(plain + HTML) with the recording attached
|
||||
- **DMs Telegram** optionally, as a playable voice note with the summary
|
||||
- **Resolves caller ID to a name** from Google Contacts or any CardDAV server
|
||||
- **Serves a web portal** where mailbox users log in with their existing phone
|
||||
PIN to read transcripts, play/download recordings, delete messages and
|
||||
manage their own notification settings
|
||||
|
||||
Everything runs on the PBX host. No third-party service sees your voicemail.
|
||||
|
||||
---
|
||||
|
||||
## What it looks like
|
||||
|
||||
**Email notification** — styled HTML card with metadata, summary panel, intent
|
||||
chips, a `tel:` callback link, full transcript and the audio attached. A plain
|
||||
text alternative is always included.
|
||||
|
||||
**Telegram** — voice note (ogg/opus) with the summary as its caption, intent
|
||||
hashtags, and the transcript as a follow-up message.
|
||||
|
||||
**Portal** — one card per voicemail: caller, timestamp, duration, summary,
|
||||
tags, callback link, inline player, collapsible transcript, and Mark read /
|
||||
Download / Delete.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
incoming call
|
||||
│
|
||||
▼
|
||||
┌─────────────┐ voicemail.conf: mailcmd=… vm_mailcmd.py
|
||||
│ Asterisk │ ─────────────────────────────┐
|
||||
└─────────────┘ pipes an RFC822 message │
|
||||
(notification + audio) ▼
|
||||
┌───────────────┐
|
||||
│ vm_mailcmd.py │
|
||||
└───────┬───────┘
|
||||
┌──────────────┬──────────────┬────┴─────────┬──────────────┐
|
||||
▼ ▼ ▼ ▼ ▼
|
||||
faster-whisper summarise() vm_contacts vm_store vm_telegram
|
||||
(transcribe) + intents (name lookup) (SQLite + (voice note
|
||||
+ numbers audio CAS) DM)
|
||||
│ │
|
||||
▼ ▼
|
||||
multipart email ──▶ Postfix :25 ┌──────────────┐
|
||||
│ vm_web.py │
|
||||
│ (FastAPI) │
|
||||
└──────┬───────┘
|
||||
│ :8099 loopback
|
||||
▼
|
||||
Apache (TLS)
|
||||
│
|
||||
▼
|
||||
https://vm.txt3.net
|
||||
```
|
||||
|
||||
**Delivery order is deliberate**: email first, then the database, then Telegram.
|
||||
Each later stage is wrapped so a failure only logs. If anything throws at the
|
||||
top level, the *original* Asterisk notification is relayed unchanged. A
|
||||
voicemail notification is never lost because a summariser or an API failed.
|
||||
|
||||
---
|
||||
|
||||
## Components
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `src/vm_mailcmd.py` | The `mailcmd` — entry point for every voicemail. Orchestrates everything. |
|
||||
| `src/vm_store.py` | SQLite store: schema, settings, sessions, content-addressed audio. |
|
||||
| `src/vm_telegram.py` | Telegram delivery, per-mailbox routing. |
|
||||
| `src/vm_contacts.py` | Caller-ID → name via local export, Google People API, or CardDAV. |
|
||||
| `src/vm_web.py` | FastAPI portal: login, list, play, delete, settings. |
|
||||
| `src/vm_auth.py` | Parses `voicemail.conf` so users log in with their phone PIN. |
|
||||
| `src/vm_import.py` | Backfills existing spool recordings into the database. |
|
||||
| `src/vm_tg_setup.py` | Helper: discover Telegram chat IDs, test a route. |
|
||||
|
||||
---
|
||||
|
||||
## Install
|
||||
|
||||
See **[docs/INSTALL.md](docs/INSTALL.md)** for the full walkthrough. Short version:
|
||||
|
||||
```bash
|
||||
sudo scripts/install.sh # venv, deps, model cache, mailcmd wiring
|
||||
sudo cp systemd/vm-portal.service /etc/systemd/system/
|
||||
sudo systemctl enable --now vm-portal
|
||||
# then follow docs/INSTALL.md §5 for the Apache vhost + TLS
|
||||
```
|
||||
|
||||
Backfill your existing voicemails:
|
||||
|
||||
```bash
|
||||
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||
/opt/vm-transcribe/vm_import.py --dry-run # preview
|
||||
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||
/opt/vm-transcribe/vm_import.py # do it
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
| Document | Contents |
|
||||
|---|---|
|
||||
| [docs/INSTALL.md](docs/INSTALL.md) | Step-by-step install, including TLS ordering |
|
||||
| [docs/CONFIGURATION.md](docs/CONFIGURATION.md) | Every config file and option |
|
||||
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Design decisions and why |
|
||||
| [docs/OPERATIONS.md](docs/OPERATIONS.md) | Day-to-day running, backup, troubleshooting |
|
||||
| [docs/SECURITY.md](docs/SECURITY.md) | Threat model, hardening, privacy |
|
||||
| [docs/TESTING.md](docs/TESTING.md) | How to verify each part |
|
||||
| [CHANGELOG.md](CHANGELOG.md) | Version history |
|
||||
|
||||
---
|
||||
|
||||
## Requirements
|
||||
|
||||
- Asterisk with `app_voicemail` (file-based spool, not ODBC/IMAP storage)
|
||||
- Python 3.9+
|
||||
- `ffmpeg` (Telegram voice-note transcoding; `sox` optionally for gsm)
|
||||
- An MTA listening on `localhost:25` (Postfix here)
|
||||
- ~200 MB disk for the whisper `base.en` model
|
||||
- Apache with `proxy`, `proxy_http`, `headers`, `rewrite`, `ssl` for the portal
|
||||
|
||||
Runs comfortably on 4 CPU cores with no GPU: ~8 s to transcribe 23 s of audio.
|
||||
|
||||
---
|
||||
|
||||
## Design notes worth knowing
|
||||
|
||||
**Summarisation is local and extractive.** Frequency-scored sentence selection
|
||||
with position/digit weighting, plus regex intent tags. No LLM, by choice — it
|
||||
keeps voicemail content on your own hardware. `summarise()` in
|
||||
`vm_mailcmd.py` is a single swap-in point if you want an abstractive model.
|
||||
|
||||
**Audio is content-addressed, not referenced by spool path.** Asterisk renumbers
|
||||
`msgNNNN` files when a message is deleted, so a stored path silently starts
|
||||
pointing at the wrong recording. Recordings are copied to
|
||||
`audio/<sha[:2]>/<sha>.wav`.
|
||||
|
||||
**App passwords cannot read Google Contacts.** Google disabled basic auth for
|
||||
CardDAV/CalDAV/IMAP/SMTP/POP on 2024-09-30. Use the `file` backend (a vCard
|
||||
export) or the `google` backend (OAuth). The `carddav` backend with an app
|
||||
password works for Nextcloud, Fastmail and iCloud.
|
||||
|
||||
**The portal ships zero JavaScript**, which lets its CSP be `script-src 'none'`.
|
||||
|
||||
---
|
||||
|
||||
## Licence
|
||||
|
||||
MIT — see [LICENSE](LICENSE).
|
||||
12
apache/vm.txt3.net-step1.conf
Normal file
12
apache/vm.txt3.net-step1.conf
Normal file
@ -0,0 +1,12 @@
|
||||
<VirtualHost 51.68.212.39:80 [2001:41d0:801:2000::2245]:80>
|
||||
ServerName vm.txt3.net
|
||||
DocumentRoot /home/txt3/domains/vm.txt3.net/public_html
|
||||
ErrorLog /var/log/virtualmin/vm.txt3.net_error_log
|
||||
CustomLog /var/log/virtualmin/vm.txt3.net_access_log combined
|
||||
|
||||
<Directory /home/txt3/domains/vm.txt3.net/public_html>
|
||||
Require all granted
|
||||
Options -Indexes
|
||||
AllowOverride None
|
||||
</Directory>
|
||||
</VirtualHost>
|
||||
54
apache/vm.txt3.net.conf
Normal file
54
apache/vm.txt3.net.conf
Normal file
@ -0,0 +1,54 @@
|
||||
<VirtualHost 51.68.212.39:80 [2001:41d0:801:2000::2245]:80>
|
||||
ServerName vm.txt3.net
|
||||
ErrorLog /var/log/virtualmin/vm.txt3.net_error_log
|
||||
CustomLog /var/log/virtualmin/vm.txt3.net_access_log combined
|
||||
|
||||
# Let certbot answer HTTP-01 challenges from the webroot
|
||||
Alias /.well-known/acme-challenge/ /home/txt3/domains/vm.txt3.net/public_html/.well-known/acme-challenge/
|
||||
<Directory /home/txt3/domains/vm.txt3.net/public_html/.well-known/acme-challenge>
|
||||
Require all granted
|
||||
Options -Indexes
|
||||
</Directory>
|
||||
ProxyPass /.well-known !
|
||||
|
||||
# Everything else goes to HTTPS
|
||||
RewriteEngine on
|
||||
RewriteCond %{HTTPS} !=on
|
||||
RewriteRule ^/(?!\.well-known)(.*)$ https://vm.txt3.net/$1 [R=301,L]
|
||||
</VirtualHost>
|
||||
|
||||
<VirtualHost 51.68.212.39:443 [2001:41d0:801:2000::2245]:443>
|
||||
ServerName vm.txt3.net
|
||||
ErrorLog /var/log/virtualmin/vm.txt3.net_error_log
|
||||
CustomLog /var/log/virtualmin/vm.txt3.net_access_log combined
|
||||
|
||||
SSLEngine on
|
||||
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
|
||||
# Replaced by certbot with the vm.txt3.net cert once issued.
|
||||
SSLCertificateFile /etc/letsencrypt/live/vm.txt3.net/fullchain.pem
|
||||
SSLCertificateKeyFile /etc/letsencrypt/live/vm.txt3.net/privkey.pem
|
||||
|
||||
# --- security headers -------------------------------------------------
|
||||
Header always set X-Content-Type-Options "nosniff"
|
||||
Header always set X-Frame-Options "DENY"
|
||||
Header always set Referrer-Policy "strict-origin-when-cross-origin"
|
||||
Header always set Strict-Transport-Security "max-age=15768000"
|
||||
# The app uses only inline <style>, no external or inline JS.
|
||||
Header always set Content-Security-Policy "default-src 'self'; style-src 'self' 'unsafe-inline'; media-src 'self'; img-src 'self' data:; script-src 'none'; frame-ancestors 'none'; base-uri 'none'; form-action 'self'"
|
||||
|
||||
# --- reverse proxy to the voicemail portal ----------------------------
|
||||
ProxyPreserveHost On
|
||||
ProxyRequests Off
|
||||
# Recordings can be a few MB; give uploads/streams room.
|
||||
ProxyTimeout 120
|
||||
|
||||
ProxyPass /.well-known !
|
||||
ProxyPass / http://127.0.0.1:8099/ retry=0
|
||||
ProxyPassReverse / http://127.0.0.1:8099/
|
||||
|
||||
RequestHeader set X-Forwarded-Proto "https"
|
||||
|
||||
<Location />
|
||||
Require all granted
|
||||
</Location>
|
||||
</VirtualHost>
|
||||
14
config/README.md
Normal file
14
config/README.md
Normal file
@ -0,0 +1,14 @@
|
||||
# Examples
|
||||
|
||||
These files are templates. **Copy to a live path and fill in secrets** — the
|
||||
live files are git-ignored so credentials never get committed.
|
||||
|
||||
- `config/telegram.conf.example` → `/opt/vm-transcribe/telegram.conf`
|
||||
- `config/contacts.conf.example` → `/opt/vm-transcribe/contacts.conf`
|
||||
|
||||
```bash
|
||||
sudo install -o root -g asterisk -m 640 config/telegram.conf.example /opt/vm-transcribe/telegram.conf
|
||||
sudo install -o root -g asterisk -m 640 config/contacts.conf.example /opt/vm-transcribe/contacts.conf
|
||||
```
|
||||
|
||||
Then edit the real files with `sudo` and add your bot token / contacts source.
|
||||
73
config/contacts.conf.example
Normal file
73
config/contacts.conf.example
Normal file
@ -0,0 +1,73 @@
|
||||
# /opt/vm-transcribe/contacts.conf
|
||||
#
|
||||
# Caller-ID -> name lookup for voicemail notifications.
|
||||
# Re-read on every voicemail; no restart needed.
|
||||
#
|
||||
# chmod 640, owned root:asterisk - may contain a password.
|
||||
|
||||
[contacts]
|
||||
# Master switch. If no, notifications just show the raw caller ID.
|
||||
enabled = yes
|
||||
|
||||
# Which backends to try, in order, comma separated:
|
||||
# file - a local vCard (.vcf) or CSV export. No auth, fast, works offline.
|
||||
# google - Google People API using the Hermes OAuth token (recommended
|
||||
# for Google accounts; see note about app passwords below).
|
||||
# carddav - any CardDAV server with username + app password.
|
||||
backends = file, google
|
||||
|
||||
# Successful and failed lookups are cached here to avoid hammering an API
|
||||
# on every call. Delete the file to force a refresh.
|
||||
cache_path = /var/lib/vm-transcribe/contacts_cache.json
|
||||
cache_ttl = 86400
|
||||
|
||||
# Match on the last N digits of the number, so +447941223856,
|
||||
# 07941223856 and 447941223856 all resolve to the same contact.
|
||||
# 9 is a sane default for UK/US. Lower it only if you get misses.
|
||||
match_digits = 9
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
[file]
|
||||
# vCard (.vcf) or CSV exported from Google Contacts:
|
||||
# contacts.google.com -> Export -> vCard (or Google CSV)
|
||||
# This is the most reliable option: no tokens, no API limits, no network.
|
||||
path = /var/lib/vm-transcribe/contacts.vcf
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
[google]
|
||||
# Uses the OAuth token from the Hermes google-workspace skill, which must be
|
||||
# authorised with the People API / contacts.readonly scope.
|
||||
token_path = /var/lib/vm-transcribe/google_token.json
|
||||
|
||||
# Optional: refresh the local export from the People API on this schedule
|
||||
# (seconds). 0 = only look up on demand. Bulk-syncing to the file backend is
|
||||
# much faster per call than a live query.
|
||||
sync_interval = 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
[carddav]
|
||||
# Generic CardDAV with an app password.
|
||||
#
|
||||
# !! IMPORTANT: this will NOT work against Google. Google disabled basic
|
||||
# !! authentication for CardDAV, CalDAV, IMAP, SMTP and POP on 2024-09-30;
|
||||
# !! app passwords are rejected and Google's CardDAV now requires OAuth 2.0.
|
||||
# !! Use the 'google' or 'file' backend for Google Contacts instead.
|
||||
#
|
||||
# These settings do work for Nextcloud, Fastmail, iCloud, Radicale, etc.
|
||||
url =
|
||||
username =
|
||||
|
||||
# App password (NOT the main account password). Generate one in your
|
||||
# provider's security settings.
|
||||
app_password =
|
||||
|
||||
# Some providers need an explicit addressbook path, e.g.
|
||||
# Nextcloud: /remote.php/dav/addressbooks/users/<user>/contacts/
|
||||
# iCloud: discovered automatically from the principal URL
|
||||
addressbook_path =
|
||||
|
||||
verify_tls = yes
|
||||
timeout = 20
|
||||
51
config/telegram.conf.example
Normal file
51
config/telegram.conf.example
Normal file
@ -0,0 +1,51 @@
|
||||
# /opt/vm-transcribe/telegram.conf
|
||||
#
|
||||
# Telegram delivery for voicemail notifications.
|
||||
# Reloaded on every voicemail - no restart needed after editing.
|
||||
#
|
||||
# chmod 640, owned root:asterisk - it contains a bot token.
|
||||
|
||||
[telegram]
|
||||
# Master switch. Set to no to disable Telegram entirely (email still sends).
|
||||
enabled = yes
|
||||
|
||||
# Bot token from @BotFather, e.g. 123456789:AAH...
|
||||
token =
|
||||
|
||||
# Where to send when a mailbox has no [mailbox:N] section below.
|
||||
# Leave blank to send nothing for unmapped mailboxes.
|
||||
default_chat_id =
|
||||
|
||||
# Attach the recording as a Telegram voice note (ogg/opus, plays inline).
|
||||
send_audio = yes
|
||||
|
||||
# Include the full transcript as a follow-up message when it is longer
|
||||
# than the summary. Set to no for summary-only.
|
||||
send_transcript = yes
|
||||
|
||||
# Seconds to wait on the Telegram API before giving up. Email is already
|
||||
# sent by this point, so a timeout is never fatal.
|
||||
timeout = 20
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-mailbox routing. Section name is [mailbox:<mailbox number>]
|
||||
# chat_id may be a comma-separated list to fan out to several people.
|
||||
#
|
||||
# To find a chat ID: message your bot, then visit
|
||||
# https://api.telegram.org/bot<TOKEN>/getUpdates
|
||||
# and read result[].message.chat.id (negative numbers are groups).
|
||||
#
|
||||
# Any key from [telegram] can be overridden per mailbox.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# [mailbox:1001]
|
||||
# chat_id = 123456789
|
||||
|
||||
# [mailbox:1002]
|
||||
# chat_id = 987654321, 123456789
|
||||
# send_transcript = no
|
||||
|
||||
# [mailbox:1003]
|
||||
# chat_id = -1001234567890
|
||||
# send_audio = no
|
||||
137
docs/ARCHITECTURE.md
Normal file
137
docs/ARCHITECTURE.md
Normal file
@ -0,0 +1,137 @@
|
||||
# Architecture
|
||||
|
||||
How the pieces fit and *why* they are shaped the way they are. Read this
|
||||
before changing anything in `vm_mailcmd.py` or `vm_store.py`.
|
||||
|
||||
---
|
||||
|
||||
## Flow
|
||||
|
||||
```
|
||||
Asterisk ──(RFC822 on stdin)──▶ vm_mailcmd.py
|
||||
│
|
||||
┌──────────┬───────────────┼───────────────┬───────────────┐
|
||||
▼ ▼ ▼ ▼ ▼
|
||||
transcribe summarise resolve store telegram
|
||||
(whisper) +intents contact (sqlite) (DM)
|
||||
│ │ │ │ │
|
||||
└────▶ build multipart email ─▶ Postfix :25
|
||||
(plain + HTML + audio attached)
|
||||
│
|
||||
└──▶ vm_store.add_message() (best-effort)
|
||||
└──▶ vm_telegram.send() (best-effort)
|
||||
```
|
||||
|
||||
**Email is sent before anything else.** The portal and Telegram are conveniences;
|
||||
the email is the contract with the user. If the database is down or Telegram's
|
||||
API throws, the mail has already gone. Each later stage is in its own `try/except`
|
||||
that only logs. The reverse — a transcription error blocking the email — never
|
||||
happens, because the email builder is given whatever text we have, even `""`.
|
||||
|
||||
At the very top, the whole `main()` body is wrapped so that *any* unexpected
|
||||
exception relays the **original** Asterisk message unchanged. Nothing we add can
|
||||
silently swallow a voicemail.
|
||||
|
||||
---
|
||||
|
||||
## Why faster-whisper, CPU, int8, base.en
|
||||
|
||||
- **faster-whisper** (CTranslate2) is dramatically faster than OpenAI's
|
||||
`whisper` on CPU — ~8 s for 23 s of 8 kHz audio here, versus minutes.
|
||||
- **CPU/int8** because this host has no GPU and the voicemail rate is low. A
|
||||
GPU would be idle 99.9 % of the time.
|
||||
- **base.en** is plenty for clear phone audio; `small.en` only if you see
|
||||
accuracy problems with names/numbers. Set `VM_WHISPER_MODEL`.
|
||||
|
||||
The model is loaded once per process. Asterisk fires a fresh `mailcmd` per
|
||||
voicemail, so there is a ~1 s model-load cost *per message* — acceptable at PBX
|
||||
volumes, and the model is pre-cached at install so the first call doesn't also
|
||||
pay a download.
|
||||
|
||||
---
|
||||
|
||||
## Why the summary is extractive, not an LLM
|
||||
|
||||
The user explicitly declined both OpenAI and Ollama. `summarise()` is:
|
||||
|
||||
1. split into sentences,
|
||||
2. score each by word frequency (stop-words removed) plus a boost for
|
||||
position (first/last sentence) and for sentences containing digits,
|
||||
3. pick the top sentences, ordered as spoken, capped at ~3 and a length limit,
|
||||
4. wrap with regex intent tags (`Call back requested`, `Urgent`,
|
||||
`Payment / invoice`, …) and number extraction.
|
||||
|
||||
It reads well on phone voicemail and needs no network. The single function
|
||||
`summarise()` is a deliberate **swap-in point** — replace it with a local model
|
||||
or an API call without touching the rest of the pipeline.
|
||||
|
||||
---
|
||||
|
||||
## Why audio is content-addressed
|
||||
|
||||
Asterisk **renumbers** `msgNNNN` files when a message is deleted from a folder.
|
||||
A stored path like `…/INBOX/msg0042.wav` therefore points at a *different*
|
||||
recording a week later. So:
|
||||
|
||||
- On store, the audio is copied to
|
||||
`audio/<sha256[:2]>/<sha256>.<ext>` (blake2/sha256 in `vm_store`).
|
||||
- The spool path is kept only as a *delete hint*.
|
||||
- On delete, the blob is unlinked only if no other row references that hash.
|
||||
|
||||
This decouples the portal's playback from Asterisk's ever-shifting filenames.
|
||||
|
||||
---
|
||||
|
||||
## Why the portal is its own service behind Apache
|
||||
|
||||
- A long-running `uvicorn` is the natural fit; Asterisk's `mailcmd` model
|
||||
(one short-lived process per call) is wrong for a web app.
|
||||
- It binds **127.0.0.1:8099 only** (`IPAddressAllow=localhost` in the unit), so
|
||||
the only way in is the Apache reverse proxy, which terminates TLS and enforces
|
||||
headers.
|
||||
- Running as the `asterisk` user means it can read `voicemail.conf` (for PIN
|
||||
auth) and the spool (for future direct-file features) without widening root.
|
||||
|
||||
---
|
||||
|
||||
## Auth model
|
||||
|
||||
Users log in with their **mailbox number + existing voicemail PIN** parsed from
|
||||
`/etc/asterisk/voicemail.conf` (`vm_auth.py`). No password database to keep in
|
||||
sync, no new credentials to provision. Sessions are stored in the DB (random
|
||||
token, `Secure` cookie, `HttpOnly`, `SameSite=lax`, 12 h) so logout and expiry
|
||||
are enforced server-side, not just by deleting a cookie.
|
||||
|
||||
Every data query is filtered by the session's mailbox. Cross-mailbox access must
|
||||
404 — there is a test for this in [TESTING.md](TESTING.md).
|
||||
|
||||
---
|
||||
|
||||
## Lockout
|
||||
|
||||
Voicemail PINs are short (often 4 digits) and the portal is internet-facing, so
|
||||
failed logins are throttled in-process: 5 per `(mailbox, IP)` then 15 minutes.
|
||||
State is in memory by design (single worker). A restart clears it — fine, because
|
||||
a restart is an admin action, not an attacker one.
|
||||
|
||||
---
|
||||
|
||||
## Telegram routing
|
||||
|
||||
`telegram.conf` is re-read on every voicemail (no reload). Resolution order:
|
||||
`[mailbox:N]` `chat_id` → `default_chat_id` → nothing. A chat id may fan out to
|
||||
several recipients (comma list) or be a group (negative). Per-mailbox sections
|
||||
can override any global (`send_audio`, `send_transcript`). Delivery degrades
|
||||
`sendVoice → sendDocument → sendMessage` so a summary always gets through.
|
||||
|
||||
---
|
||||
|
||||
## Privacy posture
|
||||
|
||||
All audio and transcripts stay on the PBX. The only egress is:
|
||||
|
||||
- **Postfix → your MX** for email (the existing path),
|
||||
- **Telegram API** (only if you enable Telegram),
|
||||
- **Google People API** (only if you enable the `google` contacts backend).
|
||||
|
||||
Nothing is sent to OpenAI, Anthropic, or any transcription vendor.
|
||||
159
docs/CONFIGURATION.md
Normal file
159
docs/CONFIGURATION.md
Normal file
@ -0,0 +1,159 @@
|
||||
# Configuration
|
||||
|
||||
All config lives in `/opt/vm-transcribe/`. Templates are committed as
|
||||
`*.example`; the live files are created by `install.sh` or by you. Secret-bearing
|
||||
files (`telegram.conf`, `contacts.conf`) are mode `640 root:asterisk` and git-
|
||||
ignored.
|
||||
|
||||
Most settings can also be set via environment variables, which is how the systemd
|
||||
unit wires production values. Env vars override file values when both are present
|
||||
unless noted.
|
||||
|
||||
---
|
||||
|
||||
## Environment variables (shared)
|
||||
|
||||
| Variable | Default | Used by | Meaning |
|
||||
|---|---|---|---|
|
||||
| `VM_DB` | `/var/lib/vm-transcribe/voicemail.db` | store, web, import | SQLite path. Created with WAL mode if missing. |
|
||||
| `VM_AUDIO_DIR` | `/var/lib/vm-transcribe/audio` | store, web, import | Where copied recordings live (`<sha[:2]>/<sha>.wav`). |
|
||||
| `VM_LOG` | `/var/log/asterisk/vm_mailcmd.log` | mailcmd | Log path. |
|
||||
| `VM_MODEL_CACHE` | `/opt/vm-transcribe/models` | mailcmd, import | Whisper model download/load dir. |
|
||||
| `VM_WHISPER_MODEL` | `base.en` | mailcmd, import | Model name. `tiny`/`base`/`small` trade speed for accuracy. |
|
||||
| `VM_SPOOL` | `/var/spool/asterisk/voicemail` | import | Spool root to scan. |
|
||||
| `VM_ASTERISK_CONF` | `/etc/asterisk/voicemail.conf` | web, auth | For PIN lookup. |
|
||||
|
||||
---
|
||||
|
||||
## voicemail.conf (Asterisk)
|
||||
|
||||
One line is all the pipeline needs:
|
||||
|
||||
```ini
|
||||
mailcmd=/opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_mailcmd.py
|
||||
```
|
||||
|
||||
Place it in the `[general]` section (or any context). After editing:
|
||||
|
||||
```bash
|
||||
sudo asterisk -rx 'voicemail reload'
|
||||
```
|
||||
|
||||
`install.sh` backs up the file and adds this line automatically.
|
||||
|
||||
---
|
||||
|
||||
## telegram.conf
|
||||
|
||||
```ini
|
||||
[telegram]
|
||||
enabled = yes
|
||||
token = 123456789:AAH... ; from @BotFather
|
||||
default_chat_id = ; fallback for unmapped mailboxes (blank = none)
|
||||
send_audio = yes ; voice note with caption
|
||||
send_transcript = yes ; full transcript as a follow-up message
|
||||
timeout = 20 ; seconds per send attempt
|
||||
|
||||
[mailbox:1001]
|
||||
chat_id = 123456789 ; single recipient
|
||||
|
||||
[mailbox:1002]
|
||||
chat_id = 5551, 5552 ; comma list fans out to several people
|
||||
send_transcript = no ; any global key can be overridden per mailbox
|
||||
|
||||
[mailbox:1003]
|
||||
chat_id = -1001234567890 ; negative = group chat
|
||||
send_audio = no ; text-only DM
|
||||
```
|
||||
|
||||
**Routing rules** (evaluated per voicemail):
|
||||
|
||||
1. Match the `[mailbox:N]` section for the voicemail's mailbox number.
|
||||
2. `chat_id` may be a single id, a comma-separated list (fan-out), or a group id
|
||||
(negative).
|
||||
3. If no mailbox section matches, fall back to `default_chat_id`.
|
||||
4. If `enabled = no` or no route resolves, nothing is sent — email still goes out.
|
||||
5. The file is re-read on every voicemail; no reload needed after editing.
|
||||
|
||||
**Delivery fallback chain** (per chat id, tried in order): `sendVoice` →
|
||||
`sendDocument` → `sendMessage` (text-only). So a Telegram outage or a bad audio
|
||||
file still delivers the summary.
|
||||
|
||||
> Contact resolution note: a user must message the bot once before it can DM
|
||||
> them. Telegram does not permit bots to initiate conversations.
|
||||
|
||||
---
|
||||
|
||||
## contacts.conf
|
||||
|
||||
```ini
|
||||
[contacts]
|
||||
backends = file ; space/comma separated: file google carddav
|
||||
cache_ttl = 86400 ; seconds; cache hits AND misses
|
||||
cache_file = /var/lib/vm-transcribe/contacts-cache.json
|
||||
|
||||
# --- 'file' backend (recommended): a vCard or CSV export -------------
|
||||
[file]
|
||||
path = /var/lib/vm-transcribe/contacts.vcf
|
||||
# CSV (Google export "All contact data") is auto-detected:
|
||||
# path = /var/lib/vm-transcribe/contacts.csv
|
||||
|
||||
# --- 'google' backend: People API via OAuth ---------------------------
|
||||
[google]
|
||||
token_file = /var/lib/vm-transcribe/google-token.json
|
||||
# Obtain a token with the People API (contacts.readonly) scope. See
|
||||
# docs/SECURITY.md. No app password here — OAuth only.
|
||||
|
||||
# --- 'carddav' backend: app password, Nextcloud/Fastmail/iCloud --------
|
||||
[carddav]
|
||||
# Google rejects app passwords (basic auth disabled 2024-09-30); a Google
|
||||
# URL here is refused with a log warning and the backend is skipped.
|
||||
url = https://contacts.fastmail.com/dav/addressbooks/user/xxx/Default
|
||||
username = jp@txt3.com
|
||||
app_password = ; app-specific password, not your login password
|
||||
```
|
||||
|
||||
**Matching** is on the **last 9 digits** of the caller ID, so `+447700900123`,
|
||||
`07700900123` and `447700900123` all collapse to the same contact. Names resolve
|
||||
best-effort; on any backend error the caller ID is used as-is.
|
||||
|
||||
---
|
||||
|
||||
## Portal settings (per mailbox, in the SQLite store)
|
||||
|
||||
These are edited from the portal's **Settings** page by each user; they are not
|
||||
config files. Every value defaults to `yes`.
|
||||
|
||||
| Setting | Effect |
|
||||
|---|---|
|
||||
| `summarise` | Run the summariser (if off, the raw transcript is shown). |
|
||||
| `telegram` | Forward new voicemails to the mailbox's Telegram route (requires telegram.conf). |
|
||||
| `email_notify` | Send the enriched email (if off, the message is stored but not emailed). |
|
||||
| `highlight` | Colour the caller chip when a name is resolved from contacts. |
|
||||
| `max_inbox` | Keep at most N messages; oldest beyond N are auto-deleted (0 = unlimited). |
|
||||
|
||||
---
|
||||
|
||||
## vm-portal.service
|
||||
|
||||
Production values are passed as environment in the unit file:
|
||||
|
||||
```ini
|
||||
Environment=VM_DB=/var/lib/vm-transcribe/voicemail.db
|
||||
Environment=VM_AUDIO_DIR=/var/lib/vm-transcribe/audio
|
||||
Environment=VM_ASTERISK_CONF=/etc/asterisk/voicemail.conf
|
||||
Environment=VM_SESSION_HOURS=12
|
||||
```
|
||||
|
||||
Login hardening (in-process):
|
||||
|
||||
| Var | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `VM_MAX_FAILS` | `5` | Failed attempts per (mailbox, IP) before locking. |
|
||||
| `VM_LOCK_MINUTES` | `15` | Lock duration. |
|
||||
| `VM_SESSION_HOURS` | `12` | Session lifetime. |
|
||||
| `VM_INSECURE_COOKIE` | unset | Set to `1` only for plain-HTTP testing; normally cookies are `Secure`. |
|
||||
| `VM_BASE_PATH` | `""` | Set to e.g. `/voicemail` if the app is served under a sub-path. |
|
||||
|
||||
> A service restart clears the in-memory lockout counters. Testing lockout will
|
||||
> lock you out of your own verification for `VM_LOCK_MINUTES`.
|
||||
244
docs/INSTALL.md
Normal file
244
docs/INSTALL.md
Normal file
@ -0,0 +1,244 @@
|
||||
# Installation
|
||||
|
||||
Written against Debian 12 + Asterisk 20 + Apache 2.4 + Postfix, running under
|
||||
Virtualmin. Adjust paths if your layout differs.
|
||||
|
||||
Everything installs to `/opt/vm-transcribe` with data in
|
||||
`/var/lib/vm-transcribe`. Nothing is installed into system Python.
|
||||
|
||||
---
|
||||
|
||||
## 0. Prerequisites
|
||||
|
||||
```bash
|
||||
# Asterisk voicemail must use file-based storage (the default), not ODBC/IMAP
|
||||
grep -E '^(odbcstorage|imapserver)' /etc/asterisk/voicemail.conf # expect nothing
|
||||
|
||||
# tools
|
||||
sudo apt install ffmpeg sox python3-venv # sox optional (gsm)
|
||||
which certbot # for the portal's TLS
|
||||
|
||||
# an MTA on localhost:25
|
||||
sudo ss -ltnp | grep ':25 '
|
||||
```
|
||||
|
||||
**Check disk space before you start.** A full `/var` makes Postfix reject all
|
||||
mail with `452 4.3.1 Insufficient system storage`, which looks like a bug in
|
||||
this pipeline but is not:
|
||||
|
||||
```bash
|
||||
df -h /var
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Core pipeline
|
||||
|
||||
```bash
|
||||
sudo scripts/install.sh
|
||||
```
|
||||
|
||||
That script:
|
||||
|
||||
1. creates `/opt/vm-transcribe` and `/opt/vm-transcribe/models` owned by `asterisk`
|
||||
2. builds a venv and installs `faster-whisper`
|
||||
3. installs the Python modules
|
||||
4. seeds `telegram.conf` / `contacts.conf` if absent (never overwrites)
|
||||
5. creates `/var/log/asterisk/vm_mailcmd.log`
|
||||
6. **pre-downloads the whisper model as the `asterisk` user** — do not skip
|
||||
this, or the first real voicemail pays a ~150 MB download while the caller
|
||||
waits for their notification
|
||||
7. backs up `voicemail.conf` and rewrites `mailcmd=`
|
||||
8. reloads Asterisk
|
||||
|
||||
Verify the wiring:
|
||||
|
||||
```bash
|
||||
grep '^mailcmd' /etc/asterisk/voicemail.conf
|
||||
# mailcmd=/opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_mailcmd.py
|
||||
```
|
||||
|
||||
### Test it before trusting it
|
||||
|
||||
Build a fake notification and pipe it through **as the `asterisk` user** —
|
||||
testing as yourself hides permission problems:
|
||||
|
||||
```bash
|
||||
cd /path/to/repo
|
||||
python3 tests/make_test_mail.py /path/to/some.wav you@example.com > /tmp/t.eml
|
||||
sudo chmod 644 /tmp/t.eml
|
||||
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||
/opt/vm-transcribe/vm_mailcmd.py < /tmp/t.eml
|
||||
sudo tail -5 /var/log/asterisk/vm_mailcmd.log
|
||||
```
|
||||
|
||||
You should see `transcribed … chars` then `sent enriched notification to …`.
|
||||
Confirm the mail actually left:
|
||||
|
||||
```bash
|
||||
sudo grep "to=<you@example.com>" /var/log/mail.log | tail -1 # expect status=sent
|
||||
```
|
||||
|
||||
Also verify the fail-safe — a message with **no** audio must relay unchanged
|
||||
rather than erroring:
|
||||
|
||||
```bash
|
||||
printf 'From: a@b\nTo: you@example.com\nSubject: no audio\n\nbody\n' > /tmp/n.eml
|
||||
sudo chmod 644 /tmp/n.eml
|
||||
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||
/opt/vm-transcribe/vm_mailcmd.py < /tmp/n.eml
|
||||
# log: "no audio attachment; relaying original"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Telegram (optional)
|
||||
|
||||
1. Create a bot: message [@BotFather](https://t.me/BotFather) → `/newbot` → copy the token.
|
||||
2. Put it in `/opt/vm-transcribe/telegram.conf` (`sudo`, mode 640 root:asterisk).
|
||||
3. **Each recipient must message the bot once** (`/start`) — Telegram forbids
|
||||
bots from initiating conversations.
|
||||
4. Discover chat IDs and test:
|
||||
|
||||
```bash
|
||||
sudo /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py ids
|
||||
sudo /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py test 1001
|
||||
```
|
||||
|
||||
The `test` subcommand sends a real DM with a playable voice note and exits
|
||||
non-zero on failure. See [CONFIGURATION.md](CONFIGURATION.md#telegramconf) for
|
||||
per-mailbox routing.
|
||||
|
||||
---
|
||||
|
||||
## 3. Contact lookup (optional)
|
||||
|
||||
Simplest and most reliable — a local export, no tokens, no rate limits:
|
||||
|
||||
1. contacts.google.com → **Export** → **vCard**
|
||||
2. `sudo install -o asterisk -g asterisk -m 640 contacts.vcf /var/lib/vm-transcribe/contacts.vcf`
|
||||
3. Confirm `contacts.conf` has `backends = file` and the matching `path`.
|
||||
|
||||
```bash
|
||||
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||
/opt/vm-transcribe/vm_contacts.py '<07700900123>'
|
||||
```
|
||||
|
||||
> **Google app passwords do not work for contacts.** Google disabled basic auth
|
||||
> for CardDAV/CalDAV/IMAP/SMTP/POP on 2024-09-30. Use the `file` or `google`
|
||||
> (OAuth) backend. The `carddav` backend is for Nextcloud/Fastmail/iCloud.
|
||||
|
||||
---
|
||||
|
||||
## 4. Portal service
|
||||
|
||||
```bash
|
||||
sudo install -d -o asterisk -g asterisk -m 750 \
|
||||
/var/lib/vm-transcribe /var/lib/vm-transcribe/audio
|
||||
sudo /opt/vm-transcribe/venv/bin/pip install fastapi 'uvicorn[standard]' python-multipart
|
||||
sudo install -o root -g root -m 644 systemd/vm-portal.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now vm-portal
|
||||
sudo systemctl is-active vm-portal
|
||||
curl -s http://127.0.0.1:8099/healthz # {"ok":true,"messages":N}
|
||||
```
|
||||
|
||||
The service runs as `asterisk` (so it can read `voicemail.conf` and the spool)
|
||||
and binds **loopback only** — `IPAddressAllow=localhost` means Apache is the
|
||||
only possible client.
|
||||
|
||||
---
|
||||
|
||||
## 5. Apache vhost + TLS
|
||||
|
||||
**Order matters.** Installing a vhost that references a cert which does not yet
|
||||
exist breaks `apache2ctl configtest`, which blocks reloads for *every* site on
|
||||
the box. So: DNS → cert → full vhost.
|
||||
|
||||
### 5a. DNS
|
||||
|
||||
```bash
|
||||
dig +short vm.example.com A # must return your ORIGIN ip
|
||||
```
|
||||
|
||||
Behind Cloudflare, set the record to **DNS only (grey cloud)** first, or the
|
||||
HTTP-01 challenge is intercepted by the edge.
|
||||
|
||||
### 5b. Minimal :80 vhost, so certbot can answer the challenge
|
||||
|
||||
```bash
|
||||
sudo install -d -o www-data -g www-data -m 755 /var/www/vm.example.com/public_html
|
||||
sudo cp apache/vm.txt3.net-step1.conf /etc/apache2/sites-available/vm.example.com.conf
|
||||
# edit ServerName / paths / IPs to match your host
|
||||
sudo a2ensite vm.example.com
|
||||
sudo apache2ctl configtest && sudo systemctl reload apache2
|
||||
```
|
||||
|
||||
### 5c. Issue the certificate
|
||||
|
||||
```bash
|
||||
sudo certbot certonly --webroot -w /var/www/vm.example.com/public_html \
|
||||
-d vm.example.com --cert-name vm.example.com
|
||||
```
|
||||
|
||||
### 5d. Full vhost — HTTPS + reverse proxy
|
||||
|
||||
```bash
|
||||
sudo a2enmod proxy proxy_http headers rewrite ssl
|
||||
sudo cp apache/vm.txt3.net.conf /etc/apache2/sites-available/vm.example.com.conf
|
||||
# edit ServerName, cert paths, IPs
|
||||
sudo install -d /var/www/vm.example.com/public_html/.well-known/acme-challenge
|
||||
sudo apache2ctl configtest && sudo systemctl reload apache2
|
||||
```
|
||||
|
||||
### 5e. Verify through the real hostname
|
||||
|
||||
Not localhost — the whole point is to exercise Apache, TLS and the proxy:
|
||||
|
||||
```bash
|
||||
R="--resolve vm.example.com:443:YOUR.ORIGIN.IP"
|
||||
curl -s $R https://vm.example.com/healthz
|
||||
curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.example.com/login
|
||||
curl -s $R -D- -o /dev/null https://vm.example.com/login | grep -i 'strict-transport\|content-security'
|
||||
```
|
||||
|
||||
Behind a CDN, `curl https://host/` tests the *CDN*, not your origin. Always
|
||||
`--resolve` to the origin IP when verifying a change.
|
||||
|
||||
---
|
||||
|
||||
## 6. Backfill existing voicemails
|
||||
|
||||
```bash
|
||||
V=/opt/vm-transcribe/venv/bin/python3
|
||||
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --dry-run # preview
|
||||
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --limit 5 # trial
|
||||
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py # all
|
||||
```
|
||||
|
||||
Idempotent — safe to re-run; it resumes rather than duplicating. Expect a
|
||||
`no_speech` count: 44-byte WAVs are hung-up calls with no audio, not failures.
|
||||
|
||||
Reckon on ~1.5 s per message plus transcription time; 157 messages took 3m40s
|
||||
on 4 cores.
|
||||
|
||||
---
|
||||
|
||||
## 7. Log in
|
||||
|
||||
Browse to `https://vm.example.com/` and log in with a **mailbox number and its
|
||||
existing voicemail PIN** from `voicemail.conf`. No new passwords are created.
|
||||
|
||||
---
|
||||
|
||||
## Uninstall
|
||||
|
||||
```bash
|
||||
sudo systemctl disable --now vm-portal
|
||||
sudo rm /etc/systemd/system/vm-portal.service && sudo systemctl daemon-reload
|
||||
sudo a2dissite vm.example.com && sudo systemctl reload apache2
|
||||
# restore the original mailcmd
|
||||
sudo cp /etc/asterisk/voicemail.conf.bak-<timestamp> /etc/asterisk/voicemail.conf
|
||||
sudo asterisk -rx 'voicemail reload'
|
||||
sudo rm -rf /opt/vm-transcribe # keep /var/lib/vm-transcribe for the data
|
||||
```
|
||||
138
docs/OPERATIONS.md
Normal file
138
docs/OPERATIONS.md
Normal file
@ -0,0 +1,138 @@
|
||||
# Operations
|
||||
|
||||
Day-to-day running: backups, monitoring, common problems, upgrades.
|
||||
|
||||
---
|
||||
|
||||
## Service status
|
||||
|
||||
```bash
|
||||
sudo systemctl status vm-portal # the portal
|
||||
sudo journalctl -u vm-portal -n 50 # portal logs (uvicorn access/startup)
|
||||
sudo tail -f /var/log/asterisk/vm_mailcmd.log # per-voicemail pipeline log
|
||||
```
|
||||
|
||||
Health of the portal (works locally or via the real hostname):
|
||||
|
||||
```bash
|
||||
curl -s http://127.0.0.1:8099/healthz # {"ok":true,"messages":N}
|
||||
curl -s --resolve vm.txt3.net:443:ORIGIN_IP https://vm.txt3.net/healthz
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Logs to watch
|
||||
|
||||
| Signal | Where | Meaning |
|
||||
|---|---|---|
|
||||
| `transcribed …` | vm_mailcmd.log | a voicemail was processed OK |
|
||||
| `stored message for mailbox …` | vm_mailcmd.log | it also reached the portal DB |
|
||||
| `no audio attachment; relaying original` | vm_mailcmd.log | Asterisk sent a text-only notice (missed-call style) — expected |
|
||||
| `Telegram disabled` / `no route` | vm_mailcmd.log | Telegram skipped (per config) — expected if unset |
|
||||
| `relaying original, mailcmd error` | vm_mailcmd.log | **pipeline threw**; original email was preserved (fail-safe worked) |
|
||||
| `status=sent` | /var/log/mail.log | the enriched email left Postfix |
|
||||
| `452 4.3.1 Insufficient system storage` | /var/log/mail.log | **/var is full** — see below |
|
||||
|
||||
---
|
||||
|
||||
## Backups
|
||||
|
||||
Two things to back up — the code is reproducible, the *data* is not:
|
||||
|
||||
```bash
|
||||
# database + audio (once or twice a day is plenty)
|
||||
sudo -u asterisk tar czf /backup/vm-$(date +%F).tgz \
|
||||
-C / var/lib/vm-transcribe/voicemail.db var/lib/vm-transcribe/audio
|
||||
|
||||
# config (the live, secret-bearing files)
|
||||
sudo tar czf /backup/vm-conf-$(date +%F).tgz \
|
||||
/opt/vm-transcribe/telegram.conf /opt/vm-transcribe/contacts.conf
|
||||
```
|
||||
|
||||
The whisper model cache (`/opt/vm-transcribe/models`) is reproducible — no need
|
||||
to back it up, just re-run `install.sh`.
|
||||
|
||||
Restoring: stop the service, extract, `systemctl start vm-portal`. The DB schema
|
||||
is created on first connect, but **preserve the existing file** to keep history.
|
||||
|
||||
---
|
||||
|
||||
## Disk
|
||||
|
||||
`/var/lib/vm-transcribe/audio` grows with every voicemail. At ~290 KB per
|
||||
WAV (8000 Hz, mono, 23 s) that is ~12 MB per 40 messages. The portal's
|
||||
`max_inbox` per-user auto-delete keeps individual mailboxes bounded, but the
|
||||
*total* store only shrinks when messages are deleted (and their audio blob is
|
||||
unreferenced by any other message).
|
||||
|
||||
A full `/var` is the single most common cause of "voicemail emails stopped
|
||||
arriving" — Postfix rejects everything with `452 4.3.1 Insufficient system
|
||||
storage`. Check `df -h /var` first when the pipeline looks dead.
|
||||
|
||||
---
|
||||
|
||||
## Upgrading
|
||||
|
||||
```bash
|
||||
git -C /home/jp/Work/asterisk-voicemail pull
|
||||
cd /home/jp/Work/asterisk-voicemail
|
||||
sudo cp src/*.py /opt/vm-transcribe/
|
||||
sudo /opt/vm-transcribe/venv/bin/pip install -U faster-whisper # occasionally
|
||||
sudo systemctl restart vm-portal
|
||||
```
|
||||
|
||||
`install.sh` is idempotent-ish for the first install but is **not** a general
|
||||
upgrade tool — it backs up `voicemail.conf` each run, so avoid re-running it
|
||||
blindly. For upgrades, copy `src/*.py` as above.
|
||||
|
||||
---
|
||||
|
||||
## Common problems
|
||||
|
||||
**No transcription email arrived.**
|
||||
1. `df -h /var` — full disk blocks Postfix.
|
||||
2. `sudo grep '^mailcmd' /etc/asterisk/voicemail.conf` — is it our script?
|
||||
3. `sudo tail /var/log/asterisk/vm_mailcmd.log` — look for `relaying original, mailcmd error`.
|
||||
4. `sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 -c 'import faster_whisper'` — is the venv intact?
|
||||
|
||||
**Portal returns 502 / can't connect.**
|
||||
The uvicorn backend is down or not on loopback: `sudo systemctl status vm-portal`;
|
||||
`ss -ltnp | grep 8099`. Also confirm Apache has `proxy`/`proxy_http` enabled.
|
||||
|
||||
**Login fails for a real mailbox.**
|
||||
The PIN in `voicemail.conf` is what the portal checks — not anything in the
|
||||
portal. If you changed a voicemail PIN, that change is picked up immediately
|
||||
(the file is reparsed each login). If the mailbox line is commented out or in a
|
||||
context the parser doesn't reach, login fails. Test the parser directly:
|
||||
|
||||
```bash
|
||||
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||
-c "import sys; sys.path.insert(0,'/opt/vm-transcribe'); import vm_auth; print(vm_auth.check_login('7940','5159'))"
|
||||
```
|
||||
|
||||
**Certbot renewal.**
|
||||
Renewal runs from `/etc/cron.d/certbot`; `certbot.timer` is masked (normal on
|
||||
Debian's package). Test non-destructively in the background (it can be slow):
|
||||
|
||||
```bash
|
||||
sudo certbot renew --cert-name vm.txt3.net --dry-run &
|
||||
```
|
||||
|
||||
**I locked myself out testing lockout.**
|
||||
Restart the service: `sudo systemctl restart vm-portal`. In-memory counters clear.
|
||||
|
||||
**Backfill shows many `no_speech`.**
|
||||
Expected. 44-byte WAVs (`duration=0`) are hung-up calls with no audio. Not a bug.
|
||||
|
||||
---
|
||||
|
||||
## Re-running the backfill
|
||||
|
||||
Idempotent — safe any time:
|
||||
|
||||
```bash
|
||||
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_import.py
|
||||
```
|
||||
|
||||
Useful after importing a large batch of new spool messages, or after a fresh
|
||||
install on a box with existing voicemails.
|
||||
117
docs/SECURITY.md
Normal file
117
docs/SECURITY.md
Normal file
@ -0,0 +1,117 @@
|
||||
# Security
|
||||
|
||||
Threat model and the hardening already in place. Read before exposing the portal
|
||||
on a hostile network.
|
||||
|
||||
---
|
||||
|
||||
## What we are protecting
|
||||
|
||||
Voicemail is sensitive: it contains callers' names, numbers, and the content of
|
||||
their messages — often personal or commercial. The portal also proves who
|
||||
*isn't* a mailbox owner (knowing a PIN is the only gate). Breaches of concern:
|
||||
|
||||
1. An attacker reads someone else's voicemail (broken authz or IDOR).
|
||||
2. An attacker brute-forces a 4-digit PIN at scale.
|
||||
3. A stored credential (Telegram token, contacts app password) leaks.
|
||||
4. XSS / injection poisons the HTML shown to a user.
|
||||
5. The recording audio leaks off-box.
|
||||
|
||||
---
|
||||
|
||||
## What is already done
|
||||
|
||||
**Authentication — mailbox PIN from voicemail.conf.**
|
||||
No new password store to breach; the portal trusts Asterisk's PINs. Sessions are
|
||||
DB-backed (random token, `Secure`+`HttpOnly`+`SameSite=lax`, 12 h). Logout and
|
||||
expiry are enforced server-side, not merely by dropping the cookie.
|
||||
|
||||
**Authorization — every query is mailbox-scoped.**
|
||||
`vm_store` and `vm_web` filter every read and delete by the session's mailbox.
|
||||
Cross-mailbox reads/deletes return **404**. There is a test that asserts this
|
||||
([TESTING.md](TESTING.md)). A mailbox owner can only ever see their own messages.
|
||||
|
||||
**Brute-force lockout.**
|
||||
5 failed attempts per `(mailbox, IP)` → 15-minute lockout. The correct PIN is
|
||||
also refused while locked. State is in-memory (single worker); a restart clears
|
||||
it.
|
||||
|
||||
**Transport — TLS terminated at Apache, with headers.**
|
||||
`Strict-Transport-Security`, `X-Frame-Options: DENY`, `X-Content-Type-Options:
|
||||
nosniff`, `Referrer-Policy`, and a tight CSP:
|
||||
|
||||
```
|
||||
default-src 'self'; style-src 'self' 'unsafe-inline';
|
||||
media-src 'self'; img-src 'self' data:;
|
||||
script-src 'none'; frame-ancestors 'none';
|
||||
base-uri 'none'; form-action 'self'
|
||||
```
|
||||
|
||||
**No JavaScript in the portal at all**, which is what lets `script-src 'none'`
|
||||
be genuinely enforceable — there is nothing to inject.
|
||||
|
||||
**Network exposure is minimal.**
|
||||
The uvicorn backend binds **loopback only** (`IPAddressAllow=localhost` in the
|
||||
unit). The only ingress is Apache. The systemd unit also sets
|
||||
`NoNewPrivileges`, `ProtectSystem=full`, `ProtectHome`, `PrivateTmp`,
|
||||
`ProtectKernelTunables`, `ProtectControlGroups`, `RestrictSUIDSGID`, and writable
|
||||
paths are limited to `/var/lib/vm-transcribe` and `/var/log/asterisk`.
|
||||
|
||||
**Secrets are separated and git-ignored.**
|
||||
`telegram.conf`, `contacts.conf`, `*.db`, `audio/` are in `.gitignore`. The
|
||||
repo ships `*.example` templates only. Both secret files are mode `640 root:asterisk`.
|
||||
|
||||
**Fail-safe preserves mail.**
|
||||
A pipeline exception relays the original Asterisk message unchanged — we never
|
||||
lose a notification to a bug we introduced.
|
||||
|
||||
---
|
||||
|
||||
## Residual risks and how to close them
|
||||
|
||||
**(R1) 4-digit PINs.** Lockout helps but a distributed attack from many IPs
|
||||
defeats per-IP throttling. If the portal is on a hostile network, put it behind
|
||||
your VPN (there is already a `t01.vpn.conf` on this host) or fail2ban the Apache
|
||||
access log. Recommended.
|
||||
|
||||
**(R2) No TLS on the loopback hop.** Apache→uvicorn is plain HTTP on localhost.
|
||||
Acceptable (same host, no network path), but if you ever run uvicorn on another
|
||||
host, use a unix socket or mTLS.
|
||||
|
||||
**(R3) Session token in SQLite.** If the DB file is stolen, sessions are
|
||||
replayable until they expire. DB is `640 asterisk:asterisk` and not web-served.
|
||||
For higher assurance, store sessions in a server-side cache with shorter TTLs.
|
||||
|
||||
**(R4) Contact resolver tokens.** A Google OAuth token or a CardDAV app password
|
||||
grants read access to your address book. Scope the Google token to
|
||||
`contacts.readonly`, use a dedicated app password (never your login password),
|
||||
and keep these files `640`. Prefer the `file` backend (a periodic vCard export) —
|
||||
no live token at all.
|
||||
|
||||
**(R5) Telegram bot token.** Whoever holds it can post as your bot. Keep
|
||||
`telegram.conf` `640 root:asterisk`; rotate via @BotFather if leaked.
|
||||
|
||||
**(R6) Content-addressed audio filenames.** The sha256 of the audio is in the URL
|
||||
(`/audio/<sha>`). An attacker who guesses a sha could fetch that recording
|
||||
without a session. The portal checks the session's mailbox *owns* that message
|
||||
before serving, so this is not directly exploitable — but consider a random
|
||||
per-message token instead of the content hash if you want defence in depth.
|
||||
|
||||
**(R7) CDN / edge.** If `vm.txt3.net` is orange-clouded at Cloudflare, recording
|
||||
audio streams through the edge. Grey-cloud it (DNS only) to keep voice data off
|
||||
the CDN. Your call.
|
||||
|
||||
---
|
||||
|
||||
## Obtaining a Google token (if you use the `google` backend)
|
||||
|
||||
1. Google Cloud console → OAuth consent screen (External) → add your account as
|
||||
a test user.
|
||||
2. Credentials → OAuth client ID → **Desktop app**.
|
||||
3. Scope `https://www.googleapis.com/auth/contacts.readonly`.
|
||||
4. Authorize once (the token is written to `google-token.json`, mode `640`).
|
||||
The People API is used, not CardDAV, because basic auth / app passwords were
|
||||
disabled by Google on **2024-09-30**.
|
||||
|
||||
There is no support for an app-password read of Google Contacts — it does not
|
||||
work. Use the `file` or `google` backend.
|
||||
159
docs/TESTING.md
Normal file
159
docs/TESTING.md
Normal file
@ -0,0 +1,159 @@
|
||||
# Testing
|
||||
|
||||
How to verify each part of the system. Do the mailcmd tests **as the `asterisk`
|
||||
user**, not as yourself — that's where permission bugs hide.
|
||||
|
||||
---
|
||||
|
||||
## 0. Pre-checks
|
||||
|
||||
```bash
|
||||
df -h /var # must not be full (Postfix 452)
|
||||
grep '^mailcmd' /etc/asterisk/voicemail.conf
|
||||
sudo systemctl is-active vm-portal
|
||||
curl -s http://127.0.0.1:8099/healthz
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Unit tests (no sudo, no asterisk)
|
||||
|
||||
```bash
|
||||
cd /home/jp/Work/asterisk-voicemail
|
||||
./venv/bin/python tests/test_telegram.py # routing, caption clipping, opus transcode
|
||||
./venv/bin/python tests/test_contacts.py # vCard/CSV parse, digit-normalised match
|
||||
```
|
||||
|
||||
`test_telegram.py` asserts: per-mailbox → one chat; comma list → N chats;
|
||||
group id negative; unmapped → default; `enabled=no` → no route; caption clipped
|
||||
to 1017 chars on a word boundary when over 1024; a 374 KB wav → 64 KB `OggS`
|
||||
opus.
|
||||
|
||||
`test_contacts.py` asserts: `+447700900123`, `07700900123`, `447700900123` all
|
||||
match one contact (last-9-digit key); multi-TEL cards; Google CSV `:::` split.
|
||||
|
||||
---
|
||||
|
||||
## 2. Build a fake voicemail and run the real pipeline
|
||||
|
||||
```bash
|
||||
# needs ffmpeg; uses test_vm.wav shipped in the repo (or any wav)
|
||||
python3 tests/make_test_mail.py test_vm.wav you@example.com > /tmp/t.eml
|
||||
sudo chmod 644 /tmp/t.eml
|
||||
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||
/opt/vm-transcribe/vm_mailcmd.py < /tmp/t.eml
|
||||
sudo tail -5 /var/log/asterisk/vm_mailcmd.log
|
||||
```
|
||||
|
||||
Expect `transcribed … chars` then `sent enriched notification to you@example.com`.
|
||||
Then confirm real delivery:
|
||||
|
||||
```bash
|
||||
sudo grep "to=<you@example.com>" /var/log/mail.log | tail -1 # status=sent
|
||||
```
|
||||
|
||||
### Fail-safe: a message with no audio
|
||||
|
||||
```bash
|
||||
printf 'From: a@b\nTo: you@example.com\nSubject: no audio\n\nbody\n' > /tmp/n.eml
|
||||
sudo chmod 644 /tmp/n.eml
|
||||
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||
/opt/vm-transcribe/vm_mailcmd.py < /tmp/n.eml
|
||||
# log must show: "no audio attachment; relaying original"
|
||||
```
|
||||
|
||||
### Fail-safe: a thrown error relays the original
|
||||
|
||||
Temporarily point `VM_DB` at an unreadable path; the script should fall back to
|
||||
relaying the original Asterisk message and log `relaying original, mailcmd error`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Telegram (needs a real token)
|
||||
|
||||
```bash
|
||||
sudo /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py ids
|
||||
# each recipient must have messaged the bot once (/start) to appear
|
||||
sudo /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py test 1001
|
||||
# sends a real voice note + summary; exits non-zero on any failure
|
||||
```
|
||||
|
||||
Offline, the routing logic is covered by `tests/test_telegram.py` so you don't
|
||||
need a token to verify the config parser.
|
||||
|
||||
---
|
||||
|
||||
## 4. Contacts
|
||||
|
||||
```bash
|
||||
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||
/opt/vm-transcribe/vm_contacts.py '<07700900123>'
|
||||
# resolves against the configured backend; prints name or the raw caller id
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Portal — auth, authz, playback, delete
|
||||
|
||||
Run the app locally (or against the live service) and exercise it with curl.
|
||||
Use `VM_INSECURE_COOKIE=1` only for plain-HTTP local tests so the `Secure`
|
||||
cookie can be set.
|
||||
|
||||
```bash
|
||||
# login ok
|
||||
curl -s -c /tmp/j -o /dev/null -w '%{redirect_url}\n' \
|
||||
-d 'mailbox=7940&pin=5159' http://127.0.0.1:8099/login
|
||||
|
||||
# wrong pin rejected
|
||||
curl -s -o /dev/null -w '%{http_code}\n' -d 'mailbox=7940&pin=1111' http://127.0.0.1:8099/login
|
||||
|
||||
# authenticated list
|
||||
curl -s -b /tmp/j http://127.0.0.1:8099/ | grep -c 'class="card'
|
||||
|
||||
# audio streams
|
||||
curl -s -b /tmp/j -o /tmp/x.wav http://127.0.0.1:8099/audio/1
|
||||
file /tmp/x.wav # RIFF WAVE, 8000 Hz mono
|
||||
|
||||
# delete
|
||||
curl -s -b /tmp/j -X POST -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8099/delete/1
|
||||
```
|
||||
|
||||
### Authorization (IDOR) — must 404
|
||||
|
||||
A session for mailbox A must never reach mailbox B's data. With a cookie for
|
||||
mailbox 7940, hitting `/audio/<id-owned-by-1001>` and `/delete/<id-owned-by-1001>`
|
||||
must both return **404/303**, never serve or delete the other mailbox's message.
|
||||
Add this assertion whenever you change `vm_store` or `vm_web`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Portal over the real URL (TLS + proxy)
|
||||
|
||||
```bash
|
||||
R="--resolve vm.txt3.net:443:ORIGIN.IP"
|
||||
curl -s $R https://vm.txt3.net/healthz
|
||||
curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.txt3.net/login
|
||||
curl -s $R -D- -o /dev/null https://vm.txt3.net/login | grep -iE 'strict-transport|content-security|x-frame'
|
||||
# http -> https redirect
|
||||
curl -s $R -o /dev/null -w '%{redirect_url}\n' http://vm.txt3.net/
|
||||
```
|
||||
|
||||
Behind a CDN, `--resolve` to the **origin** IP; otherwise you are testing the CDN,
|
||||
not your server.
|
||||
|
||||
---
|
||||
|
||||
## 7. Backfill
|
||||
|
||||
```bash
|
||||
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||
/opt/vm-transcribe/vm_import.py --dry-run --limit 5
|
||||
# preview: lists 5 messages with caller + date, no writes
|
||||
|
||||
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
|
||||
/opt/vm-transcribe/vm_import.py --limit 3
|
||||
# real: 3 imported, others skipped; re-running is a no-op (idempotent)
|
||||
```
|
||||
|
||||
Verify they appear in the portal and are playable (§5). Expect a `no_speech`
|
||||
count — 44-byte WAVs are hung-up calls, not failures.
|
||||
62
scripts/install.sh
Executable file
62
scripts/install.sh
Executable file
@ -0,0 +1,62 @@
|
||||
#!/bin/bash
|
||||
# Installs the voicemail transcription mailcmd. Run with sudo.
|
||||
set -euo pipefail
|
||||
|
||||
SRC=/home/jp/asterisk-vm
|
||||
DEST=/opt/vm-transcribe
|
||||
|
||||
echo "== creating $DEST"
|
||||
install -d -o asterisk -g asterisk -m 755 "$DEST" "$DEST/models"
|
||||
|
||||
echo "== python venv + faster-whisper"
|
||||
if [ ! -x "$DEST/venv/bin/python3" ]; then
|
||||
python3 -m venv "$DEST/venv"
|
||||
"$DEST/venv/bin/pip" install -q --upgrade pip
|
||||
"$DEST/venv/bin/pip" install -q faster-whisper
|
||||
fi
|
||||
|
||||
echo "== installing script"
|
||||
install -o root -g root -m 755 "$SRC/vm_mailcmd.py" "$DEST/vm_mailcmd.py"
|
||||
install -o root -g root -m 755 "$SRC/vm_telegram.py" "$DEST/vm_telegram.py"
|
||||
install -o root -g root -m 755 "$SRC/vm_tg_setup.py" "$DEST/vm_tg_setup.py"
|
||||
chown -R asterisk:asterisk "$DEST/models"
|
||||
|
||||
echo "== telegram config (not overwritten if it already exists)"
|
||||
if [ ! -f "$DEST/telegram.conf" ]; then
|
||||
install -o root -g asterisk -m 640 "$SRC/telegram.conf" "$DEST/telegram.conf"
|
||||
echo " created $DEST/telegram.conf - add your bot token and chat IDs"
|
||||
else
|
||||
echo " kept existing $DEST/telegram.conf"
|
||||
fi
|
||||
|
||||
echo "== log file"
|
||||
touch /var/log/asterisk/vm_mailcmd.log
|
||||
chown asterisk:asterisk /var/log/asterisk/vm_mailcmd.log
|
||||
chmod 640 /var/log/asterisk/vm_mailcmd.log
|
||||
|
||||
echo "== pre-downloading the whisper model as the asterisk user"
|
||||
sudo -u asterisk "$DEST/venv/bin/python3" - <<EOF
|
||||
from faster_whisper import WhisperModel
|
||||
WhisperModel("base.en", device="cpu", compute_type="int8", download_root="$DEST/models")
|
||||
print("model cached")
|
||||
EOF
|
||||
|
||||
echo "== patching /etc/asterisk/voicemail.conf"
|
||||
CONF=/etc/asterisk/voicemail.conf
|
||||
cp -a "$CONF" "$CONF.bak-$(date +%Y%m%d%H%M%S)"
|
||||
python3 - <<EOF
|
||||
import re
|
||||
p="$CONF"
|
||||
s=open(p).read()
|
||||
old="mailcmd=sendmail -t"
|
||||
new="mailcmd=$DEST/venv/bin/python3 $DEST/vm_mailcmd.py"
|
||||
assert s.count(old)==1, "anchor not found exactly once: %d" % s.count(old)
|
||||
s=s.replace(old,new)
|
||||
open(p,"w").write(s)
|
||||
print("mailcmd set to:",new)
|
||||
EOF
|
||||
|
||||
echo "== reloading asterisk voicemail"
|
||||
asterisk -rx "voicemail reload" || asterisk -rx "module reload app_voicemail"
|
||||
|
||||
echo "DONE. Watch: tail -f /var/log/asterisk/vm_mailcmd.log"
|
||||
67
src/vm_auth.py
Normal file
67
src/vm_auth.py
Normal file
@ -0,0 +1,67 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Voicemail portal - parses Asterisk voicemail.conf for mailbox auth.
|
||||
|
||||
voicemail.conf mailbox lines look like:
|
||||
7940 => 5159,Jamie,jp@txt3.com ; 01273 805515
|
||||
mailbox => password,name,email,pager,options
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
|
||||
VM_CONF = os.environ.get("VM_ASTERISK_CONF", "/etc/asterisk/voicemail.conf")
|
||||
|
||||
# Sections that are not mailbox contexts
|
||||
_NON_CONTEXT = {"general", "zonemessages"}
|
||||
|
||||
|
||||
def parse_mailboxes(path=None):
|
||||
"""Return {mailbox: {"pin","name","email","context"}}."""
|
||||
p = path or VM_CONF
|
||||
out = {}
|
||||
context = "default"
|
||||
try:
|
||||
with open(p, encoding="utf-8", errors="replace") as fh:
|
||||
for raw in fh:
|
||||
line = raw.strip()
|
||||
if not line or line.startswith(";") or line.startswith("#"):
|
||||
continue
|
||||
m = re.match(r"^\[([^\]]+)\]", line)
|
||||
if m:
|
||||
context = m.group(1).strip()
|
||||
continue
|
||||
if context in _NON_CONTEXT:
|
||||
continue
|
||||
m = re.match(r"^(\d+)\s*=>\s*(.+)$", line)
|
||||
if not m:
|
||||
continue
|
||||
mbox, rest = m.group(1), m.group(2)
|
||||
rest = rest.split(";", 1)[0].strip() # strip trailing comment
|
||||
parts = [x.strip() for x in rest.split(",")]
|
||||
pin = parts[0] if parts else ""
|
||||
name = parts[1] if len(parts) > 1 else mbox
|
||||
mail = parts[2] if len(parts) > 2 else ""
|
||||
out[mbox] = {"pin": pin, "name": name, "email": mail,
|
||||
"context": context}
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
return out
|
||||
|
||||
|
||||
def check_login(mailbox, pin, path=None):
|
||||
"""Constant-time-ish PIN check. Returns the mailbox dict or None."""
|
||||
import hmac
|
||||
boxes = parse_mailboxes(path)
|
||||
info = boxes.get(str(mailbox).strip())
|
||||
if not info or not info.get("pin"):
|
||||
return None
|
||||
if hmac.compare_digest(str(info["pin"]), str(pin).strip()):
|
||||
return info
|
||||
return None
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
for mb, i in parse_mailboxes().items():
|
||||
print("%-8s %-18s %-28s ctx=%s pin=%s"
|
||||
% (mb, i["name"], i["email"], i["context"], "*" * len(i["pin"])))
|
||||
334
src/vm_contacts.py
Normal file
334
src/vm_contacts.py
Normal file
@ -0,0 +1,334 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Caller-ID -> contact name resolution for Asterisk voicemail notifications.
|
||||
|
||||
Backends (tried in the order given by contacts.conf 'backends='):
|
||||
file - local vCard (.vcf) or CSV export; no auth, offline, fast.
|
||||
google - Google People API via the Hermes OAuth token.
|
||||
carddav - generic CardDAV with username + app password.
|
||||
|
||||
Everything here is best-effort: a lookup failure returns None and the caller
|
||||
falls back to the raw caller-ID string. Results (hits AND misses) are cached
|
||||
to avoid per-call API traffic.
|
||||
|
||||
NOTE: app-password / basic auth does NOT work against Google - Google disabled
|
||||
it for CardDAV/CalDAV/IMAP on 2024-09-30 and now requires OAuth for CardDAV.
|
||||
The carddav backend is for Nextcloud / Fastmail / iCloud / Radicale etc.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import configparser
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
CONF_PATH = os.environ.get("VM_CONTACTS_CONF", "/opt/vm-transcribe/contacts.conf")
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- helpers
|
||||
def digits_of(s):
|
||||
return re.sub(r"\D", "", s or "")
|
||||
|
||||
|
||||
def extract_number(caller):
|
||||
"""Pull a dialable number out of a CallerID string like 'Dave <079...>'."""
|
||||
if not caller:
|
||||
return ""
|
||||
m = re.search(r"<([^>]+)>", caller)
|
||||
cand = m.group(1) if m else caller
|
||||
return digits_of(cand)
|
||||
|
||||
|
||||
def _key(num_digits, n):
|
||||
d = digits_of(num_digits)
|
||||
return d[-n:] if len(d) >= n else d
|
||||
|
||||
|
||||
class Cache:
|
||||
def __init__(self, path, ttl):
|
||||
self.path, self.ttl, self.data = path, ttl, {}
|
||||
try:
|
||||
with open(path) as fh:
|
||||
self.data = json.load(fh)
|
||||
except Exception:
|
||||
self.data = {}
|
||||
|
||||
def get(self, key):
|
||||
e = self.data.get(key)
|
||||
if not e:
|
||||
return None # unknown -> caller should look up
|
||||
if self.ttl and time.time() - e.get("t", 0) > self.ttl:
|
||||
return None
|
||||
return e # {"t":..., "name": <str or None>}
|
||||
|
||||
def put(self, key, name):
|
||||
self.data[key] = {"t": time.time(), "name": name}
|
||||
try:
|
||||
os.makedirs(os.path.dirname(self.path), exist_ok=True)
|
||||
tmp = self.path + ".tmp"
|
||||
with open(tmp, "w") as fh:
|
||||
json.dump(self.data, fh)
|
||||
os.replace(tmp, self.path)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
# ------------------------------------------------------------- file backend
|
||||
def _parse_vcf(text):
|
||||
"""Return list of (name, [numbers])."""
|
||||
out = []
|
||||
name, nums = None, []
|
||||
for raw in text.splitlines():
|
||||
line = raw.strip()
|
||||
u = line.upper()
|
||||
if u == "BEGIN:VCARD":
|
||||
name, nums = None, []
|
||||
elif u.startswith("FN"):
|
||||
name = line.split(":", 1)[1].strip() if ":" in line else None
|
||||
elif u.startswith("TEL"):
|
||||
if ":" in line:
|
||||
nums.append(line.split(":", 1)[1].strip())
|
||||
elif u == "END:VCARD":
|
||||
if name and nums:
|
||||
out.append((name, nums))
|
||||
return out
|
||||
|
||||
|
||||
def _parse_csv(path):
|
||||
import csv
|
||||
out = []
|
||||
with open(path, newline="", encoding="utf-8", errors="replace") as fh:
|
||||
r = csv.DictReader(fh)
|
||||
cols = r.fieldnames or []
|
||||
name_cols = [c for c in cols if c and ("Name" == c or c.endswith("Name"))]
|
||||
phone_cols = [c for c in cols if c and "Phone" in c and "Value" in c] or \
|
||||
[c for c in cols if c and "Phone" in c]
|
||||
for row in r:
|
||||
name = ""
|
||||
if "Name" in row and row["Name"]:
|
||||
name = row["Name"]
|
||||
else:
|
||||
parts = [row.get(c, "") for c in ("Given Name", "Family Name") if row.get(c)]
|
||||
name = " ".join(parts) or (row.get(name_cols[0], "") if name_cols else "")
|
||||
nums = []
|
||||
for c in phone_cols:
|
||||
v = row.get(c, "")
|
||||
if v:
|
||||
nums.extend(re.split(r"\s*:::\s*|\s*;\s*", v))
|
||||
if name and nums:
|
||||
out.append((name.strip(), nums))
|
||||
return out
|
||||
|
||||
|
||||
def _index(entries, n):
|
||||
idx = {}
|
||||
for name, nums in entries:
|
||||
for num in nums:
|
||||
k = _key(num, n)
|
||||
if k:
|
||||
idx.setdefault(k, name)
|
||||
return idx
|
||||
|
||||
|
||||
def lookup_file(cfg, num_digits, n, log):
|
||||
path = cfg.get("path", "").strip()
|
||||
if not path or not os.path.exists(path):
|
||||
return None
|
||||
try:
|
||||
if path.lower().endswith(".csv"):
|
||||
entries = _parse_csv(path)
|
||||
else:
|
||||
with open(path, encoding="utf-8", errors="replace") as fh:
|
||||
entries = _parse_vcf(fh.read())
|
||||
return _index(entries, n).get(_key(num_digits, n))
|
||||
except Exception as e:
|
||||
log("contacts file backend error: %s" % e)
|
||||
return None
|
||||
|
||||
|
||||
# ----------------------------------------------------------- google backend
|
||||
def _google_access_token(token_path, log):
|
||||
try:
|
||||
with open(token_path) as fh:
|
||||
tok = json.load(fh)
|
||||
except Exception as e:
|
||||
log("google token unreadable (%s): %s" % (token_path, e))
|
||||
return None
|
||||
# try existing token first; refresh if People API 401s
|
||||
at = tok.get("token") or tok.get("access_token")
|
||||
refresh = tok.get("refresh_token")
|
||||
cid = tok.get("client_id")
|
||||
secret = tok.get("client_secret")
|
||||
if at:
|
||||
return at, (refresh, cid, secret, token_path, tok)
|
||||
return _google_refresh((refresh, cid, secret, token_path, tok), log)
|
||||
|
||||
|
||||
def _google_refresh(ctx, log):
|
||||
refresh, cid, secret, token_path, tok = ctx
|
||||
if not (refresh and cid and secret):
|
||||
log("google token missing refresh_token/client_id/client_secret")
|
||||
return None
|
||||
try:
|
||||
data = urllib.parse.urlencode({
|
||||
"client_id": cid, "client_secret": secret,
|
||||
"refresh_token": refresh, "grant_type": "refresh_token",
|
||||
}).encode()
|
||||
req = urllib.request.Request("https://oauth2.googleapis.com/token", data=data)
|
||||
with urllib.request.urlopen(req, timeout=20) as r:
|
||||
new = json.loads(r.read().decode())
|
||||
at = new.get("access_token")
|
||||
if at:
|
||||
tok["token"] = at
|
||||
try:
|
||||
with open(token_path, "w") as fh:
|
||||
json.dump(tok, fh)
|
||||
except Exception:
|
||||
pass
|
||||
return at, ctx
|
||||
except Exception as e:
|
||||
log("google token refresh failed: %s" % e)
|
||||
return None
|
||||
|
||||
|
||||
def lookup_google(cfg, num_digits, n, log):
|
||||
token_path = cfg.get("token_path", "").strip()
|
||||
if not token_path or not os.path.exists(token_path):
|
||||
return None
|
||||
got = _google_access_token(token_path, log)
|
||||
if not got:
|
||||
return None
|
||||
at, ctx = got
|
||||
want = _key(num_digits, n)
|
||||
|
||||
def query(access):
|
||||
url = ("https://people.googleapis.com/v1/people:searchContacts"
|
||||
"?query=%s&readMask=names,phoneNumbers"
|
||||
% urllib.parse.quote(num_digits[-7:] or num_digits))
|
||||
req = urllib.request.Request(url, headers={"Authorization": "Bearer %s" % access})
|
||||
with urllib.request.urlopen(req, timeout=20) as r:
|
||||
return json.loads(r.read().decode())
|
||||
|
||||
try:
|
||||
try:
|
||||
d = query(at)
|
||||
except urllib.error.HTTPError as he:
|
||||
if he.code == 401: # refresh once
|
||||
got = _google_refresh(ctx if isinstance(ctx, tuple) and len(ctx) == 5
|
||||
else ctx, log)
|
||||
if not got:
|
||||
return None
|
||||
d = query(got[0])
|
||||
else:
|
||||
raise
|
||||
for res in d.get("results", []):
|
||||
person = res.get("person", {})
|
||||
for ph in person.get("phoneNumbers", []):
|
||||
if _key(ph.get("value", ""), n) == want:
|
||||
names = person.get("names", [])
|
||||
if names:
|
||||
return names[0].get("displayName")
|
||||
except Exception as e:
|
||||
log("google People API error: %s" % e)
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------- carddav backend
|
||||
def lookup_carddav(cfg, num_digits, n, log):
|
||||
import ssl
|
||||
url = cfg.get("url", "").strip()
|
||||
user = cfg.get("username", "").strip()
|
||||
pw = cfg.get("app_password", "").strip()
|
||||
if not (url and user and pw):
|
||||
return None
|
||||
if "google.com" in url:
|
||||
log("carddav backend points at Google, which rejects app passwords "
|
||||
"since 2024-09-30; use the 'google' or 'file' backend instead")
|
||||
return None
|
||||
|
||||
path = cfg.get("addressbook_path", "").strip()
|
||||
base = url.rstrip("/") + (path if path.startswith("/") else "/" + path if path else "")
|
||||
ctx = None
|
||||
if cfg.get("verify_tls", "yes").lower() in ("no", "false", "0"):
|
||||
ctx = ssl.create_default_context()
|
||||
ctx.check_hostname = False
|
||||
ctx.verify_mode = ssl.CERT_NONE
|
||||
timeout = int(cfg.get("timeout", "20") or 20)
|
||||
auth = base64.b64encode(("%s:%s" % (user, pw)).encode()).decode()
|
||||
|
||||
body = ('<?xml version="1.0"?>'
|
||||
'<C:addressbook-query xmlns:D="DAV:" '
|
||||
'xmlns:C="urn:ietf:params:xml:ns:carddav">'
|
||||
'<D:prop><C:address-data/></D:prop></C:addressbook-query>')
|
||||
try:
|
||||
req = urllib.request.Request(
|
||||
base, data=body.encode(), method="REPORT",
|
||||
headers={"Authorization": "Basic %s" % auth, "Depth": "1",
|
||||
"Content-Type": "application/xml; charset=utf-8"})
|
||||
with urllib.request.urlopen(req, timeout=timeout, context=ctx) as r:
|
||||
xml = r.read().decode("utf-8", "replace")
|
||||
vcards = re.findall(r"BEGIN:VCARD.*?END:VCARD", xml, re.S | re.I)
|
||||
entries = []
|
||||
for v in vcards:
|
||||
entries.extend(_parse_vcf(v.replace(" ", "").replace("\r", "")))
|
||||
return _index(entries, n).get(_key(num_digits, n))
|
||||
except Exception as e:
|
||||
log("carddav backend error: %s" % e)
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- entrypoint
|
||||
_BACKENDS = {"file": lookup_file, "google": lookup_google, "carddav": lookup_carddav}
|
||||
|
||||
|
||||
def resolve(caller_id, log=print):
|
||||
"""Return a contact name for a CallerID string, or None."""
|
||||
num = extract_number(caller_id)
|
||||
if not num or not os.path.exists(CONF_PATH):
|
||||
return None
|
||||
try:
|
||||
cp = configparser.ConfigParser(inline_comment_prefixes=("#", ";"))
|
||||
cp.read(CONF_PATH)
|
||||
if not cp.has_section("contacts") or not cp["contacts"].getboolean("enabled", False):
|
||||
return None
|
||||
g = cp["contacts"]
|
||||
n = int(g.get("match_digits", "9") or 9)
|
||||
cache = Cache(g.get("cache_path", "/var/lib/vm-transcribe/contacts_cache.json"),
|
||||
int(g.get("cache_ttl", "86400") or 86400))
|
||||
|
||||
key = _key(num, n)
|
||||
cached = cache.get(key)
|
||||
if cached is not None:
|
||||
return cached.get("name")
|
||||
|
||||
order = [b.strip() for b in g.get("backends", "file").split(",") if b.strip()]
|
||||
name = None
|
||||
for b in order:
|
||||
fn = _BACKENDS.get(b)
|
||||
if not fn:
|
||||
log("unknown contacts backend: %s" % b)
|
||||
continue
|
||||
sect = cp[b] if cp.has_section(b) else {}
|
||||
try:
|
||||
name = fn(sect, num, n, log)
|
||||
except Exception as e:
|
||||
log("contacts backend %s crashed: %s" % (b, e))
|
||||
name = None
|
||||
if name:
|
||||
log("contacts: %s -> %s (via %s)" % (num, name, b))
|
||||
break
|
||||
|
||||
cache.put(key, name) # cache misses too (name=None)
|
||||
return name
|
||||
except Exception as e:
|
||||
log("contacts resolve error: %s" % e)
|
||||
return None
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
print(resolve(sys.argv[1] if len(sys.argv) > 1 else "", log=lambda m: print("[log]", m)))
|
||||
229
src/vm_import.py
Normal file
229
src/vm_import.py
Normal file
@ -0,0 +1,229 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Backfill existing Asterisk voicemails into the portal database.
|
||||
|
||||
Walks the voicemail spool, reads each message's .txt metadata, transcribes the
|
||||
recording with faster-whisper, summarises it, resolves the caller against
|
||||
contacts, and inserts it into the same SQLite store the live mailcmd writes to.
|
||||
|
||||
Safe to re-run: rows are unique on (mailbox, origtime, callerid), so an
|
||||
interrupted run resumes rather than duplicating. Already-imported messages are
|
||||
skipped without being transcribed again (the expensive part).
|
||||
|
||||
Usage (run as the asterisk user so it can read the spool):
|
||||
vm_import.py --dry-run # show what would be done
|
||||
vm_import.py # import everything
|
||||
vm_import.py --mailbox 7940 # one mailbox
|
||||
vm_import.py --folder INBOX # one folder
|
||||
vm_import.py --limit 10 # first N (useful for a trial run)
|
||||
vm_import.py --no-transcribe # metadata + audio only, no whisper
|
||||
vm_import.py --reverse # newest first
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import configparser
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
SPOOL = os.environ.get("VM_SPOOL", "/var/spool/asterisk/voicemail")
|
||||
AUDIO_EXTS = (".wav", ".WAV", ".gsm", ".wav49", ".ogg", ".mp3")
|
||||
|
||||
|
||||
def log(msg):
|
||||
print(msg, flush=True)
|
||||
|
||||
|
||||
def parse_info(path):
|
||||
"""Parse an Asterisk msgNNNN.txt message information file."""
|
||||
out = {}
|
||||
try:
|
||||
cp = configparser.ConfigParser(strict=False, inline_comment_prefixes=None)
|
||||
with open(path, encoding="utf-8", errors="replace") as fh:
|
||||
text = fh.read()
|
||||
cp.read_string(text)
|
||||
if cp.has_section("message"):
|
||||
out = dict(cp["message"])
|
||||
except Exception:
|
||||
# fall back to a naive key=value scrape
|
||||
try:
|
||||
with open(path, encoding="utf-8", errors="replace") as fh:
|
||||
for line in fh:
|
||||
if "=" in line and not line.strip().startswith(";"):
|
||||
k, v = line.split("=", 1)
|
||||
out[k.strip()] = v.strip()
|
||||
except Exception:
|
||||
pass
|
||||
return out
|
||||
|
||||
|
||||
def find_messages(spool, want_mailbox=None, want_folder=None):
|
||||
"""Yield (context, mailbox, folder, basepath, audio_path, info_path)."""
|
||||
if not os.path.isdir(spool):
|
||||
return
|
||||
for context in sorted(os.listdir(spool)):
|
||||
cdir = os.path.join(spool, context)
|
||||
if not os.path.isdir(cdir):
|
||||
continue
|
||||
for mailbox in sorted(os.listdir(cdir)):
|
||||
if want_mailbox and mailbox != want_mailbox:
|
||||
continue
|
||||
mdir = os.path.join(cdir, mailbox)
|
||||
if not os.path.isdir(mdir):
|
||||
continue
|
||||
for folder in sorted(os.listdir(mdir)):
|
||||
if folder in ("tmp",):
|
||||
continue
|
||||
if want_folder and folder != want_folder:
|
||||
continue
|
||||
fdir = os.path.join(mdir, folder)
|
||||
if not os.path.isdir(fdir):
|
||||
continue
|
||||
# group by msgNNNN stem
|
||||
stems = set()
|
||||
for fn in os.listdir(fdir):
|
||||
m = re.match(r"^(msg\d+)\.", fn)
|
||||
if m:
|
||||
stems.add(m.group(1))
|
||||
for stem in sorted(stems):
|
||||
base = os.path.join(fdir, stem)
|
||||
audio = None
|
||||
for ext in AUDIO_EXTS:
|
||||
if os.path.exists(base + ext):
|
||||
audio = base + ext
|
||||
break
|
||||
info = base + ".txt"
|
||||
if audio:
|
||||
yield (context, mailbox, folder, base, audio,
|
||||
info if os.path.exists(info) else None)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--spool", default=SPOOL)
|
||||
ap.add_argument("--mailbox")
|
||||
ap.add_argument("--folder")
|
||||
ap.add_argument("--limit", type=int)
|
||||
ap.add_argument("--dry-run", action="store_true")
|
||||
ap.add_argument("--no-transcribe", action="store_true")
|
||||
ap.add_argument("--reverse", action="store_true")
|
||||
ap.add_argument("--model", default=os.environ.get("VM_WHISPER_MODEL", "base.en"))
|
||||
args = ap.parse_args()
|
||||
|
||||
import vm_store
|
||||
con = vm_store.connect()
|
||||
|
||||
items = list(find_messages(args.spool, args.mailbox, args.folder))
|
||||
if args.reverse:
|
||||
items.reverse()
|
||||
log("found %d message(s) in %s" % (len(items), args.spool))
|
||||
if not items:
|
||||
return 0
|
||||
|
||||
# Which are already imported? Compare on (mailbox, origtime, callerid).
|
||||
existing = set()
|
||||
for r in con.execute("SELECT mailbox, origtime, callerid FROM messages"):
|
||||
existing.add((str(r["mailbox"]), r["origtime"], r["callerid"]))
|
||||
|
||||
# Load whisper once, lazily - it is the slow part.
|
||||
model = None
|
||||
|
||||
def transcribe(path):
|
||||
nonlocal model
|
||||
if model is None:
|
||||
from faster_whisper import WhisperModel
|
||||
log(" loading whisper model %s (cpu/int8)..." % args.model)
|
||||
model = WhisperModel(
|
||||
args.model, device="cpu", compute_type="int8",
|
||||
cpu_threads=max(1, (os.cpu_count() or 2) - 1),
|
||||
download_root=os.environ.get("VM_MODEL_CACHE",
|
||||
"/opt/vm-transcribe/models"))
|
||||
segs, _info = model.transcribe(
|
||||
path, beam_size=1, vad_filter=True,
|
||||
vad_parameters=dict(min_silence_duration_ms=500),
|
||||
condition_on_previous_text=False)
|
||||
return re.sub(r"\s+", " ", " ".join(s.text.strip() for s in segs)).strip()
|
||||
|
||||
# Reuse the live pipeline's summariser / tagger / contact lookup so
|
||||
# backfilled messages look identical to new ones.
|
||||
import vm_mailcmd
|
||||
try:
|
||||
import vm_contacts
|
||||
except Exception:
|
||||
vm_contacts = None
|
||||
|
||||
stats = {"imported": 0, "skipped": 0, "failed": 0, "no_speech": 0}
|
||||
t0 = time.time()
|
||||
todo = items[:args.limit] if args.limit else items
|
||||
|
||||
for i, (context, mailbox, folder, base, audio, info) in enumerate(todo, 1):
|
||||
meta = parse_info(info) if info else {}
|
||||
callerid = meta.get("callerid") or None
|
||||
try:
|
||||
origtime = int(meta.get("origtime") or 0) or int(os.path.getmtime(audio))
|
||||
except Exception:
|
||||
origtime = int(os.path.getmtime(audio))
|
||||
try:
|
||||
duration = int(float(meta.get("duration") or 0)) or None
|
||||
except Exception:
|
||||
duration = None
|
||||
|
||||
tag = "%s/%s/%s" % (mailbox, folder, os.path.basename(base))
|
||||
if (str(mailbox), origtime, callerid) in existing:
|
||||
stats["skipped"] += 1
|
||||
continue
|
||||
|
||||
if args.dry_run:
|
||||
log(" [%d/%d] WOULD IMPORT %-28s caller=%-22s %s"
|
||||
% (i, len(todo), tag, callerid or "?",
|
||||
time.strftime("%Y-%m-%d %H:%M", time.localtime(origtime))))
|
||||
stats["imported"] += 1
|
||||
continue
|
||||
|
||||
try:
|
||||
transcript = "" if args.no_transcribe else transcribe(audio)
|
||||
if not transcript:
|
||||
stats["no_speech"] += 1
|
||||
summary = vm_mailcmd.summarise(transcript) if transcript else ""
|
||||
intents = vm_mailcmd.detect_intents(transcript) if transcript else []
|
||||
numbers = vm_mailcmd.find_numbers(transcript) if transcript else []
|
||||
|
||||
contact = None
|
||||
if vm_contacts and callerid:
|
||||
try:
|
||||
contact = vm_contacts.resolve(callerid, log=lambda m: None)
|
||||
except Exception:
|
||||
contact = None
|
||||
|
||||
with open(audio, "rb") as fh:
|
||||
audio_bytes = fh.read()
|
||||
|
||||
vm_store.add_message(
|
||||
con, mailbox, callerid=callerid, contact_name=contact,
|
||||
origtime=origtime, duration=duration, transcript=transcript,
|
||||
summary=summary, intents=intents, numbers=numbers,
|
||||
audio_bytes=audio_bytes,
|
||||
audio_ext=os.path.splitext(audio)[1].lstrip(".").lower() or "wav",
|
||||
spool_path=audio, context=context, folder=folder)
|
||||
existing.add((str(mailbox), origtime, callerid))
|
||||
stats["imported"] += 1
|
||||
log(" [%d/%d] %-28s %-22s %3ds %s"
|
||||
% (i, len(todo), tag, (callerid or "?")[:22], duration or 0,
|
||||
(summary[:60] + "...") if len(summary) > 60 else summary))
|
||||
except Exception as e:
|
||||
stats["failed"] += 1
|
||||
log(" [%d/%d] FAILED %s: %s" % (i, len(todo), tag, e))
|
||||
|
||||
con.close()
|
||||
el = time.time() - t0
|
||||
log("\ndone in %dm%02ds - imported=%d skipped=%d no_speech=%d failed=%d"
|
||||
% (el // 60, el % 60, stats["imported"], stats["skipped"],
|
||||
stats["no_speech"], stats["failed"]))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
533
src/vm_mailcmd.py
Executable file
533
src/vm_mailcmd.py
Executable file
@ -0,0 +1,533 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Asterisk voicemail mailcmd replacement.
|
||||
|
||||
Reads the RFC822 message Asterisk pipes to `sendmail -t` on stdin, then:
|
||||
a) transcribes the attached voicemail audio with faster-whisper (CPU, int8)
|
||||
b) produces a short extractive summary + callback-number / intent hints
|
||||
c) rebuilds the message as multipart/mixed containing a
|
||||
multipart/alternative (text/plain + styled text/html) and the original
|
||||
audio attachment, and delivers it via SMTP on localhost:25.
|
||||
|
||||
Fail-safe: any error at all -> the ORIGINAL message is relayed unchanged, so a
|
||||
voicemail notification is never lost. All diagnostics go to LOG_PATH.
|
||||
"""
|
||||
|
||||
import email
|
||||
import email.policy
|
||||
import email.utils
|
||||
import os
|
||||
import re
|
||||
import smtplib
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import traceback
|
||||
from datetime import datetime
|
||||
from email.message import EmailMessage
|
||||
from html import escape
|
||||
|
||||
# ------------------------------------------------------------------ settings
|
||||
VENV_PY = "/opt/vm-transcribe/venv/bin/python3"
|
||||
MODEL_SIZE = os.environ.get("VM_WHISPER_MODEL", "base.en")
|
||||
MODEL_CACHE = os.environ.get("VM_MODEL_CACHE", "/opt/vm-transcribe/models")
|
||||
SMTP_HOST = "localhost"
|
||||
SMTP_PORT = 25
|
||||
LOG_PATH = os.environ.get("VM_LOG", "/var/log/asterisk/vm_mailcmd.log")
|
||||
MAX_SECONDS = 900 # ignore absurdly long recordings
|
||||
BRAND = "Voicemail"
|
||||
ACCENT = "#2f6fed"
|
||||
|
||||
|
||||
def log(msg):
|
||||
try:
|
||||
with open(LOG_PATH, "a") as fh:
|
||||
fh.write("%s %s\n" % (datetime.now().isoformat(timespec="seconds"), msg))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
# ------------------------------------------------------------- transcription
|
||||
def transcribe(wav_path):
|
||||
"""Return transcript text (may be '')."""
|
||||
from faster_whisper import WhisperModel
|
||||
|
||||
model = WhisperModel(
|
||||
MODEL_SIZE,
|
||||
device="cpu",
|
||||
compute_type="int8",
|
||||
cpu_threads=max(1, (os.cpu_count() or 2) - 1),
|
||||
download_root=MODEL_CACHE,
|
||||
)
|
||||
segments, info = model.transcribe(
|
||||
wav_path,
|
||||
beam_size=1,
|
||||
vad_filter=True,
|
||||
vad_parameters=dict(min_silence_duration_ms=500),
|
||||
condition_on_previous_text=False,
|
||||
)
|
||||
parts = []
|
||||
for seg in segments:
|
||||
if seg.start > MAX_SECONDS:
|
||||
break
|
||||
parts.append(seg.text.strip())
|
||||
text = " ".join(p for p in parts if p).strip()
|
||||
text = re.sub(r"\s+", " ", text)
|
||||
log("transcribed %.1fs audio, lang=%s, %d chars"
|
||||
% (getattr(info, "duration", 0.0), getattr(info, "language", "?"), len(text)))
|
||||
return text
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- summarise
|
||||
_STOP = set("""a an and are as at be been but by for from had has have he her his i if in into is it
|
||||
its me my not of on or our she so than that the their them then there these they this to was we were
|
||||
what when which who will with would you your um uh yeah okay ok just like know really got get""".split())
|
||||
|
||||
_PHONE_RE = re.compile(
|
||||
r"(?:(?:\+|00)\d{1,3}[ .\-]?)?(?:\(?\d{2,5}\)?[ .\-]?){2,5}\d{2,4}")
|
||||
_SPOKEN_DIGITS = {
|
||||
"zero": "0", "oh": "0", "one": "1", "two": "2", "three": "3", "four": "4",
|
||||
"five": "5", "six": "6", "seven": "7", "eight": "8", "nine": "9",
|
||||
"double": "", "triple": "",
|
||||
}
|
||||
|
||||
_INTENTS = [
|
||||
("Call back requested", r"\b(call (me|us|him|her|them)? ?back|give (me|us) a (call|ring|bell)|"
|
||||
r"ring (me|us) back|get back to (me|us)|reach (me|us))\b"),
|
||||
("Urgent", r"\b(urgent|asap|as soon as possible|emergency|immediately|straight away|"
|
||||
r"right away|critical)\b"),
|
||||
("Appointment", r"\b(appointment|meeting|schedule|reschedule|booking|book(ed)? (you )?in|"
|
||||
r"confirm(ing)? (the|your)? ?(time|date|slot))\b"),
|
||||
("Payment / invoice", r"\b(invoice|payment|pay(ing|ment)? ?(due|late)?|billing|account balance|"
|
||||
r"overdue|quote|estimate)\b"),
|
||||
("Delivery", r"\b(deliver(y|ies)?|parcel|package|courier|dispatch|shipment)\b"),
|
||||
("Complaint / issue", r"\b(complaint|complain|problem|issue|not working|broken|fault|unhappy|"
|
||||
r"disappointed)\b"),
|
||||
("Cancellation", r"\b(cancel(l(ed|ing))?|can't make it|cannot make it|postpone)\b"),
|
||||
("Sales / marketing", r"\b(special offer|promotion|no obligation|free quote|marketing|"
|
||||
r"we noticed your website|SEO)\b"),
|
||||
]
|
||||
|
||||
|
||||
def _split_sentences(text):
|
||||
parts = re.split(r"(?<=[.!?])\s+", text)
|
||||
out = []
|
||||
for p in parts:
|
||||
p = p.strip()
|
||||
if not p:
|
||||
continue
|
||||
# very long run-ons with no punctuation: chop on discourse markers
|
||||
if len(p) > 220:
|
||||
out.extend(s.strip() for s in re.split(r"\s+(?:and then|but|so|however)\s+", p) if s.strip())
|
||||
else:
|
||||
out.append(p)
|
||||
return out
|
||||
|
||||
|
||||
def find_numbers(text):
|
||||
"""Callback numbers, both digit-written and spoken-out."""
|
||||
found = []
|
||||
for m in _PHONE_RE.finditer(text):
|
||||
cand = m.group(0).strip(" .-")
|
||||
digits = re.sub(r"\D", "", cand)
|
||||
if 7 <= len(digits) <= 15:
|
||||
found.append(cand)
|
||||
# spoken digit runs: "oh seven nine one ..."
|
||||
words = re.findall(r"[a-z]+", text.lower())
|
||||
run, runs = [], []
|
||||
for w in words:
|
||||
if w in _SPOKEN_DIGITS:
|
||||
run.append(_SPOKEN_DIGITS[w])
|
||||
else:
|
||||
if len(run) >= 7:
|
||||
runs.append("".join(run))
|
||||
run = []
|
||||
if len(run) >= 7:
|
||||
runs.append("".join(run))
|
||||
found.extend(runs)
|
||||
seen, uniq = set(), []
|
||||
for f in found:
|
||||
k = re.sub(r"\D", "", f)
|
||||
if k and k not in seen:
|
||||
seen.add(k)
|
||||
uniq.append(f)
|
||||
return uniq[:3]
|
||||
|
||||
|
||||
def detect_intents(text):
|
||||
low = text.lower()
|
||||
return [label for label, pat in _INTENTS if re.search(pat, low)]
|
||||
|
||||
|
||||
def summarise(text, max_sentences=3):
|
||||
"""Frequency-scored extractive summary, original order preserved."""
|
||||
sents = _split_sentences(text)
|
||||
if not sents:
|
||||
return ""
|
||||
if len(sents) <= max_sentences:
|
||||
return " ".join(sents)
|
||||
|
||||
freq = {}
|
||||
for w in re.findall(r"[a-z']+", text.lower()):
|
||||
if w in _STOP or len(w) < 3:
|
||||
continue
|
||||
freq[w] = freq.get(w, 0) + 1
|
||||
if not freq:
|
||||
return " ".join(sents[:max_sentences])
|
||||
top = max(freq.values())
|
||||
|
||||
scored = []
|
||||
for i, s in enumerate(sents):
|
||||
words = [w for w in re.findall(r"[a-z']+", s.lower()) if w not in _STOP and len(w) >= 3]
|
||||
if not words:
|
||||
score = 0.0
|
||||
else:
|
||||
score = sum(freq.get(w, 0) / top for w in words) / (len(words) ** 0.5)
|
||||
if i == 0:
|
||||
score *= 1.35 # openers carry the reason for the call
|
||||
if re.search(r"\d", s):
|
||||
score *= 1.15 # numbers/dates matter
|
||||
scored.append((score, i, s))
|
||||
|
||||
keep = sorted(sorted(scored, reverse=True)[:max_sentences], key=lambda t: t[1])
|
||||
return " ".join(s for _, _, s in keep)
|
||||
|
||||
|
||||
# --------------------------------------------------------------- MIME output
|
||||
def header_val(msg, name, default=""):
|
||||
v = msg.get(name)
|
||||
return str(v) if v else default
|
||||
|
||||
|
||||
def parse_vm_fields(body_text, subject):
|
||||
"""Best-effort scrape of the Asterisk notification body for context."""
|
||||
f = {}
|
||||
m = re.search(r"in mailbox (\S+?)[ ,]", body_text)
|
||||
if m:
|
||||
f["mailbox"] = m.group(1)
|
||||
m = re.search(r"from (.+?), on (.+?),? so you might", body_text, re.S)
|
||||
if m:
|
||||
f["from"] = m.group(1).strip()
|
||||
f["date"] = m.group(2).strip().rstrip(",").replace("\n", " ")
|
||||
else:
|
||||
m = re.search(r"from (.+?), on (.+?),", body_text)
|
||||
if m:
|
||||
f["from"] = m.group(1).strip()
|
||||
f["date"] = m.group(2).strip()
|
||||
m = re.search(r"a (\d+:\d+) long message", body_text)
|
||||
if m:
|
||||
f["duration"] = m.group(1)
|
||||
m = re.search(r"number (\d+)", body_text)
|
||||
if m:
|
||||
f["msgnum"] = m.group(1)
|
||||
m = re.search(r"mailbox (\S+)", subject)
|
||||
if m and "mailbox" not in f:
|
||||
f["mailbox"] = m.group(1)
|
||||
return f
|
||||
|
||||
|
||||
def build_plain(fields, summary, intents, numbers, transcript):
|
||||
L = []
|
||||
L.append("NEW VOICEMAIL")
|
||||
L.append("=" * 40)
|
||||
meta = [("From", fields.get("from")), ("Mailbox", fields.get("mailbox")),
|
||||
("Received", fields.get("date")), ("Duration", fields.get("duration")),
|
||||
("Message", fields.get("msgnum"))]
|
||||
for k, v in meta:
|
||||
if v:
|
||||
L.append("%-10s %s" % (k + ":", v))
|
||||
L.append("")
|
||||
L.append("SUMMARY")
|
||||
L.append("-" * 40)
|
||||
L.append(summary or "(no speech detected in this recording)")
|
||||
if intents:
|
||||
L.append("")
|
||||
L.append("Tags: " + ", ".join(intents))
|
||||
if numbers:
|
||||
L.append("Callback number(s) heard: " + ", ".join(numbers))
|
||||
L.append("")
|
||||
L.append("FULL TRANSCRIPT")
|
||||
L.append("-" * 40)
|
||||
L.append(transcript or "(no speech detected)")
|
||||
L.append("")
|
||||
L.append("The original recording is attached.")
|
||||
return "\n".join(L)
|
||||
|
||||
|
||||
def build_html(fields, summary, intents, numbers, transcript):
|
||||
def row(label, value):
|
||||
if not value:
|
||||
return ""
|
||||
return (
|
||||
'<tr>'
|
||||
'<td style="padding:4px 14px 4px 0;font:600 12px/1.5 -apple-system,Segoe UI,Roboto,'
|
||||
'Helvetica,Arial,sans-serif;color:#8a94a6;text-transform:uppercase;letter-spacing:.5px;'
|
||||
'white-space:nowrap;vertical-align:top;">%s</td>'
|
||||
'<td style="padding:4px 0;font:400 14px/1.5 -apple-system,Segoe UI,Roboto,Helvetica,'
|
||||
'Arial,sans-serif;color:#1c2430;">%s</td></tr>' % (escape(label), escape(value))
|
||||
)
|
||||
|
||||
chips = "".join(
|
||||
'<span style="display:inline-block;margin:0 6px 6px 0;padding:4px 11px;border-radius:999px;'
|
||||
'background:#eef3ff;color:%s;font:600 12px/1.4 -apple-system,Segoe UI,Roboto,Helvetica,'
|
||||
'Arial,sans-serif;">%s</span>' % (ACCENT, escape(t)) for t in intents)
|
||||
|
||||
numbers_html = ""
|
||||
if numbers:
|
||||
links = " ".join(
|
||||
'<a href="tel:%s" style="color:%s;text-decoration:none;font-weight:600;">%s</a>'
|
||||
% (escape(re.sub(r"[^\d+]", "", n)), ACCENT, escape(n)) for n in numbers)
|
||||
numbers_html = (
|
||||
'<div style="margin-top:14px;padding:12px 14px;background:#f6f8fc;border-left:3px solid %s;'
|
||||
'border-radius:4px;font:400 14px/1.5 -apple-system,Segoe UI,Roboto,Helvetica,Arial,'
|
||||
'sans-serif;color:#1c2430;">📞 Callback number heard: %s</div>' % (ACCENT, links))
|
||||
|
||||
transcript_html = escape(transcript or "(no speech detected)").replace("\n", "<br>")
|
||||
summary_html = escape(summary or "(no speech detected in this recording)")
|
||||
|
||||
return """<!DOCTYPE html>
|
||||
<html><head><meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width,initial-scale=1"></head>
|
||||
<body style="margin:0;padding:24px 12px;background:#eef1f6;">
|
||||
<table role="presentation" width="100%%" cellpadding="0" cellspacing="0" style="border-collapse:collapse;">
|
||||
<tr><td align="center">
|
||||
<table role="presentation" width="600" cellpadding="0" cellspacing="0"
|
||||
style="width:600px;max-width:100%%;border-collapse:collapse;background:#ffffff;
|
||||
border-radius:12px;overflow:hidden;box-shadow:0 2px 10px rgba(20,30,50,.08);">
|
||||
|
||||
<tr><td style="background:linear-gradient(135deg,%(accent)s,#1b4bb8);padding:22px 28px;">
|
||||
<div style="font:700 19px/1.3 -apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif;color:#fff;">
|
||||
✉️ New %(brand)s</div>
|
||||
<div style="font:400 13px/1.5 -apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif;
|
||||
color:#d9e3ff;margin-top:3px;">Transcribed and summarised automatically</div>
|
||||
</td></tr>
|
||||
|
||||
<tr><td style="padding:22px 28px 6px 28px;">
|
||||
<table role="presentation" cellpadding="0" cellspacing="0" style="border-collapse:collapse;">
|
||||
%(rows)s
|
||||
</table>
|
||||
</td></tr>
|
||||
|
||||
<tr><td style="padding:16px 28px 0 28px;">
|
||||
<div style="font:700 12px/1.5 -apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif;
|
||||
color:#8a94a6;text-transform:uppercase;letter-spacing:.6px;">Summary</div>
|
||||
<div style="margin-top:8px;padding:16px 18px;background:#f9fbff;border:1px solid #e3e9f5;
|
||||
border-radius:8px;font:400 15px/1.6 -apple-system,Segoe UI,Roboto,Helvetica,Arial,
|
||||
sans-serif;color:#101828;">%(summary)s</div>
|
||||
%(chipwrap)s
|
||||
%(numbers)s
|
||||
</td></tr>
|
||||
|
||||
<tr><td style="padding:22px 28px 0 28px;">
|
||||
<div style="font:700 12px/1.5 -apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif;
|
||||
color:#8a94a6;text-transform:uppercase;letter-spacing:.6px;">Full transcript</div>
|
||||
<div style="margin-top:8px;padding:16px 18px;background:#ffffff;border:1px solid #e8ecf3;
|
||||
border-radius:8px;font:400 14px/1.7 -apple-system,Segoe UI,Roboto,Helvetica,Arial,
|
||||
sans-serif;color:#39424e;">%(transcript)s</div>
|
||||
</td></tr>
|
||||
|
||||
<tr><td style="padding:20px 28px 26px 28px;">
|
||||
<div style="padding:12px 14px;background:#f3f5f9;border-radius:8px;
|
||||
font:400 13px/1.5 -apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif;
|
||||
color:#5b6675;">🎧 The original recording is attached to this email.</div>
|
||||
</td></tr>
|
||||
|
||||
<tr><td style="padding:14px 28px;background:#fafbfd;border-top:1px solid #eceff5;
|
||||
font:400 11px/1.5 -apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif;
|
||||
color:#98a1b0;">
|
||||
Machine transcription may contain errors — listen to the recording if in doubt.
|
||||
</td></tr>
|
||||
|
||||
</table>
|
||||
</td></tr></table>
|
||||
</body></html>""" % {
|
||||
"accent": ACCENT,
|
||||
"brand": escape(BRAND),
|
||||
"rows": (row("From", fields.get("from")) + row("Mailbox", fields.get("mailbox")) +
|
||||
row("Received", fields.get("date")) + row("Duration", fields.get("duration")) +
|
||||
row("Message", fields.get("msgnum"))),
|
||||
"summary": summary_html,
|
||||
"chipwrap": ('<div style="margin-top:12px;">%s</div>' % chips) if chips else "",
|
||||
"numbers": numbers_html,
|
||||
"transcript": transcript_html,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------- main
|
||||
def relay(raw_bytes, envelope_from, rcpts):
|
||||
dry = os.environ.get("VM_DRYRUN")
|
||||
if dry:
|
||||
with open(dry, "wb") as fh:
|
||||
fh.write(raw_bytes if isinstance(raw_bytes, bytes) else raw_bytes.encode())
|
||||
log("DRYRUN wrote %s (%d bytes) for %s" % (dry, len(raw_bytes), rcpts))
|
||||
return
|
||||
with smtplib.SMTP(SMTP_HOST, SMTP_PORT, timeout=30) as s:
|
||||
s.sendmail(envelope_from, rcpts, raw_bytes)
|
||||
|
||||
|
||||
def main():
|
||||
raw = sys.stdin.buffer.read()
|
||||
orig = email.message_from_bytes(raw, policy=email.policy.default)
|
||||
|
||||
rcpts = []
|
||||
for h in ("To", "Cc", "Bcc"):
|
||||
for addr in email.utils.getaddresses(orig.get_all(h, [])):
|
||||
if addr[1]:
|
||||
rcpts.append(addr[1])
|
||||
env_from = (email.utils.parseaddr(header_val(orig, "From"))[1]
|
||||
or "voicemail@localhost")
|
||||
if not rcpts:
|
||||
log("no recipients found; relaying original")
|
||||
relay(raw, env_from, rcpts or [env_from])
|
||||
return
|
||||
|
||||
try:
|
||||
# ---- pull out audio + original text body
|
||||
audio_part, body_text = None, ""
|
||||
for part in orig.walk():
|
||||
ctype = part.get_content_type()
|
||||
if part.get_content_maintype() == "audio" or (
|
||||
part.get_filename() or "").lower().endswith(
|
||||
(".wav", ".gsm", ".mp3", ".ogg", ".WAV")):
|
||||
if audio_part is None:
|
||||
audio_part = part
|
||||
elif ctype == "text/plain" and not body_text:
|
||||
try:
|
||||
body_text = part.get_content()
|
||||
except Exception:
|
||||
body_text = part.get_payload(decode=True).decode("utf-8", "replace")
|
||||
|
||||
if audio_part is None:
|
||||
log("no audio attachment; relaying original")
|
||||
relay(raw, env_from, rcpts)
|
||||
return
|
||||
|
||||
audio_bytes = audio_part.get_payload(decode=True)
|
||||
fname = audio_part.get_filename() or "voicemail.wav"
|
||||
suffix = os.path.splitext(fname)[1] or ".wav"
|
||||
|
||||
with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tf:
|
||||
tf.write(audio_bytes)
|
||||
tmp_audio = tf.name
|
||||
|
||||
try:
|
||||
src = tmp_audio
|
||||
if suffix.lower() not in (".wav", ".mp3", ".ogg", ".flac", ".m4a"):
|
||||
conv = tmp_audio + ".wav"
|
||||
subprocess.run(["/usr/bin/sox", tmp_audio, "-r", "16000", "-c", "1", conv],
|
||||
check=True, capture_output=True)
|
||||
src = conv
|
||||
transcript = transcribe(src)
|
||||
finally:
|
||||
for p in (tmp_audio, tmp_audio + ".wav"):
|
||||
try:
|
||||
os.unlink(p)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
fields = parse_vm_fields(body_text, header_val(orig, "Subject"))
|
||||
|
||||
# ---- caller-ID -> contact name (best-effort)
|
||||
try:
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import vm_contacts
|
||||
who = vm_contacts.resolve(fields.get("from", ""), log=log)
|
||||
if who:
|
||||
fields["contact"] = who
|
||||
num = vm_contacts.extract_number(fields.get("from", ""))
|
||||
fields["from"] = "%s (%s)" % (who, num) if num else who
|
||||
except Exception:
|
||||
log("contact lookup failed (non-fatal):\n" + traceback.format_exc())
|
||||
|
||||
summary = summarise(transcript)
|
||||
intents = detect_intents(transcript)
|
||||
numbers = find_numbers(transcript)
|
||||
|
||||
# ---- assemble multipart/mixed > multipart/alternative + attachment
|
||||
out = EmailMessage()
|
||||
for h in ("From", "To", "Cc", "Reply-To", "Date", "Message-ID",
|
||||
"X-Asterisk-CallerID", "X-Asterisk-VM-Mailbox"):
|
||||
if orig.get(h):
|
||||
out[h] = orig[h]
|
||||
if not out.get("From"):
|
||||
out["From"] = env_from
|
||||
|
||||
subj_bits = []
|
||||
if fields.get("from"):
|
||||
subj_bits.append(fields["from"])
|
||||
if fields.get("mailbox"):
|
||||
subj_bits.append("mbox %s" % fields["mailbox"])
|
||||
gist = (summary or transcript or "no speech detected").strip()
|
||||
if len(gist) > 90:
|
||||
gist = gist[:87].rsplit(" ", 1)[0] + "..."
|
||||
out["Subject"] = "Voicemail%s: %s" % (
|
||||
(" from " + subj_bits[0]) if subj_bits else "", gist)
|
||||
out["X-Voicemail-Transcribed"] = "faster-whisper/%s" % MODEL_SIZE
|
||||
if intents:
|
||||
out["X-Voicemail-Tags"] = ", ".join(intents)
|
||||
|
||||
out.set_content(build_plain(fields, summary, intents, numbers, transcript))
|
||||
out.add_alternative(build_html(fields, summary, intents, numbers, transcript),
|
||||
subtype="html")
|
||||
|
||||
maintype, _, subtype = (audio_part.get_content_type() or "audio/x-wav").partition("/")
|
||||
out.add_attachment(audio_bytes, maintype=maintype or "audio",
|
||||
subtype=subtype or "x-wav", filename=fname)
|
||||
|
||||
relay(out.as_bytes(), env_from, rcpts)
|
||||
log("sent enriched notification to %s (%d char transcript, tags=%s)"
|
||||
% (",".join(rcpts), len(transcript), intents))
|
||||
|
||||
# ---- persist to the SQLite store for the web app (best-effort)
|
||||
try:
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import vm_store
|
||||
con = vm_store.connect()
|
||||
origtime = None
|
||||
d = orig.get("Date")
|
||||
if d:
|
||||
try:
|
||||
origtime = int(email.utils.mktime_tz(email.utils.parsedate_tz(str(d))))
|
||||
except Exception:
|
||||
origtime = None
|
||||
dur = None
|
||||
if fields.get("duration") and ":" in fields["duration"]:
|
||||
mm, ss = fields["duration"].split(":")[:2]
|
||||
try:
|
||||
dur = int(mm) * 60 + int(ss)
|
||||
except ValueError:
|
||||
dur = None
|
||||
vm_store.add_message(
|
||||
con, fields.get("mailbox") or "unknown",
|
||||
callerid=fields.get("from"), contact_name=fields.get("contact"),
|
||||
origtime=origtime, duration=dur, transcript=transcript,
|
||||
summary=summary, intents=intents, numbers=numbers,
|
||||
audio_bytes=audio_bytes,
|
||||
audio_ext=(os.path.splitext(fname)[1].lstrip(".") or "wav"))
|
||||
con.close()
|
||||
log("stored message for mailbox %s in the web-app database"
|
||||
% fields.get("mailbox"))
|
||||
except Exception:
|
||||
log("db store failed (non-fatal):\n" + traceback.format_exc())
|
||||
|
||||
# ---- Telegram: strictly best-effort, email is already delivered
|
||||
try:
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import vm_telegram
|
||||
vm_telegram.notify(
|
||||
fields.get("mailbox"), fields, summary, intents, numbers,
|
||||
transcript, audio_bytes=audio_bytes, audio_name=fname, log=log)
|
||||
except Exception:
|
||||
log("telegram notify failed (email was sent OK):\n" + traceback.format_exc())
|
||||
|
||||
except Exception:
|
||||
log("FAILED, relaying original:\n" + traceback.format_exc())
|
||||
try:
|
||||
relay(raw, env_from, rcpts)
|
||||
except Exception:
|
||||
log("relay of original ALSO failed:\n" + traceback.format_exc())
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
158
src/vm_store.py
Normal file
158
src/vm_store.py
Normal file
@ -0,0 +1,158 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
SQLite store for voicemail transcripts, shared by the mailcmd hook and the
|
||||
web app.
|
||||
|
||||
Design notes:
|
||||
* WAL mode, because the mailcmd process writes while the web app reads.
|
||||
* The audio is copied into a content-addressed store rather than referencing
|
||||
the Asterisk spool, since Asterisk renumbers msgNNNN files whenever a
|
||||
message is deleted - a stored path would silently point at the wrong
|
||||
recording. The spool path is kept only as a hint for delete-on-disk.
|
||||
* Every write is idempotent on (mailbox, origtime, callerid) so a re-run of
|
||||
the importer cannot duplicate rows.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import time
|
||||
|
||||
DB_PATH = os.environ.get("VM_DB", "/var/lib/vm-transcribe/voicemail.db")
|
||||
AUDIO_DIR = os.environ.get("VM_AUDIO_DIR", "/var/lib/vm-transcribe/audio")
|
||||
|
||||
SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS messages (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
mailbox TEXT NOT NULL,
|
||||
context TEXT DEFAULT 'default',
|
||||
folder TEXT DEFAULT 'INBOX',
|
||||
callerid TEXT,
|
||||
contact_name TEXT,
|
||||
origtime INTEGER,
|
||||
duration INTEGER,
|
||||
transcript TEXT,
|
||||
summary TEXT,
|
||||
intents TEXT,
|
||||
numbers TEXT,
|
||||
audio_sha TEXT,
|
||||
audio_ext TEXT DEFAULT 'wav',
|
||||
spool_path TEXT,
|
||||
is_read INTEGER DEFAULT 0,
|
||||
created_at INTEGER,
|
||||
UNIQUE (mailbox, origtime, callerid)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_msg_mailbox ON messages (mailbox, origtime DESC);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS settings (
|
||||
mailbox TEXT NOT NULL,
|
||||
key TEXT NOT NULL,
|
||||
value TEXT,
|
||||
PRIMARY KEY (mailbox, key)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS sessions (
|
||||
token TEXT PRIMARY KEY,
|
||||
mailbox TEXT NOT NULL,
|
||||
created_at INTEGER,
|
||||
expires_at INTEGER
|
||||
);
|
||||
"""
|
||||
|
||||
# Settings a mailbox user is allowed to change, with defaults.
|
||||
USER_SETTINGS = {
|
||||
"email_enabled": ("yes", "Send an email notification"),
|
||||
"email_address": ("", "Override the notification email address"),
|
||||
"attach_audio": ("yes", "Attach the recording to the email"),
|
||||
"telegram_enabled": ("no", "Send a Telegram DM"),
|
||||
"telegram_chat_id": ("", "Telegram chat ID (message the bot first)"),
|
||||
"telegram_audio": ("yes", "Include the recording as a Telegram voice note"),
|
||||
"telegram_transcript":("yes", "Send the full transcript as a follow-up"),
|
||||
"transcribe": ("yes", "Transcribe recordings to text"),
|
||||
"summarise": ("yes", "Include an automatic summary"),
|
||||
"contact_lookup": ("yes", "Resolve caller ID against contacts"),
|
||||
}
|
||||
|
||||
|
||||
def connect(path=None):
|
||||
p = os.path.abspath(path or DB_PATH)
|
||||
os.makedirs(os.path.dirname(p), exist_ok=True)
|
||||
con = sqlite3.connect(p, timeout=20)
|
||||
con.row_factory = sqlite3.Row
|
||||
con.execute("PRAGMA journal_mode=WAL")
|
||||
con.execute("PRAGMA busy_timeout=10000")
|
||||
con.executescript(SCHEMA)
|
||||
return con
|
||||
|
||||
|
||||
def store_audio(audio_bytes, ext="wav", audio_dir=None):
|
||||
"""Content-addressed write. Returns the sha256 hex digest."""
|
||||
d = audio_dir or AUDIO_DIR
|
||||
sha = hashlib.sha256(audio_bytes).hexdigest()
|
||||
sub = os.path.join(d, sha[:2])
|
||||
os.makedirs(sub, exist_ok=True)
|
||||
dest = os.path.join(sub, "%s.%s" % (sha, ext))
|
||||
if not os.path.exists(dest):
|
||||
tmp = dest + ".tmp"
|
||||
with open(tmp, "wb") as fh:
|
||||
fh.write(audio_bytes)
|
||||
os.replace(tmp, dest)
|
||||
return sha
|
||||
|
||||
|
||||
def audio_path(sha, ext="wav", audio_dir=None):
|
||||
d = audio_dir or AUDIO_DIR
|
||||
return os.path.join(d, sha[:2], "%s.%s" % (sha, ext))
|
||||
|
||||
|
||||
def add_message(con, mailbox, callerid=None, contact_name=None, origtime=None,
|
||||
duration=None, transcript="", summary="", intents=None,
|
||||
numbers=None, audio_bytes=None, audio_ext="wav",
|
||||
spool_path=None, context="default", folder="INBOX"):
|
||||
sha = store_audio(audio_bytes, audio_ext) if audio_bytes else None
|
||||
now = int(time.time())
|
||||
cur = con.execute(
|
||||
"""INSERT OR IGNORE INTO messages
|
||||
(mailbox, context, folder, callerid, contact_name, origtime, duration,
|
||||
transcript, summary, intents, numbers, audio_sha, audio_ext,
|
||||
spool_path, created_at)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)""",
|
||||
(str(mailbox), context, folder, callerid, contact_name,
|
||||
int(origtime or now), duration, transcript, summary,
|
||||
json.dumps(intents or []), json.dumps(numbers or []),
|
||||
sha, audio_ext, spool_path, now))
|
||||
con.commit()
|
||||
return cur.lastrowid
|
||||
|
||||
|
||||
def get_setting(con, mailbox, key, default=None):
|
||||
row = con.execute("SELECT value FROM settings WHERE mailbox=? AND key=?",
|
||||
(str(mailbox), key)).fetchone()
|
||||
if row is not None:
|
||||
return row["value"]
|
||||
if default is not None:
|
||||
return default
|
||||
return USER_SETTINGS.get(key, ("", ""))[0]
|
||||
|
||||
|
||||
def get_settings(con, mailbox):
|
||||
out = {k: v[0] for k, v in USER_SETTINGS.items()}
|
||||
for r in con.execute("SELECT key, value FROM settings WHERE mailbox=?",
|
||||
(str(mailbox),)):
|
||||
if r["key"] in USER_SETTINGS:
|
||||
out[r["key"]] = r["value"]
|
||||
return out
|
||||
|
||||
|
||||
def set_setting(con, mailbox, key, value):
|
||||
if key not in USER_SETTINGS:
|
||||
raise KeyError("unknown setting %r" % key)
|
||||
con.execute("""INSERT INTO settings (mailbox, key, value) VALUES (?,?,?)
|
||||
ON CONFLICT(mailbox, key) DO UPDATE SET value=excluded.value""",
|
||||
(str(mailbox), key, value))
|
||||
con.commit()
|
||||
|
||||
|
||||
def truthy(v):
|
||||
return str(v).strip().lower() in ("1", "yes", "true", "on")
|
||||
247
src/vm_telegram.py
Normal file
247
src/vm_telegram.py
Normal file
@ -0,0 +1,247 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Telegram delivery for Asterisk voicemail notifications.
|
||||
|
||||
Routing is driven by /opt/vm-transcribe/telegram.conf, re-read on every call so
|
||||
edits take effect without restarting anything.
|
||||
|
||||
Delivery is strictly best-effort: the email has already been sent by the time
|
||||
this runs, so every failure is logged and swallowed rather than raised.
|
||||
"""
|
||||
|
||||
import configparser
|
||||
import json
|
||||
import mimetypes
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from html import escape
|
||||
|
||||
CONF_PATH = os.environ.get("VM_TG_CONF", "/opt/vm-transcribe/telegram.conf")
|
||||
API = "https://api.telegram.org/bot%s/%s"
|
||||
|
||||
# Telegram hard limits
|
||||
CAPTION_LIMIT = 1024
|
||||
MESSAGE_LIMIT = 4096
|
||||
|
||||
|
||||
# --------------------------------------------------------------- config load
|
||||
class Route:
|
||||
__slots__ = ("chat_ids", "send_audio", "send_transcript", "timeout", "token")
|
||||
|
||||
def __init__(self, token, chat_ids, send_audio, send_transcript, timeout):
|
||||
self.token = token
|
||||
self.chat_ids = chat_ids
|
||||
self.send_audio = send_audio
|
||||
self.send_transcript = send_transcript
|
||||
self.timeout = timeout
|
||||
|
||||
|
||||
def _split_ids(raw):
|
||||
return [c.strip() for c in (raw or "").replace(";", ",").split(",") if c.strip()]
|
||||
|
||||
|
||||
def load_route(mailbox):
|
||||
"""Return a Route for this mailbox, or None if Telegram is off/unmapped."""
|
||||
if not os.path.exists(CONF_PATH):
|
||||
return None
|
||||
|
||||
cp = configparser.ConfigParser(inline_comment_prefixes=("#", ";"))
|
||||
cp.read(CONF_PATH)
|
||||
if not cp.has_section("telegram"):
|
||||
return None
|
||||
|
||||
g = cp["telegram"]
|
||||
if not g.getboolean("enabled", fallback=False):
|
||||
return None
|
||||
|
||||
token = (g.get("token", "") or "").strip()
|
||||
if not token:
|
||||
return None
|
||||
|
||||
sect = "mailbox:%s" % mailbox if mailbox else None
|
||||
m = cp[sect] if (sect and cp.has_section(sect)) else None
|
||||
|
||||
def val(key, fallback):
|
||||
if m is not None and key in m:
|
||||
return m[key]
|
||||
return g.get(key, fallback)
|
||||
|
||||
def boolval(key, fallback):
|
||||
if m is not None and key in m:
|
||||
return m.getboolean(key, fallback=fallback)
|
||||
return g.getboolean(key, fallback=fallback)
|
||||
|
||||
chat_ids = _split_ids(val("chat_id", "")) if m is not None else []
|
||||
if not chat_ids:
|
||||
chat_ids = _split_ids(g.get("default_chat_id", ""))
|
||||
if not chat_ids:
|
||||
return None
|
||||
|
||||
return Route(
|
||||
token=token,
|
||||
chat_ids=chat_ids,
|
||||
send_audio=boolval("send_audio", True),
|
||||
send_transcript=boolval("send_transcript", True),
|
||||
timeout=int(val("timeout", "20") or 20),
|
||||
)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ HTTP bits
|
||||
def _multipart(fields, files):
|
||||
"""Build a multipart/form-data body. files = [(name, filename, bytes, ctype)]"""
|
||||
boundary = "----vmtg%s" % os.urandom(12).hex()
|
||||
out = bytearray()
|
||||
for k, v in fields.items():
|
||||
out += b"--%s\r\n" % boundary.encode()
|
||||
out += b'Content-Disposition: form-data; name="%s"\r\n\r\n' % k.encode()
|
||||
out += str(v).encode() + b"\r\n"
|
||||
for name, fname, data, ctype in files:
|
||||
out += b"--%s\r\n" % boundary.encode()
|
||||
out += (b'Content-Disposition: form-data; name="%s"; filename="%s"\r\n'
|
||||
% (name.encode(), fname.encode()))
|
||||
out += b"Content-Type: %s\r\n\r\n" % ctype.encode()
|
||||
out += data + b"\r\n"
|
||||
out += b"--%s--\r\n" % boundary.encode()
|
||||
return bytes(out), "multipart/form-data; boundary=%s" % boundary
|
||||
|
||||
|
||||
def _call(route, method, fields, files=None, log=print):
|
||||
url = API % (route.token, method)
|
||||
try:
|
||||
if files:
|
||||
body, ctype = _multipart(fields, files)
|
||||
else:
|
||||
body = urllib.parse.urlencode(fields).encode()
|
||||
ctype = "application/x-www-form-urlencoded"
|
||||
req = urllib.request.Request(url, data=body, headers={"Content-Type": ctype})
|
||||
with urllib.request.urlopen(req, timeout=route.timeout) as r:
|
||||
resp = json.loads(r.read().decode("utf-8", "replace"))
|
||||
if not resp.get("ok"):
|
||||
log("telegram %s failed: %s" % (method, resp.get("description")))
|
||||
return False
|
||||
return True
|
||||
except Exception as e:
|
||||
log("telegram %s error: %s" % (method, e))
|
||||
return False
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- formatting
|
||||
def _clip(s, limit):
|
||||
if len(s) <= limit:
|
||||
return s
|
||||
return s[: limit - 20].rsplit(" ", 1)[0] + "\n\n[...truncated]"
|
||||
|
||||
|
||||
def build_caption(fields, summary, intents, numbers):
|
||||
L = ["\U0001F4E7 <b>New voicemail</b>"]
|
||||
if fields.get("from"):
|
||||
L.append("\U0001F464 <b>From:</b> %s" % escape(fields["from"]))
|
||||
meta = []
|
||||
if fields.get("mailbox"):
|
||||
meta.append("mailbox %s" % escape(fields["mailbox"]))
|
||||
if fields.get("duration"):
|
||||
meta.append(escape(fields["duration"]))
|
||||
if fields.get("date"):
|
||||
meta.append(escape(fields["date"]))
|
||||
if meta:
|
||||
L.append("\U0001F553 %s" % " \u00b7 ".join(meta))
|
||||
L.append("")
|
||||
L.append("<b>Summary</b>")
|
||||
L.append(escape(summary or "(no speech detected in this recording)"))
|
||||
if intents:
|
||||
L.append("")
|
||||
L.append("\U0001F3F7 " + " ".join("#" + re.sub(r"[^A-Za-z]+", "", t) for t in intents))
|
||||
if numbers:
|
||||
pretty = ", ".join(
|
||||
'<a href="tel:%s">%s</a>' % (escape(re.sub(r"[^\d+]", "", n)), escape(n))
|
||||
for n in numbers)
|
||||
L.append("\U0001F4DE Callback: %s" % pretty)
|
||||
return _clip("\n".join(L), CAPTION_LIMIT)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- audio prep
|
||||
def to_voice_ogg(audio_bytes, suffix, log=print):
|
||||
"""Transcode to ogg/opus for a native Telegram voice note. None on failure."""
|
||||
ff = shutil.which("ffmpeg") or "/usr/bin/ffmpeg"
|
||||
if not os.path.exists(ff):
|
||||
log("ffmpeg not found; sending original audio as a document")
|
||||
return None
|
||||
src = dst = None
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(suffix=suffix or ".wav", delete=False) as tf:
|
||||
tf.write(audio_bytes)
|
||||
src = tf.name
|
||||
dst = src + ".ogg"
|
||||
subprocess.run(
|
||||
[ff, "-v", "error", "-y", "-i", src,
|
||||
"-c:a", "libopus", "-b:a", "24k", "-ar", "48000", "-ac", "1",
|
||||
"-application", "voip", dst],
|
||||
check=True, capture_output=True, timeout=120)
|
||||
with open(dst, "rb") as fh:
|
||||
return fh.read()
|
||||
except Exception as e:
|
||||
log("opus transcode failed (%s); falling back to raw audio" % e)
|
||||
return None
|
||||
finally:
|
||||
for p in (src, dst):
|
||||
if p:
|
||||
try:
|
||||
os.unlink(p)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- entrypoint
|
||||
def notify(mailbox, fields, summary, intents, numbers, transcript,
|
||||
audio_bytes=None, audio_name="voicemail.wav", log=print):
|
||||
"""Best-effort Telegram delivery. Returns number of chats notified."""
|
||||
try:
|
||||
route = load_route(mailbox)
|
||||
except Exception as e:
|
||||
log("telegram config error: %s" % e)
|
||||
return 0
|
||||
if route is None:
|
||||
return 0
|
||||
|
||||
caption = build_caption(fields, summary, intents, numbers)
|
||||
suffix = os.path.splitext(audio_name)[1] or ".wav"
|
||||
|
||||
voice = None
|
||||
if route.send_audio and audio_bytes:
|
||||
voice = to_voice_ogg(audio_bytes, suffix, log=log)
|
||||
|
||||
sent = 0
|
||||
for chat_id in route.chat_ids:
|
||||
ok = False
|
||||
if route.send_audio and audio_bytes:
|
||||
if voice is not None:
|
||||
ok = _call(route, "sendVoice",
|
||||
{"chat_id": chat_id, "caption": caption, "parse_mode": "HTML"},
|
||||
files=[("voice", "voicemail.ogg", voice, "audio/ogg")], log=log)
|
||||
if not ok:
|
||||
ctype = mimetypes.guess_type(audio_name)[0] or "audio/wav"
|
||||
ok = _call(route, "sendDocument",
|
||||
{"chat_id": chat_id, "caption": caption, "parse_mode": "HTML"},
|
||||
files=[("document", audio_name, audio_bytes, ctype)], log=log)
|
||||
if not ok:
|
||||
ok = _call(route, "sendMessage",
|
||||
{"chat_id": chat_id, "text": caption, "parse_mode": "HTML",
|
||||
"disable_web_page_preview": "true"}, log=log)
|
||||
|
||||
if ok and route.send_transcript and transcript and transcript.strip() != (summary or "").strip():
|
||||
body = "\U0001F4DD <b>Full transcript</b>\n\n" + escape(transcript)
|
||||
_call(route, "sendMessage",
|
||||
{"chat_id": chat_id, "text": _clip(body, MESSAGE_LIMIT),
|
||||
"parse_mode": "HTML", "disable_web_page_preview": "true"}, log=log)
|
||||
|
||||
if ok:
|
||||
sent += 1
|
||||
|
||||
log("telegram: notified %d/%d chat(s) for mailbox %s"
|
||||
% (sent, len(route.chat_ids), mailbox))
|
||||
return sent
|
||||
89
src/vm_tg_setup.py
Normal file
89
src/vm_tg_setup.py
Normal file
@ -0,0 +1,89 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Helper: discover Telegram chat IDs, and send a test voicemail notification.
|
||||
|
||||
# 1. who has messaged the bot?
|
||||
/opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py ids
|
||||
|
||||
# 2. verify a route end-to-end (uses telegram.conf, real audio optional)
|
||||
/opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py test 1001
|
||||
"""
|
||||
import configparser
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import vm_telegram as T
|
||||
|
||||
|
||||
def token():
|
||||
cp = configparser.ConfigParser(inline_comment_prefixes=("#", ";"))
|
||||
cp.read(T.CONF_PATH)
|
||||
tok = (cp.get("telegram", "token", fallback="") or "").strip()
|
||||
if not tok:
|
||||
sys.exit("No token set in %s" % T.CONF_PATH)
|
||||
return tok
|
||||
|
||||
|
||||
def cmd_ids():
|
||||
tok = token()
|
||||
url = T.API % (tok, "getUpdates")
|
||||
with urllib.request.urlopen(url, timeout=20) as r:
|
||||
d = json.loads(r.read().decode())
|
||||
if not d.get("ok"):
|
||||
sys.exit("API error: %s" % d.get("description"))
|
||||
seen = {}
|
||||
for u in d.get("result", []):
|
||||
msg = u.get("message") or u.get("channel_post") or {}
|
||||
ch = msg.get("chat") or {}
|
||||
if ch.get("id") is not None:
|
||||
name = " ".join(x for x in (ch.get("title"), ch.get("first_name"),
|
||||
ch.get("last_name"),
|
||||
("@" + ch["username"]) if ch.get("username") else None) if x)
|
||||
seen[ch["id"]] = "%s [%s]" % (name or "?", ch.get("type"))
|
||||
if not seen:
|
||||
print("No chats found. Send your bot a message first (or /start), then re-run.")
|
||||
print("Note: getUpdates returns nothing if a webhook is set, and only ~24h of history.")
|
||||
return
|
||||
print("Chat IDs that have talked to this bot:")
|
||||
for cid, who in seen.items():
|
||||
print(" chat_id = %-16s %s" % (cid, who))
|
||||
|
||||
|
||||
def cmd_test(mailbox):
|
||||
route = T.load_route(mailbox)
|
||||
if route is None:
|
||||
sys.exit("No route for mailbox %r (check enabled/token/chat_id in %s)"
|
||||
% (mailbox, T.CONF_PATH))
|
||||
print("Route: chats=%s audio=%s transcript=%s"
|
||||
% (route.chat_ids, route.send_audio, route.send_transcript))
|
||||
|
||||
audio = None
|
||||
for cand in ("/home/jp/asterisk-vm/test_vm.wav",):
|
||||
if os.path.exists(cand):
|
||||
audio = open(cand, "rb").read()
|
||||
break
|
||||
|
||||
fields = {"from": "Test Caller <07700900123>", "mailbox": mailbox or "test",
|
||||
"date": "now", "duration": "0:23", "msgnum": "1"}
|
||||
n = T.notify(
|
||||
mailbox, fields,
|
||||
"This is a test of the voicemail Telegram delivery. If you can read this "
|
||||
"and play the attached voice note, the route works.",
|
||||
["Call back requested"], ["07700900123"],
|
||||
"This is a test of the voicemail Telegram delivery. If you can read this "
|
||||
"and play the attached voice note, the route works.",
|
||||
audio_bytes=audio, audio_name="test.wav")
|
||||
print("notified %d chat(s)" % n)
|
||||
sys.exit(0 if n else 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) < 2 or sys.argv[1] not in ("ids", "test"):
|
||||
sys.exit(__doc__)
|
||||
if sys.argv[1] == "ids":
|
||||
cmd_ids()
|
||||
else:
|
||||
cmd_test(sys.argv[2] if len(sys.argv) > 2 else None)
|
||||
427
src/vm_web.py
Normal file
427
src/vm_web.py
Normal file
@ -0,0 +1,427 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Voicemail portal - FastAPI app.
|
||||
|
||||
Mailbox users log in with their existing voicemail mailbox number + PIN from
|
||||
/etc/asterisk/voicemail.conf, then can:
|
||||
* list voicemails with transcript, summary and tags
|
||||
* play or download the recording
|
||||
* mark read / delete (DB row, stored audio, and the spool file if present)
|
||||
* edit their own notification settings
|
||||
|
||||
Runs as the 'asterisk' user behind Apache. Sessions are signed cookies backed
|
||||
by a DB table so logout/expiry is enforced server-side.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import secrets
|
||||
import sys
|
||||
import time
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from fastapi import Cookie, FastAPI, Form, HTTPException, Request, Response
|
||||
from fastapi.responses import FileResponse, HTMLResponse, RedirectResponse
|
||||
from html import escape
|
||||
|
||||
import vm_auth
|
||||
import vm_store
|
||||
|
||||
SESSION_HOURS = int(os.environ.get("VM_SESSION_HOURS", "12"))
|
||||
COOKIE = "vm_session"
|
||||
BASE = os.environ.get("VM_BASE_PATH", "") # e.g. "/voicemail" if sub-pathed
|
||||
# Cookies are Secure by default (the app is served over HTTPS). Set
|
||||
# VM_INSECURE_COOKIE=1 only for local plain-HTTP testing.
|
||||
SECURE_COOKIE = os.environ.get("VM_INSECURE_COOKIE", "") not in ("1", "yes", "true")
|
||||
|
||||
app = FastAPI(title="Voicemail Portal", docs_url=None, redoc_url=None,
|
||||
openapi_url=None)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- sessions
|
||||
def new_session(mailbox):
|
||||
con = vm_store.connect()
|
||||
tok = secrets.token_urlsafe(32)
|
||||
now = int(time.time())
|
||||
con.execute("INSERT INTO sessions (token, mailbox, created_at, expires_at)"
|
||||
" VALUES (?,?,?,?)",
|
||||
(tok, str(mailbox), now, now + SESSION_HOURS * 3600))
|
||||
con.execute("DELETE FROM sessions WHERE expires_at < ?", (now,))
|
||||
con.commit()
|
||||
con.close()
|
||||
return tok
|
||||
|
||||
|
||||
def session_mailbox(token):
|
||||
if not token:
|
||||
return None
|
||||
con = vm_store.connect()
|
||||
r = con.execute("SELECT mailbox, expires_at FROM sessions WHERE token=?",
|
||||
(token,)).fetchone()
|
||||
con.close()
|
||||
if not r or r["expires_at"] < time.time():
|
||||
return None
|
||||
return r["mailbox"]
|
||||
|
||||
|
||||
def require(token):
|
||||
mb = session_mailbox(token)
|
||||
if not mb:
|
||||
raise HTTPException(status_code=303, detail="login",
|
||||
headers={"Location": BASE + "/login"})
|
||||
return mb
|
||||
|
||||
|
||||
# ------------------------------------------------------- brute-force lockout
|
||||
# Voicemail PINs are short (often 4 digits) and this app is internet-facing,
|
||||
# so failed logins are throttled per (mailbox, source IP). In-process state is
|
||||
# fine: the service is a single uvicorn worker.
|
||||
MAX_FAILS = int(os.environ.get("VM_MAX_FAILS", "5"))
|
||||
LOCK_MINUTES = int(os.environ.get("VM_LOCK_MINUTES", "15"))
|
||||
_fails = {} # (mailbox, ip) -> [count, first_ts]
|
||||
|
||||
|
||||
def _lock_key(mailbox, ip):
|
||||
return (str(mailbox).strip(), ip)
|
||||
|
||||
|
||||
def _lock_check(mailbox, ip):
|
||||
"""Return remaining lock time in minutes, or 0 if not locked."""
|
||||
e = _fails.get(_lock_key(mailbox, ip))
|
||||
if not e or e[0] < MAX_FAILS:
|
||||
return 0
|
||||
elapsed = time.time() - e[1]
|
||||
if elapsed > LOCK_MINUTES * 60:
|
||||
_fails.pop(_lock_key(mailbox, ip), None)
|
||||
return 0
|
||||
return max(1, int((LOCK_MINUTES * 60 - elapsed) // 60) + 1)
|
||||
|
||||
|
||||
def _lock_fail(mailbox, ip):
|
||||
k = _lock_key(mailbox, ip)
|
||||
e = _fails.get(k)
|
||||
now = time.time()
|
||||
if not e or now - e[1] > LOCK_MINUTES * 60:
|
||||
_fails[k] = [1, now]
|
||||
else:
|
||||
e[0] += 1
|
||||
|
||||
|
||||
def _lock_clear(mailbox, ip):
|
||||
_fails.pop(_lock_key(mailbox, ip), None)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- layout
|
||||
CSS = """
|
||||
*{box-sizing:border-box}
|
||||
body{margin:0;font:15px/1.55 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,
|
||||
Helvetica,Arial,sans-serif;background:#eef1f6;color:#1c2430}
|
||||
a{color:#2f6fed}
|
||||
header{background:linear-gradient(135deg,#2f6fed,#1b4bb8);color:#fff;
|
||||
padding:16px 22px;display:flex;align-items:center;gap:14px;flex-wrap:wrap}
|
||||
header h1{margin:0;font-size:18px;font-weight:700}
|
||||
header .sp{flex:1}
|
||||
header a{color:#dce6ff;text-decoration:none;font-size:14px;font-weight:600}
|
||||
header a:hover{color:#fff;text-decoration:underline}
|
||||
.wrap{max-width:880px;margin:22px auto;padding:0 14px}
|
||||
.card{background:#fff;border:1px solid #e3e9f5;border-radius:12px;
|
||||
padding:18px 20px;margin-bottom:14px;box-shadow:0 1px 4px rgba(20,30,50,.05)}
|
||||
.meta{font-size:13px;color:#6b7686;margin-bottom:6px}
|
||||
.who{font-weight:700;font-size:16px}
|
||||
.sum{background:#f9fbff;border:1px solid #e3e9f5;border-radius:8px;
|
||||
padding:12px 14px;margin:10px 0}
|
||||
.tag{display:inline-block;background:#eef3ff;color:#2f6fed;border-radius:999px;
|
||||
padding:3px 10px;font-size:12px;font-weight:600;margin:0 5px 5px 0}
|
||||
audio{width:100%;margin-top:10px}
|
||||
details{margin-top:8px}
|
||||
summary{cursor:pointer;color:#2f6fed;font-size:14px;font-weight:600}
|
||||
.tr{white-space:pre-wrap;color:#39424e;font-size:14px;margin-top:8px;
|
||||
background:#fafbfd;padding:12px;border-radius:8px;border:1px solid #eceff5}
|
||||
.row{display:flex;gap:8px;margin-top:12px;flex-wrap:wrap}
|
||||
button,.btn{font:600 14px/1 inherit;padding:9px 14px;border-radius:8px;
|
||||
border:1px solid #d4dcea;background:#fff;color:#2b3442;cursor:pointer}
|
||||
button:hover{background:#f4f7fd}
|
||||
.danger{border-color:#f0c8c8;color:#b93a3a}
|
||||
.danger:hover{background:#fdf3f3}
|
||||
.primary{background:#2f6fed;border-color:#2f6fed;color:#fff}
|
||||
.primary:hover{background:#2860d8}
|
||||
input[type=text],input[type=password],input[type=email]{width:100%;padding:10px 12px;
|
||||
border:1px solid #d4dcea;border-radius:8px;font:15px inherit;background:#fff}
|
||||
label{display:block;margin:12px 0 5px;font-weight:600;font-size:14px}
|
||||
.hint{font-size:12px;color:#7b8494;font-weight:400}
|
||||
.empty{text-align:center;color:#7b8494;padding:40px 10px}
|
||||
.unread{border-left:4px solid #2f6fed}
|
||||
.err{background:#fdecec;border:1px solid #f5c2c2;color:#a32c2c;padding:10px 12px;
|
||||
border-radius:8px;margin-bottom:12px;font-size:14px}
|
||||
.ok{background:#eaf7ee;border:1px solid #bfe3ca;color:#22683a;padding:10px 12px;
|
||||
border-radius:8px;margin-bottom:12px;font-size:14px}
|
||||
table{width:100%;border-collapse:collapse}
|
||||
td{padding:6px 0;vertical-align:top}
|
||||
"""
|
||||
|
||||
|
||||
def page(title, body, mailbox=None, name=None):
|
||||
nav = ""
|
||||
if mailbox:
|
||||
nav = ('<a href="%s/">Messages</a><a href="%s/settings">Settings</a>'
|
||||
'<a href="%s/logout">Log out</a>' % (BASE, BASE, BASE))
|
||||
return HTMLResponse("""<!DOCTYPE html><html><head><meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>%s</title><style>%s</style></head><body>
|
||||
<header><h1>✉ Voicemail</h1>%s<span class="sp"></span>%s</header>
|
||||
<div class="wrap">%s</div></body></html>""" % (
|
||||
escape(title), CSS,
|
||||
('<span style="font-size:14px;color:#dce6ff">%s · mailbox %s</span>'
|
||||
% (escape(name or ""), escape(mailbox))) if mailbox else "",
|
||||
nav, body))
|
||||
|
||||
|
||||
def fmt_time(ts):
|
||||
if not ts:
|
||||
return ""
|
||||
return time.strftime("%a %d %b %Y, %H:%M", time.localtime(ts))
|
||||
|
||||
|
||||
def fmt_dur(sec):
|
||||
if not sec:
|
||||
return ""
|
||||
return "%d:%02d" % (sec // 60, sec % 60)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------- routes
|
||||
@app.get("/login", response_class=HTMLResponse)
|
||||
def login_form(err: str = ""):
|
||||
e = '<div class="err">%s</div>' % escape(err) if err else ""
|
||||
return page("Log in", """%s<div class="card">
|
||||
<form method="post" action="%s/login">
|
||||
<label>Mailbox number</label>
|
||||
<input type="text" name="mailbox" inputmode="numeric" autocomplete="username" autofocus>
|
||||
<label>PIN <span class="hint">the same PIN you use on the phone</span></label>
|
||||
<input type="password" name="pin" inputmode="numeric" autocomplete="current-password">
|
||||
<div class="row"><button class="primary" type="submit">Log in</button></div>
|
||||
</form></div>""" % (e, BASE))
|
||||
|
||||
|
||||
@app.post("/login")
|
||||
def do_login(request: Request, mailbox: str = Form(...), pin: str = Form(...)):
|
||||
ip = request.client.host if request.client else "?"
|
||||
locked = _lock_check(mailbox, ip)
|
||||
if locked:
|
||||
return RedirectResponse(
|
||||
BASE + "/login?err=Too+many+failed+attempts.+Try+again+in+%d+minutes."
|
||||
% locked, status_code=303)
|
||||
|
||||
info = vm_auth.check_login(mailbox, pin)
|
||||
if not info:
|
||||
_lock_fail(mailbox, ip)
|
||||
time.sleep(1) # slow down brute force
|
||||
return RedirectResponse(BASE + "/login?err=Incorrect+mailbox+or+PIN",
|
||||
status_code=303)
|
||||
_lock_clear(mailbox, ip)
|
||||
tok = new_session(mailbox.strip())
|
||||
r = RedirectResponse(BASE + "/", status_code=303)
|
||||
r.set_cookie(COOKIE, tok, httponly=True, samesite="lax",
|
||||
secure=SECURE_COOKIE, max_age=SESSION_HOURS * 3600,
|
||||
path=BASE + "/")
|
||||
return r
|
||||
|
||||
|
||||
@app.get("/logout")
|
||||
def logout(vm_session: str = Cookie(default=None)):
|
||||
if vm_session:
|
||||
con = vm_store.connect()
|
||||
con.execute("DELETE FROM sessions WHERE token=?", (vm_session,))
|
||||
con.commit()
|
||||
con.close()
|
||||
r = RedirectResponse(BASE + "/login", status_code=303)
|
||||
r.delete_cookie(COOKIE, path=BASE + "/")
|
||||
return r
|
||||
|
||||
|
||||
@app.get("/", response_class=HTMLResponse)
|
||||
def index(vm_session: str = Cookie(default=None), msg: str = ""):
|
||||
mb = require(vm_session)
|
||||
boxes = vm_auth.parse_mailboxes()
|
||||
name = boxes.get(mb, {}).get("name", "")
|
||||
con = vm_store.connect()
|
||||
rows = con.execute(
|
||||
"SELECT * FROM messages WHERE mailbox=? ORDER BY origtime DESC, id DESC",
|
||||
(mb,)).fetchall()
|
||||
con.close()
|
||||
|
||||
banner = '<div class="ok">%s</div>' % escape(msg) if msg else ""
|
||||
if not rows:
|
||||
return page("Messages", banner + '<div class="card empty">'
|
||||
'No voicemails yet.<br><span class="hint">New messages appear '
|
||||
'here automatically once transcribed.</span></div>',
|
||||
mb, name)
|
||||
|
||||
out = [banner]
|
||||
for r in rows:
|
||||
tags = "".join('<span class="tag">%s</span>' % escape(t)
|
||||
for t in json.loads(r["intents"] or "[]"))
|
||||
nums = json.loads(r["numbers"] or "[]")
|
||||
callback = ""
|
||||
if nums:
|
||||
callback = ' · '.join(
|
||||
'<a href="tel:%s">%s</a>' % (escape("".join(
|
||||
ch for ch in n if ch.isdigit() or ch == "+")), escape(n))
|
||||
for n in nums)
|
||||
callback = '<div class="meta">📞 Callback: %s</div>' % callback
|
||||
|
||||
audio = ""
|
||||
if r["audio_sha"]:
|
||||
audio = ('<audio controls preload="none" src="%s/audio/%d"></audio>'
|
||||
% (BASE, r["id"]))
|
||||
|
||||
transcript = ""
|
||||
if r["transcript"]:
|
||||
transcript = ('<details><summary>Full transcript</summary>'
|
||||
'<div class="tr">%s</div></details>'
|
||||
% escape(r["transcript"]))
|
||||
|
||||
out.append("""<div class="card%s">
|
||||
<div class="who">%s</div>
|
||||
<div class="meta">%s%s</div>
|
||||
<div class="sum">%s</div>
|
||||
%s%s
|
||||
%s
|
||||
%s
|
||||
<div class="row">
|
||||
<form method="post" action="%s/read/%d"><button>%s</button></form>
|
||||
<a class="btn" href="%s/audio/%d?dl=1">Download</a>
|
||||
<form method="post" action="%s/delete/%d"
|
||||
onsubmit="return confirm('Delete this voicemail permanently?')">
|
||||
<button class="danger">Delete</button></form>
|
||||
</div></div>""" % (
|
||||
"" if r["is_read"] else " unread",
|
||||
escape(r["contact_name"] or r["callerid"] or "Unknown caller"),
|
||||
escape(fmt_time(r["origtime"])),
|
||||
(" · " + escape(fmt_dur(r["duration"]))) if r["duration"] else "",
|
||||
escape(r["summary"] or "(no speech detected)"),
|
||||
('<div style="margin-top:8px">%s</div>' % tags) if tags else "",
|
||||
callback, audio, transcript,
|
||||
BASE, r["id"], "Mark unread" if r["is_read"] else "Mark read",
|
||||
BASE, r["id"], BASE, r["id"]))
|
||||
|
||||
return page("Messages", "".join(out), mb, name)
|
||||
|
||||
|
||||
@app.get("/audio/{msg_id}")
|
||||
def audio(msg_id: int, dl: int = 0, vm_session: str = Cookie(default=None)):
|
||||
mb = require(vm_session)
|
||||
con = vm_store.connect()
|
||||
r = con.execute("SELECT * FROM messages WHERE id=? AND mailbox=?",
|
||||
(msg_id, mb)).fetchone()
|
||||
con.close()
|
||||
if not r or not r["audio_sha"]:
|
||||
raise HTTPException(404, "not found")
|
||||
p = vm_store.audio_path(r["audio_sha"], r["audio_ext"] or "wav")
|
||||
if not os.path.exists(p):
|
||||
raise HTTPException(404, "recording missing")
|
||||
fname = "voicemail-%s-%d.%s" % (mb, msg_id, r["audio_ext"] or "wav")
|
||||
return FileResponse(
|
||||
p, media_type="audio/wav",
|
||||
filename=fname if dl else None,
|
||||
headers={} if dl else {"Content-Disposition": 'inline; filename="%s"' % fname})
|
||||
|
||||
|
||||
@app.post("/read/{msg_id}")
|
||||
def toggle_read(msg_id: int, vm_session: str = Cookie(default=None)):
|
||||
mb = require(vm_session)
|
||||
con = vm_store.connect()
|
||||
con.execute("UPDATE messages SET is_read = 1 - is_read"
|
||||
" WHERE id=? AND mailbox=?", (msg_id, mb))
|
||||
con.commit()
|
||||
con.close()
|
||||
return RedirectResponse(BASE + "/", status_code=303)
|
||||
|
||||
|
||||
@app.post("/delete/{msg_id}")
|
||||
def delete(msg_id: int, vm_session: str = Cookie(default=None)):
|
||||
mb = require(vm_session)
|
||||
con = vm_store.connect()
|
||||
r = con.execute("SELECT * FROM messages WHERE id=? AND mailbox=?",
|
||||
(msg_id, mb)).fetchone()
|
||||
if not r:
|
||||
con.close()
|
||||
raise HTTPException(404, "not found")
|
||||
|
||||
# remove the stored audio only if no other row references it
|
||||
if r["audio_sha"]:
|
||||
others = con.execute("SELECT COUNT(*) c FROM messages"
|
||||
" WHERE audio_sha=? AND id<>?",
|
||||
(r["audio_sha"], msg_id)).fetchone()["c"]
|
||||
if not others:
|
||||
try:
|
||||
os.unlink(vm_store.audio_path(r["audio_sha"], r["audio_ext"] or "wav"))
|
||||
except OSError:
|
||||
pass
|
||||
# and the spool copy, when we know where it was
|
||||
if r["spool_path"]:
|
||||
base = os.path.splitext(r["spool_path"])[0]
|
||||
for ext in (".wav", ".WAV", ".gsm", ".txt", ".wav49"):
|
||||
try:
|
||||
os.unlink(base + ext)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
con.execute("DELETE FROM messages WHERE id=? AND mailbox=?", (msg_id, mb))
|
||||
con.commit()
|
||||
con.close()
|
||||
return RedirectResponse(BASE + "/?msg=Voicemail+deleted", status_code=303)
|
||||
|
||||
|
||||
@app.get("/settings", response_class=HTMLResponse)
|
||||
def settings_form(vm_session: str = Cookie(default=None), msg: str = ""):
|
||||
mb = require(vm_session)
|
||||
boxes = vm_auth.parse_mailboxes()
|
||||
info = boxes.get(mb, {})
|
||||
con = vm_store.connect()
|
||||
cur = vm_store.get_settings(con, mb)
|
||||
con.close()
|
||||
|
||||
rows = []
|
||||
for key, (default, label) in vm_store.USER_SETTINGS.items():
|
||||
val = cur.get(key, default)
|
||||
if default in ("yes", "no"):
|
||||
checked = " checked" if vm_store.truthy(val) else ""
|
||||
rows.append('<tr><td><label style="font-weight:400;margin:0">'
|
||||
'<input type="checkbox" name="%s" value="yes"%s> %s'
|
||||
'</label></td></tr>' % (key, checked, escape(label)))
|
||||
else:
|
||||
rows.append('<tr><td><label>%s</label>'
|
||||
'<input type="text" name="%s" value="%s"></td></tr>'
|
||||
% (escape(label), key, escape(val or "")))
|
||||
|
||||
banner = '<div class="ok">%s</div>' % escape(msg) if msg else ""
|
||||
return page("Settings", """%s<div class="card">
|
||||
<div class="meta">Notification settings for <b>%s</b> (mailbox %s).
|
||||
Phone PIN changes must still be made on the phone or by your administrator.</div>
|
||||
<form method="post" action="%s/settings"><table>%s</table>
|
||||
<div class="row"><button class="primary" type="submit">Save settings</button></div>
|
||||
</form></div>""" % (banner, escape(info.get("name", "")), escape(mb), BASE,
|
||||
"".join(rows)), mb, info.get("name", ""))
|
||||
|
||||
|
||||
@app.post("/settings")
|
||||
async def save_settings(request: Request, vm_session: str = Cookie(default=None)):
|
||||
mb = require(vm_session)
|
||||
form = await request.form()
|
||||
con = vm_store.connect()
|
||||
for key, (default, _label) in vm_store.USER_SETTINGS.items():
|
||||
if default in ("yes", "no"):
|
||||
vm_store.set_setting(con, mb, key, "yes" if form.get(key) else "no")
|
||||
else:
|
||||
vm_store.set_setting(con, mb, key, (form.get(key) or "").strip())
|
||||
con.close()
|
||||
return RedirectResponse(BASE + "/settings?msg=Settings+saved", status_code=303)
|
||||
|
||||
|
||||
@app.get("/healthz")
|
||||
def healthz():
|
||||
con = vm_store.connect()
|
||||
n = con.execute("SELECT COUNT(*) c FROM messages").fetchone()["c"]
|
||||
con.close()
|
||||
return {"ok": True, "messages": n}
|
||||
43
systemd/vm-portal.service
Normal file
43
systemd/vm-portal.service
Normal file
@ -0,0 +1,43 @@
|
||||
[Unit]
|
||||
Description=Voicemail portal (transcripts, playback, per-mailbox settings)
|
||||
Documentation=file:/opt/vm-transcribe/vm_web.py
|
||||
After=network.target
|
||||
Wants=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
# Runs as asterisk so it can read voicemail.conf and the spool.
|
||||
User=asterisk
|
||||
Group=asterisk
|
||||
WorkingDirectory=/opt/vm-transcribe
|
||||
|
||||
Environment=VM_DB=/var/lib/vm-transcribe/voicemail.db
|
||||
Environment=VM_AUDIO_DIR=/var/lib/vm-transcribe/audio
|
||||
Environment=VM_ASTERISK_CONF=/etc/asterisk/voicemail.conf
|
||||
Environment=VM_SESSION_HOURS=12
|
||||
Environment=PYTHONUNBUFFERED=1
|
||||
|
||||
ExecStart=/opt/vm-transcribe/venv/bin/python -m uvicorn vm_web:app \
|
||||
--host 127.0.0.1 --port 8099 \
|
||||
--proxy-headers --forwarded-allow-ips 127.0.0.1 \
|
||||
--log-level info
|
||||
|
||||
Restart=on-failure
|
||||
RestartSec=3
|
||||
|
||||
# --- hardening ---------------------------------------------------------
|
||||
NoNewPrivileges=yes
|
||||
PrivateTmp=yes
|
||||
ProtectSystem=full
|
||||
ProtectHome=yes
|
||||
ProtectKernelTunables=yes
|
||||
ProtectControlGroups=yes
|
||||
RestrictSUIDSGID=yes
|
||||
# Only these paths need to be writable.
|
||||
ReadWritePaths=/var/lib/vm-transcribe /var/log/asterisk
|
||||
# Loopback only; Apache is the only client.
|
||||
IPAddressAllow=localhost
|
||||
IPAddressDeny=any
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
20
tests/make_test_mail.py
Normal file
20
tests/make_test_mail.py
Normal file
@ -0,0 +1,20 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build a fake Asterisk voicemail notification (as sendmail -t would get it)."""
|
||||
import sys
|
||||
from email.message import EmailMessage
|
||||
|
||||
wav = sys.argv[1]
|
||||
to = sys.argv[2]
|
||||
|
||||
m = EmailMessage()
|
||||
m["From"] = "voicemail@txt3.net"
|
||||
m["To"] = to
|
||||
m["Subject"] = "New message 3 in mailbox 1001"
|
||||
m.set_content(
|
||||
"Dear Jamie:\n\n\tjust wanted to let you know you were just left a 0:37 long message "
|
||||
"(number 3)\nin mailbox 1001 from Dave Roberts <07941223856>, on Thu, 13 Aug 2026 "
|
||||
"07:12:00, so you might\nwant to check it when you get a chance. Thanks!\n\n"
|
||||
"\t\t\t\t--Asterisk\n")
|
||||
with open(wav, "rb") as fh:
|
||||
m.add_attachment(fh.read(), maintype="audio", subtype="x-wav", filename="msg0003.wav")
|
||||
sys.stdout.buffer.write(m.as_bytes())
|
||||
113
tests/test_contacts.py
Normal file
113
tests/test_contacts.py
Normal file
@ -0,0 +1,113 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Offline tests for vm_contacts: vCard/CSV parsing, digit matching, cache."""
|
||||
import os, sys, tempfile, json
|
||||
sys.path.insert(0, "/home/jp/asterisk-vm")
|
||||
|
||||
d = tempfile.mkdtemp()
|
||||
|
||||
VCF = """BEGIN:VCARD
|
||||
VERSION:3.0
|
||||
FN:Dave Roberts
|
||||
TEL;TYPE=CELL:+44 7941 223856
|
||||
END:VCARD
|
||||
BEGIN:VCARD
|
||||
VERSION:3.0
|
||||
FN:Alice Smith
|
||||
TEL;TYPE=WORK:020 7946 0018
|
||||
TEL;TYPE=CELL:07700 900123
|
||||
END:VCARD
|
||||
BEGIN:VCARD
|
||||
VERSION:3.0
|
||||
FN:No Number Person
|
||||
END:VCARD
|
||||
"""
|
||||
vcf = os.path.join(d, "c.vcf"); open(vcf, "w").write(VCF)
|
||||
|
||||
CSV = """Name,Given Name,Family Name,Phone 1 - Type,Phone 1 - Value
|
||||
Bob Jones,Bob,Jones,Mobile,+1 (555) 010-9876
|
||||
Carol White,Carol,White,Mobile,07123 456789 ::: 02012345678
|
||||
"""
|
||||
csvp = os.path.join(d, "c.csv"); open(csvp, "w").write(CSV)
|
||||
|
||||
cache = os.path.join(d, "cache.json")
|
||||
|
||||
def write_conf(path_val, backends="file"):
|
||||
c = os.path.join(d, "contacts.conf")
|
||||
open(c, "w").write(f"""[contacts]
|
||||
enabled = yes
|
||||
backends = {backends}
|
||||
cache_path = {cache}
|
||||
cache_ttl = 86400
|
||||
match_digits = 9
|
||||
|
||||
[file]
|
||||
path = {path_val}
|
||||
""")
|
||||
os.environ["VM_CONTACTS_CONF"] = c
|
||||
return c
|
||||
|
||||
import importlib
|
||||
import vm_contacts as C
|
||||
|
||||
def fresh(path_val, backends="file"):
|
||||
write_conf(path_val, backends)
|
||||
if os.path.exists(cache): os.unlink(cache)
|
||||
importlib.reload(C)
|
||||
return C
|
||||
|
||||
print("== extract_number")
|
||||
for s in ['Dave Roberts <07941223856>', '"Alice" <+447941223856>', '07700900123', 'unknown', '']:
|
||||
print(" %-30r -> %r" % (s, C.extract_number(s)))
|
||||
|
||||
print("\n== vCard lookup, various formats of the SAME number")
|
||||
c = fresh(vcf)
|
||||
for s in ["<07941223856>", "<+447941223856>", "<447941223856>", "Dave <7941223856>"]:
|
||||
print(" %-24s -> %r" % (s, c.resolve(s, log=lambda m: None)))
|
||||
|
||||
print("\n== vCard: second number on a multi-TEL contact")
|
||||
c = fresh(vcf)
|
||||
print(" Alice work 02079460018 ->", c.resolve("<02079460018>", log=lambda m: None))
|
||||
c = fresh(vcf)
|
||||
print(" Alice cell 07700900123 ->", c.resolve("<07700900123>", log=lambda m: None))
|
||||
|
||||
print("\n== unknown number -> None (and cached as a miss)")
|
||||
c = fresh(vcf)
|
||||
print(" ->", c.resolve("<07999999999>", log=lambda m: None))
|
||||
print(" cache contents:", json.load(open(cache)))
|
||||
|
||||
print("\n== CSV backend (Google CSV export format, ::: multi-value)")
|
||||
c = fresh(csvp)
|
||||
print(" Bob 5550109876 ->", c.resolve("<+15550109876>", log=lambda m: None))
|
||||
c = fresh(csvp)
|
||||
print(" Carol 07123456789 ->", c.resolve("<07123456789>", log=lambda m: None))
|
||||
c = fresh(csvp)
|
||||
print(" Carol 2nd num 02012345678 ->", c.resolve("<02012345678>", log=lambda m: None))
|
||||
|
||||
print("\n== disabled / missing file / no number")
|
||||
c = fresh(os.path.join(d, "nope.vcf"))
|
||||
print(" missing file ->", c.resolve("<07941223856>", log=lambda m: None))
|
||||
open(os.environ["VM_CONTACTS_CONF"], "a").write("\n")
|
||||
w = os.path.join(d, "off.conf")
|
||||
open(w, "w").write("[contacts]\nenabled = no\nbackends = file\n")
|
||||
os.environ["VM_CONTACTS_CONF"] = w; importlib.reload(C)
|
||||
print(" disabled ->", C.resolve("<07941223856>", log=lambda m: None))
|
||||
c = fresh(vcf)
|
||||
print(" empty callerid ->", c.resolve("", log=lambda m: None))
|
||||
|
||||
print("\n== carddav pointed at google must refuse app-password auth")
|
||||
w2 = os.path.join(d, "cd.conf")
|
||||
open(w2, "w").write(f"""[contacts]
|
||||
enabled = yes
|
||||
backends = carddav
|
||||
cache_path = {os.path.join(d,'c2.json')}
|
||||
match_digits = 9
|
||||
|
||||
[carddav]
|
||||
url = https://www.google.com/carddav/v1/principals/me/lists/default/
|
||||
username = me@gmail.com
|
||||
app_password = abcdefghijklmnop
|
||||
""")
|
||||
os.environ["VM_CONTACTS_CONF"] = w2; importlib.reload(C)
|
||||
msgs = []
|
||||
print(" ->", C.resolve("<07941223856>", log=msgs.append))
|
||||
for m in msgs: print(" [log]", m)
|
||||
67
tests/test_telegram.py
Normal file
67
tests/test_telegram.py
Normal file
@ -0,0 +1,67 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Offline tests for vm_telegram: config routing + caption building + opus."""
|
||||
import os, sys, tempfile
|
||||
sys.path.insert(0, "/home/jp/asterisk-vm")
|
||||
|
||||
CONF = """
|
||||
[telegram]
|
||||
enabled = yes
|
||||
token = 111:AAA
|
||||
default_chat_id = 999
|
||||
send_audio = yes
|
||||
send_transcript = yes
|
||||
timeout = 20
|
||||
|
||||
[mailbox:1001]
|
||||
chat_id = 123456789
|
||||
|
||||
[mailbox:1002]
|
||||
chat_id = 5551, 5552
|
||||
send_transcript = no
|
||||
|
||||
[mailbox:1003]
|
||||
chat_id = -1001234567890
|
||||
send_audio = no
|
||||
"""
|
||||
tf = tempfile.NamedTemporaryFile("w", suffix=".conf", delete=False)
|
||||
tf.write(CONF); tf.close()
|
||||
os.environ["VM_TG_CONF"] = tf.name
|
||||
import vm_telegram as T
|
||||
|
||||
print("== routing")
|
||||
for mb in ("1001", "1002", "1003", "1099", None):
|
||||
r = T.load_route(mb)
|
||||
if r is None:
|
||||
print(" mailbox %-5s -> no route" % mb)
|
||||
else:
|
||||
print(" mailbox %-5s -> chats=%s audio=%s transcript=%s"
|
||||
% (mb, r.chat_ids, r.send_audio, r.send_transcript))
|
||||
|
||||
print("\n== disabled master switch")
|
||||
open(tf.name, "w").write(CONF.replace("enabled = yes", "enabled = no"))
|
||||
print(" ->", T.load_route("1001"))
|
||||
print("== missing token")
|
||||
open(tf.name, "w").write(CONF.replace("token = 111:AAA", "token ="))
|
||||
print(" ->", T.load_route("1001"))
|
||||
open(tf.name, "w").write(CONF)
|
||||
|
||||
print("\n== caption")
|
||||
fields = {"from": "Dave Roberts <07941223856>", "mailbox": "1001",
|
||||
"date": "Thu, 13 Aug 2026 07:12:00", "duration": "0:37", "msgnum": "3"}
|
||||
cap = T.build_caption(fields,
|
||||
"Hi, this is Dave Roberts calling from Meridian Plumbing about the invoice. "
|
||||
"Could you please call me back as soon as possible on 07941-223856.",
|
||||
["Call back requested", "Urgent", "Payment / invoice"], ["07941-223856"])
|
||||
print(cap)
|
||||
print(" [caption length %d / %d]" % (len(cap), T.CAPTION_LIMIT))
|
||||
|
||||
print("\n== caption clipping (5000-char summary)")
|
||||
big = T.build_caption(fields, "word " * 1000, [], [])
|
||||
print(" length %d (limit %d) ok=%s" % (len(big), T.CAPTION_LIMIT, len(big) <= T.CAPTION_LIMIT))
|
||||
|
||||
print("\n== opus transcode")
|
||||
wav = open("/home/jp/asterisk-vm/test_vm.wav", "rb").read()
|
||||
ogg = T.to_voice_ogg(wav, ".wav")
|
||||
print(" wav %d bytes -> ogg %s bytes" % (len(wav), len(ogg) if ogg else None))
|
||||
print(" magic:", ogg[:4] if ogg else None)
|
||||
os.unlink(tf.name)
|
||||
Reference in New Issue
Block a user