asterisk-voicemail/docs/INSTALL.md

# Installation

Written against Debian 12 + Asterisk 20 + Apache 2.4 + Postfix + MySQL/MariaDB,
running under Virtualmin. Adjust paths if your layout differs.

Everything installs to `/opt/vm-transcribe` with audio in
`/var/lib/vm-transcribe`. Nothing is installed into system Python.

> **Frontend lives in a separate repo.** This backend repo ships a *synced
> copy* under `frontend/`. Build the real app in `/home/jp/Work/voicemail-ui`
> and rsync the result to the web root (see §6). Keep the two in sync with
> `rsync -a /home/jp/Work/voicemail-ui/ frontend/` then commit.

---

## 0. Prerequisites

```bash
# Asterisk voicemail must use file-based storage (the default), not ODBC/IMAP
grep -E '^(odbcstorage|imapserver)' /etc/asterisk/voicemail.conf   # expect nothing

# tools
sudo apt install ffmpeg sox python3-venv          # sox optional (gsm)
sudo apt install default-mysql-server             # or mariadb-server
which certbot                                     # for the portal's TLS
node --version && npm --version                   # Node 18+ for the frontend

# an MTA on localhost:25
sudo ss -ltnp | grep ':25 '
```

**Check disk space before you start.** A full `/var` makes Postfix reject all
mail with `452 4.3.1 Insufficient system storage`, which looks like a bug in
this pipeline but is not:

```bash
df -h /var
```

---

## 1. MySQL (storage + contacts + CDR)

The backend shares the Asterisk `asterisk` database. Create a least-privilege
user and a credentials file:

```bash
sudo mysql -e "CREATE USER IF NOT EXISTS 'asterisk'@'localhost' IDENTIFIED BY 'PICK_A_STRONG_PASSWORD';"
sudo mysql -e "GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,INDEX,ALTER ON asterisk.* TO 'asterisk'@'localhost';"
sudo mysql -e "FLUSH PRIVILEGES;"
```

Write `/opt/vm-transcribe/db_secret` (mode `640 root:root`):

```bash
sudo install -o root -g root -m 640 /dev/null /opt/vm-transcribe/db_secret
sudo tee /opt/vm-transcribe/db_secret >/dev/null <<'EOF'
MYSQL_HOST=localhost
MYSQL_USER=asterisk
MYSQL_PASSWORD=PICK_A_STRONG_PASSWORD
MYSQL_DB=asterisk
EOF
```

`vm_store` uses MySQL whenever this file (or `VM_MYSQL_*` env vars) exists;
otherwise it falls back to SQLite. The schema is created on first connect.

(Optional) To load call records into the same DB, wire Asterisk CDR via
`cdr_adaptive_odbc` + `res_odbc` + the MariaDB ODBC driver, then run
`src/vm_backfill_cdr.py` to import `Master.csv` history.

---

## 2. Core pipeline (mailcmd + venv)

The `install.sh` script builds the venv, installs `faster-whisper`, deploys the
mailcmd scripts, seeds `telegram.conf`, pre-caches the whisper model **as the
asterisk user**, and rewrites `voicemail.conf`'s `mailcmd=`.

> `install.sh` reads sources from `/home/jp/asterisk-vm` (the deploy path in
> AGENTS.md). Either symlink/clobber that path to this repo, or edit the `SRC=`
> line at the top of `scripts/install.sh` before running.

```bash
sudo scripts/install.sh
```

Verify the wiring:

```bash
grep '^mailcmd' /etc/asterisk/voicemail.conf
# mailcmd=/opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_mailcmd.py
```

### Test it before trusting it

Build a fake notification and pipe it through **as the `asterisk` user** —
testing as yourself hides permission problems:

```bash
cd /home/jp/Work/asterisk-voicemail
python3 tests/make_test_mail.py /path/to/some.wav you@example.com > /tmp/t.eml
sudo chmod 644 /tmp/t.eml
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
     /opt/vm-transcribe/vm_mailcmd.py < /tmp/t.eml
sudo tail -5 /var/log/asterisk/vm_mailcmd.log
```

You should see `transcribed … chars` then `sent enriched notification to …`.
Confirm the mail actually left:

```bash
sudo grep "to=<you@example.com>" /var/log/mail.log | tail -1   # status=sent
```

Also verify the fail-safe — a message with **no** audio must relay unchanged
rather than erroring:

```bash
printf 'From: a@b\nTo: you@example.com\nSubject: no audio\n\nbody\n' > /tmp/n.eml
sudo chmod 644 /tmp/n.eml
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
     /opt/vm-transcribe/vm_mailcmd.py < /tmp/n.eml
# log: "no audio attachment; relaying original"
```

---

## 3. Telegram (optional)

1. Create a bot: message [@BotFather](https://t.me/BotFather) → `/newbot` → copy the token.
2. Put it in `/opt/vm-transcribe/telegram.conf` (`sudo`, mode 640 root:asterisk).
3. **Each recipient must message the bot once** (`/start`) — Telegram forbids
   bots from initiating conversations.
4. Discover chat IDs and test:

```bash
sudo /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py ids
sudo /opt/vm-transcribe/venv/bin/python3 /opt/vm-transcribe/vm_tg_setup.py test 1001
```

The `test` subcommand sends a real DM with a playable voice note and exits
non-zero on failure. See [CONFIGURATION.md](CONFIGURATION.md#telegramconf) for
per-mailbox routing.

---

## 4. Contacts (MySQL)

Contacts live in the MySQL `contacts` table — there is no file/Google/CardDAV
backend anymore. Load your address book from a vCard/CSV export:

```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
     /opt/vm-transcribe/vm_import_contacts.py /path/to/contacts.vcf
```

Confirm lookup works:

```bash
sudo -u asterisk /opt/vm-transcribe/venv/bin/python3 \
     /opt/vm-transcribe/vm_contacts.py '<07700900123>'
```

---

## 5. Backend services

Install both units (API + legacy portal/audio):

```bash
sudo install -d -o asterisk -g asterisk -m 750 \
     /var/lib/vm-transcribe /var/lib/vm-transcribe/audio
sudo /opt/vm-transcribe/venv/bin/pip install fastapi 'uvicorn[standard]' python-multipart pymysql

# API unit reads MySQL creds from api.env
sudo install -o root -g root -m 640 /opt/vm-transcribe/db_secret /opt/vm-transcribe/api.env
sudo install -o root -g root -m 644 systemd/vm-api.service /etc/systemd/system/
sudo install -o root -g root -m 644 systemd/vm-portal.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now vm-api vm-portal
sudo systemctl is-active vm-api vm-portal

curl -s http://127.0.0.1:8098/api/healthz      # {"ok":true,...}
curl -s http://127.0.0.1:8099/healthz          # {"ok":true,"messages":N}
```

The services run as `asterisk` (so they can read `voicemail.conf` and the spool)
and bind **loopback only** — `IPAddressAllow=localhost` means Apache is the
only possible client.

---

## 6. React frontend (build + deploy)

The interactive UI is a React SPA built in `/home/jp/Work/voicemail-ui`:

```bash
cd /home/jp/Work/voicemail-ui
npm install
npm run build                       # outputs dist/
```

Deploy the **`dist/` output** (not the repo root) to the web root — the root
`index.html` is the Vite *dev* entry (`<script src="/src/main.tsx">`) and would
render a blank page in production:

```bash
sudo rsync -a --delete dist/ /home/txt3/domains/vm.txt3.net/public_html/
```

Keep this repo's copy in sync for documentation/commit purposes:

```bash
rsync -a /home/jp/Work/voicemail-ui/ frontend/      # then git commit under frontend/
```

---

## 7. Apache vhost + TLS

**Order matters.** Installing a vhost that references a cert which does not yet
exist breaks `apache2ctl configtest`, which blocks reloads for *every* site on
the box. So: DNS → cert → full vhost.

### 7a. DNS

```bash
dig +short vm.txt3.net A       # must return your ORIGIN ip
```

Behind Cloudflare, set the record to **DNS only (grey cloud)** first, or the
HTTP-01 challenge is intercepted by the edge.

### 7b. Minimal :80 vhost, so certbot can answer the challenge

```bash
sudo install -d -o www-data -g www-data -m 755 /var/www/vm.txt3.net/public_html
sudo cp apache/vm.txt3.net-step1.conf /etc/apache2/sites-available/vm.txt3.net.conf
# edit ServerName / paths / IPs to match your host
sudo a2ensite vm.txt3.net
sudo apache2ctl configtest && sudo systemctl reload apache2
```

### 7c. Issue the certificate

```bash
sudo certbot certonly --webroot -w /var/www/vm.txt3.net/public_html \
     -d vm.txt3.net --cert-name vm.txt3.net
```

### 7d. Full vhost — HTTPS + reverse proxy

```bash
sudo a2enmod proxy proxy_http headers rewrite ssl
sudo cp apache/vm.txt3.net.conf /etc/apache2/sites-available/vm.txt3.net.conf
# edit ServerName, cert paths, IPs, DocumentRoot
sudo install -d /var/www/vm.txt3.net/public_html/.well-known/acme-challenge
sudo apache2ctl configtest && sudo systemctl reload apache2
```

The shipped `apache/vm.txt3.net.conf` already proxies `/api/` → :8098,
`/audio/` → :8099, serves the React `dist` as `DocumentRoot`, and adds an SPA
fallback (`RewriteRule ^ /index.html`).

### 7e. Verify through the real hostname

Not localhost — the whole point is to exercise Apache, TLS and the proxy:

```bash
R="--resolve vm.txt3.net:443:ORIGIN.IP"
curl -s $R https://vm.txt3.net/api/healthz
curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.txt3.net/login
curl -s $R -D- -o /dev/null https://vm.txt3.net/login | grep -iE 'strict-transport|content-security|x-frame'
# SPA fallback
curl -s $R -o /dev/null -w '%{http_code}\n' https://vm.txt3.net/some/route
```

Behind a CDN, `curl https://host/` tests the *CDN*, not your origin. Always
`--resolve` to the origin IP when verifying a change.

---

## 8. Backfill existing voicemails

```bash
V=/opt/vm-transcribe/venv/bin/python3
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --dry-run          # preview
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py --limit 5          # trial
sudo -u asterisk $V /opt/vm-transcribe/vm_import.py                    # all
sudo -u asterisk $V /opt/vm-transcribe/vm_backfill_cdr.py              # CDR history
```

Idempotent — safe to re-run; it resumes rather than duplicating. Expect a
`no_speech` count: 44-byte WAVs are hung-up calls with no audio, not failures.

Reckon on ~1.5 s per message plus transcription time; 157 messages took 3m40s
on 4 cores.

---

## 9. Log in

Browse to `https://vm.txt3.net/` and log in with a **mailbox number and its
existing voicemail PIN** from `voicemail.conf`. No new passwords are created.

---

## Uninstall

```bash
sudo systemctl disable --now vm-api vm-portal
sudo rm /etc/systemd/system/vm-api.service /etc/systemd/system/vm-portal.service
sudo systemctl daemon-reload
sudo a2dissite vm.txt3.net && sudo systemctl reload apache2
# restore the original mailcmd
sudo cp /etc/asterisk/voicemail.conf.bak-<timestamp> /etc/asterisk/voicemail.conf
sudo asterisk -rx 'voicemail reload'
sudo rm -rf /opt/vm-transcribe          # keep /var/lib/vm-transcribe for the data
# MySQL: DROP USER 'asterisk'@'localhost';  (optionally DROP DATABASE asterisk;)
```