# McSpeak

Multi-engine text-to-speech MCP server exposed via FastMCP 3.0 Streamable HTTP. Engines: Kokoro (ONNX), Piper (Wyoming/Docker), Orpheus (llama-server + SNAC).

## Build & Run

```bash
make up        # build + start (docker compose)
make logs      # follow logs
make restart   # restart containers
make status    # show running containers + health
```

## Entry & Exit Tones (Beep System)

Queued speech playback (`speak()`) is bookended by short alert tones. `generate_audio()` is unaffected (file-only, no playback).

### Tone Positions

| Position | When | Purpose |
|----------|------|---------|
| **Entry tone** | Immediately when `speak()` is called (before synthesis) | "I heard you" acknowledgement — covers synthesis latency |
| **Exit tone** | After speech, queue empty | "Over and out" — channel clear |
| **Standby tone** | After speech, more queued | "Standby" — more messages coming |
| **Cancel tone** | After `cancel_speech()` or MCP cancellation kills playback | "Nevermind" — speech was aborted |

### Available Tones

| Name | Frequency | Duration | Inspired by |
|------|-----------|----------|-------------|
| `chirp` | 1800 Hz | ~144 ms | Nextel iDEN Talk Permit Tone (TPT) — the 24/24/24/24/48 ms on/off pattern |
| `apollo` | 2525 Hz | 250 ms | NASA quindar intro (key-up) tone used during Apollo missions |
| `roger` | 1400-1000 Hz | ~100 ms | Classic CB radio descending two-tone roger beep |
| `quindar-out` | 2475 Hz | 250 ms | NASA quindar unkey tone (distinct frequency from intro) |
| `standby` | 1000-1400 Hz | ~60 ms | Ascending blip — inverse of roger, signals "more coming" |
| `scratch` | 2000-300 Hz sweep + noise | ~120 ms | Vinyl record scratch — needle yanked off the platter |
| `reverse-roger` | 1000-1400 Hz | ~100 ms | Ascending two-tone — mathematical inverse of roger beep |

**Pink Floyd voice kit (current defaults).** The bookends now follow a telephone theme:

| Name | Used as | Inspired by |
|------|---------|-------------|
| `heartbeat` | speak entry | "Speak to Me" heartbeat (DSotM opener) woven with warm MF telephone tones |
| `soft-chord` | speak exit | Mellow resolving A3+E4 dyad, pure sines, ear-friendly on repetition |
| `mf-dial` | listen start | The **genuine Young Lust R1 MF operator dial** — KP, 0-4-4-1-8-3-1, ST (the 44 is the UK country code), real Bell-System MF pairs |
| `machine` | listen end | "Welcome to the Machine" pulsing VCS3 throb — a warm "connected / got it" |
| `mf-listen` / `mf-done` | listen (alt) | Gentler stacked-fifths swell / resolving D-major (previous listen defaults) |
| `call-waiting` | over ongoing speech | Brief C6 double-blip mixed over the current message when another project queues |

There are also softer "natural" tones (`bell-soft`, `chime-tube`, `water-drop`, `soft-pulse`, `hmm-up`, …) in `tones.py` for custom use.

### Configuration

```env
TTS_ENTRY_TONE=heartbeat        # before speech (heartbeat, chirp, apollo, none, or /path/to/custom.wav)
TTS_EXIT_TONE=soft-chord         # after speech, queue empty (soft-chord, roger, quindar-out, none, or path)
TTS_CANCEL_TONE=scratch          # on cancel (scratch, reverse-roger, none, or /path/to/custom.wav)
TTS_CALL_WAITING_TONE=call-waiting  # mixed over current speech when a DIFFERENT project queues (once/turn)
TTS_LISTEN_START_TONE=mf-dial    # played once the mic is live (see listen())
TTS_LISTEN_END_TONE=machine      # played after recording stops
TTS_SHUTDOWN_TIMEOUT=30          # max seconds to wait for current speech on container stop
```

The standby tone is always the built-in ascending blip. It plays instead of the exit tone when more items are queued.

## Secretary Announcements & Call-Waiting

When several projects speak concurrently, the queue behaves like a **secretary**: each message plays whole and in order (never interleaved — see the streaming-utterance model below), and a project is announced by name when the speaker changes or returns after a lull.

- The announcement is a short `"<project>."` preamble synthesized in a **reserved secretary voice** (`TTS_SECRETARY_VOICE`, default `bf_emma`) — always via Kokoro so it sounds identical regardless of the speaking engine. That voice is excluded from the project auto-assignment pool so no project ever sounds like the secretary.
- The play-or-skip decision is made at **play time** in the consumer (`queue.py:_should_announce`), not enqueue time, because urgent reordering means the real speaker order isn't final until then.
- **Call-waiting**: when a *different* project's message joins the queue while one is playing, a brief `call-waiting` blip is mixed over the current audio (a second `pw-play` stream — PipeWire mixes it). Fires **at most once per playing turn** (`_call_waiting_fired`, reset when a new utterance starts) so a burst of queued messages never spams the listener.

```env
TTS_ANNOUNCE_MODE=secretary            # secretary | always | off (legacy TTS_ANNOUNCE_PROJECT=true → always)
TTS_REINTRODUCE_AFTER_SECONDS=120      # same project after this much silence gets re-introduced
TTS_SECRETARY_VOICE=bf_emma            # reserved; excluded from project auto-assignment
```

## listen() — Voice Conversations

`listen()` captures the host mic (`pw-record`), transcribes via Parakeet on the gpu.supported.systems gateway, and returns the text. Pair it with `speak()` for turn-taking: speak a question (let it finish), then `listen()` for the reply — **sequentially, never in parallel**, or the mic records the TTS.

- **Defaults are conversation-first**: `wait_for_silence=True` (stop when the person stops), `duration_seconds=30` cap, `vad_aggressiveness=3`, `silence_threshold_ms=2200`. The aggressiveness/threshold defaults were tuned live to stop brief background transients from ending the turn before the real reply.
- **No first-word clip**: the "go" tone is played *after* the mic is live (a `warmup_ms` lead, default 150ms). The beep bleeds harmlessly into the head of the recording — VAD treats a pure tone as non-speech and Parakeet ignores it. See `audio.py:record_audio_until_silence`.
- Empty transcription (`text == ""`) or a gateway timeout means re-prompt rather than proceed; the recording is saved under `/tmp/mcspeak/` and can be retried with `transcribe()`.

Tones are generated programmatically at startup (48kHz, 16-bit PCM, -3 dB headroom) in `tones.py` using numpy. No bundled audio assets.

## Voice Identity (Project-Aware Voices)

When multiple Claude Code sessions connect simultaneously, voice identity gives each project a distinct voice via round-robin assignment from a curated English voice pool.

### How It Works

1. Client calls `speak()` or `generate_audio()` without specifying `voice=`
2. Project is identified via the `project` tool parameter, or falls back to MCP Roots (`list_roots()` with 2s timeout)
3. The next unused voice is assigned from the interleaved pool (alternating gender and accent for maximum contrast)
4. Assignment is persisted to `/data/voice-assignments.json` — survives server restarts

Explicit `voice=` parameter always overrides auto-assignment. Voice pools are cached for 5 minutes (picks up blacklist/engine changes).

**Note:** MCP Roots require stateful Streamable HTTP. With `stateless_http=True` (current default), roots will timeout — the `project` parameter is the primary identification method.

### Configuration

```env
TTS_VOICE_IDENTITY=true                          # Enable project-aware voice assignment
TTS_VOICE_IDENTITY_PREFIXES=af_,am_,bf_,bm_,ef_,em_  # English voice prefixes
TTS_VOICE_IDENTITY_EXCLUDE=af_nicole              # Available explicitly, excluded from auto-assign (whispery)
TTS_VOICE_IDENTITY_FILE=/data/voice-assignments.json  # Persist across restarts
TTS_ANNOUNCE_PROJECT=false                        # Prefix speech with project name
```

### Pool Interleave Order

Voices are interleaved for perceptual diversity: American female → British male → European female → American male → British female → European male. First 6 projects get maximally distinct voices.

### Files

- `voice_identity.py` — Pool filtering, interleaving, round-robin assignment, JSON persistence

## Media Ducking

When `speak()` is called, external audio streams (Firefox, Spotify, etc.) are automatically faded down via PulseAudio before speech begins, then faded back up after the exit tone. This creates a radio-broadcast-interruption effect where the entry tone crossfades over the fading media.

### Audio Timeline

```
0ms     ─ Media starts fading (vol 100%)
150ms   ─ Entry tone starts! Media at ~70%    ← crossfade overlap
294ms   ─ Entry tone ends, media at ~40%
500ms   ─ Media at 0% (ducked)
        ─ [synthesis + playback — media stays silent]
        ─ Exit tone plays (roger/standby)
        ─ Media fades back in over 1000ms
```

The duck fires in `speak()` (before synthesis), the unduck fires in the consumer (after exit/cancel tone). Media stays ducked across multiple queued items — only unducks when the queue is empty or on cancel/shutdown.

### Configuration

```env
TTS_DUCK_MEDIA=true             # Enable/disable (on by default)
TTS_DUCK_FADE_OUT_MS=500        # Fade-out duration (ms)
TTS_DUCK_FADE_IN_MS=1000        # Fade-in duration (slower = natural)
```

### Docker Requirements

The PulseAudio compatibility socket must be mounted in the container:

```yaml
volumes:
  - /run/user/1000/pulse:/run/user/1000/pulse
```

If the socket is missing or pactl fails, ducking silently no-ops — TTS still works normally.

**Why pactl, not pulsectl-asyncio?** PipeWire's PulseAudio compat layer silently drops `sink_input_volume_set` operations from pulsectl's native protocol connection, while `pactl` (using libpulse C library) works reliably. The `pulseaudio-utils` package is installed in the container for this reason.

### Non-fatal Design

All pactl errors are caught and logged. If PulseAudio is unavailable (no socket, wrong permissions, pactl not installed), duck/unduck become no-ops. This ensures TTS never breaks due to ducking failures.

### Files

- `media_duck.py` — `MediaDucker` class (async pactl subprocess volume control with stepped fades)

## Architecture

- `server.py` — FastMCP lifespan, tool definitions, engine setup
- `queue.py` — Producer-consumer queue of streaming **utterances** (one per `speak()`): priority tiers, secretary announcements + reserved voice, call-waiting, outcome tracking. Synthesized WAVs are reaped after playback (no /tmp leak).
- `tones.py` — Tone WAV generator (speak + listen bookends, call-waiting, natural set)
- `media_duck.py` — Async PulseAudio volume control for media ducking
- `audio.py` — WAV writing and `pw-play` async wrapper
- `settings.py` — Pydantic settings from env vars (prefix: `TTS_`)
- `engines/` — TTSEngine implementations (kokoro, piper, orpheus)

## `speak()` Progress Lifecycle

`speak()` blocks until playback finishes, reporting progress throughout. **Clients should call it in parallel with other tools** rather than blocking on it alone — the return value is informational (speech_id, duration) and never needed for subsequent reasoning.

### Progress notifications

A background ticker emits MCP progress notifications every ~0.5 seconds during playback. Each notification includes `progress` (0-100), `total` (100), and a `message` string. MCP-aware clients render this as a live progress bar.

| Progress | Phase | Message |
|----------|-------|---------|
| 5% | Entry tone played — audible "I heard you" | `Entry tone` |
| 30% | Synthesis complete | `Synthesizing...` |
| 35% | Enqueued for playback | `Playing audio — you can continue working` |
| 35-99% | Playing — progress tracks elapsed time vs expected duration | `Playing audio — you can continue working` |
| 100% | Playback finished | `Playback complete` |

The entry tone fires before synthesis, covering the 1-3s latency gap. The progress message intentionally says "you can continue working" to signal to LLM clients that they don't need to wait.

### Concurrency guidance

`speak()` blocks for 10-60s for typical text, but audio plays through physical speakers — the return value doesn't feed into subsequent reasoning. The server instructions and tool docstring explicitly tell clients to **call speak in parallel with other tools** in the same message. Tool annotations provide structured signals:

- `openWorldHint=True` — this tool interacts with the physical world (speakers)
- `idempotentHint=True` — safe to retry without side effects beyond replaying audio

`speech_status(speech_id)` is available for checking outcomes after the fact. Possible statuses: `completed`, `playing`, `queued`, `unknown`.

Outcomes are stored in a bounded ring (last 100 items) — old entries are evicted automatically.

## Chunked Synthesis (Pipelined Playback)

Long texts (>= 20 words with sentence boundaries) are automatically split into sentence-sized chunks and pipelined — synthesis of chunk N+1 overlaps playback of chunk N.

```
Single-shot:  [---12s synthesis---][----------48s playback----------]  First audio at T+12s
Chunked:      [~1s synth][play 1][~1s synth][play 2][~1s synth][play 3]...  First audio at T+1s
```

### Why no gaps

Kokoro synthesizes ~4x realtime on CPU, so synthesis always outpaces playback. `pw-play` adds ~10ms startup per chunk — imperceptible. Slower engines (Orpheus) may have brief gaps but still beat the full-synthesis-first approach.

### Short text bypass

Texts under 20 words or without sentence boundaries (`.!?` followed by whitespace) take the single-shot path — zero overhead, identical to pre-chunking behavior.

### Tone behavior & per-message coherence

Each `speak()` call is **one queue utterance** (not one queue item per chunk). The utterance reserves its ordering slot up front and streams its synthesized chunks in through an internal channel closed by an `_END` sentinel; the consumer stays locked to it from entry tone to exit tone. So when several projects speak at once, a message plays **whole and in order** — chunks never interleave with another project's audio (the old per-chunk `suppress_exit_tone`/`_WorkItem` model is gone).

- Entry tone plays once at the start (before first chunk synthesis)
- Exit/standby tone plays once, at the very end of the utterance
- Pipelining is preserved: synthesis of chunk N+1 overlaps playback of chunk N, but nothing else can be pulled until this utterance finishes

### Cancellation in chunked mode

- **Explicit `cancel_speech(speech_id)`** — one speech_id now covers the whole utterance; cancelling it flags an abort (synchronously), reaps un-played chunk WAVs, and plays the cancel tone.
- **MCP disconnect** — the `speak()` handler is cancelled; its `finally` closes the utterance channel so the consumer drains the chunks it already has and finishes cleanly.

### Progress lifecycle (chunked)

| Progress | Phase | Message |
|----------|-------|---------|
| 5% | Entry tone played | `Entry tone` |
| 5-30% | Synthesis progress across all chunks | `Synthesizing chunk N/M...` |
| 35% | All chunks enqueued | `Playing audio — you can continue working` |
| 35-99% | Playing — tracks elapsed time vs total duration | `Playing audio — you can continue working` |
| 100% | Final chunk playback finished | `Playback complete` |

### Files

- `server.py` — `split_text()`, unified `_speak()` (short + chunked share one path), status-aware `_await_with_progress()`
- `queue.py` — `_Utterance` (streaming chunk channel + `_END` sentinel), `create_utterance()`, `_play_utterance()`, `_should_announce()`, call-waiting

## Cancellation

Speech can be cancelled two ways:

1. **Explicit `cancel_speech(speech_id)`** — kills pw-play immediately, plays the cancel tone, and the consumer moves to the next item.
2. **MCP cancellation** — if a client disconnects or sends `notifications/cancelled`, the `speak()` handler is cancelled but playback continues. The consumer finishes the current audio naturally. This is intentional — audio is already synthesized and playing, so cutting it mid-sentence would be jarring.

When a currently-playing item is explicitly cancelled, pw-play is killed immediately and the cancel tone plays. When a queued item is cancelled, it's removed from the queue without ever playing. The consumer continues to the next item in both cases.

## Graceful Shutdown

On `docker compose down`, `make up`, or `make restart`, the server **stops accepting new speech and plays out what's already queued** before stopping — not just the currently-playing message. No dropped messages on a redeploy.

**How it works:** SIGTERM triggers a two-phase shutdown that shares Docker's wall-clock budget:

1. **Phase 1 — Handler drain (3s fixed):** Uvicorn cancels in-flight `speak()` handlers. The handlers re-raise `CancelledError` without touching the consumer — already-enqueued audio keeps playing.
2. **Phase 2 — Queue drain (`shutdown_timeout`, default 30s):** Lifespan finalizer calls `queue.stop()`, which (a) sets `_accepting=False` so new `speak()` calls are rejected immediately, then (b) lets the consumer keep playing queued utterances normally until the queue is empty and nothing's playing, then (c) stops the consumer. Whatever doesn't fit the budget is dropped and reaped. The `_accepting` flag (reject new) is deliberately separate from `_stopped` (consumer exit) so the drain window sits between them.

Docker's `stop_grace_period` must exceed the total: `3s + shutdown_timeout + safety margin` (default 38s). Because the whole queue now drains (not just the current message), a deep queue can hit the budget — raise both `TTS_SHUTDOWN_TIMEOUT` and `stop_grace_period` if you want longer drains.

**Entry tone timing:** The Nextel chirp fires immediately when `speak()` is called (before synthesis), acting as an audible "I heard you" that covers the 1-3s synthesis latency. Exit/standby tones still play from the consumer (they depend on queue state after playback).

## Key Design Decisions

- Speech queue is serialized (one playback at a time) but synthesis is parallel; each `speak()` is one **streaming utterance** so concurrent projects never interleave
- Secretary behavior: a project is announced by name on speaker-change / after a lull, in a **reserved voice** excluded from the project pool; a different project queuing mid-playback fires a **once-per-turn** call-waiting blip mixed over the current audio
- Synthesized WAVs are reaped after playback (and on cancel/shutdown) so `/tmp/mcspeak` doesn't grow unbounded
- `speak()` blocks until playback finishes with **status-aware** progress — a queued item reports "waiting in line", not a false "playing" percentage
- Progress uses a background ticker task, NOT `asyncio.wait_for` polling (see below)
- Entry tone is awaited in `speak()` before synthesis — covers latency gap
- Explicit `cancel_speech()` kills pw-play + plays cancel tone; MCP disconnect lets playback finish
- Consumer directly awaits `play_audio()`; `cancel()` targets the consumer task with `_item_cancelled` flag
- `speak()`'s CancelledError handler does NOT cancel the consumer — only explicit cancel does
- Graceful shutdown is two-phase: 3s handler drain (fixed) + `shutdown_timeout` queue drain, both sequential within Docker's `stop_grace_period`
- Tones are non-fatal: if `pw-play` fails on a tone, speech still plays
- Orpheus uses llama-server (not Ollama) for 15x throughput via continuous batching
- SNAC decoder is lazy-loaded on first Orpheus call to reduce idle memory

### Python 3.13 asyncio.wait_for pitfall

**Do NOT use `asyncio.wait_for(future, timeout)` in a polling loop to track progress.** In Python 3.13, `wait_for` cancels its inner task on timeout. When the inner task is awaiting the same `asyncio.Future` that the consumer will resolve, repeated cancel/re-await cycles cause a stale `CancelledError` to propagate to the consumer task — killing pw-play mid-playback. Instead, use a background `asyncio.create_task` ticker for progress and directly `await` the future for completion.