mcspeak/CLAUDE.md

TTS MCP Server

Multi-engine text-to-speech server exposed via FastMCP 3.0 Streamable HTTP. Engines: Kokoro (ONNX), Piper (Wyoming/Docker), Orpheus (llama-server + SNAC).

Build & Run

make up        # build + start (docker compose)
make logs      # follow logs
make restart   # restart containers
make status    # show running containers + health

Entry & Exit Tones (Beep System)

Queued speech playback (speak()) is bookended by short alert tones. generate_audio() is unaffected (file-only, no playback).

Tone Positions

Position	When	Purpose
Entry tone	Immediately when speak() is called (before synthesis)	"I heard you" acknowledgement — covers synthesis latency
Exit tone	After speech, queue empty	"Over and out" — channel clear
Standby tone	After speech, more queued	"Standby" — more messages coming
Cancel tone	After cancel_speech() or MCP cancellation kills playback	"Nevermind" — speech was aborted

Available Tones

Name	Frequency	Duration	Inspired by
chirp	1800 Hz	~144 ms	Nextel iDEN Talk Permit Tone (TPT) — the 24/24/24/24/48 ms on/off pattern
apollo	2525 Hz	250 ms	NASA quindar intro (key-up) tone used during Apollo missions
roger	1400-1000 Hz	~100 ms	Classic CB radio descending two-tone roger beep
quindar-out	2475 Hz	250 ms	NASA quindar unkey tone (distinct frequency from intro)
standby	1000-1400 Hz	~60 ms	Ascending blip — inverse of roger, signals "more coming"
scratch	2000-300 Hz sweep + noise	~120 ms	Vinyl record scratch — needle yanked off the platter
reverse-roger	1000-1400 Hz	~100 ms	Ascending two-tone — mathematical inverse of roger beep

Configuration

TTS_ENTRY_TONE=chirp        # before speech (chirp, apollo, none, or /path/to/custom.wav)
TTS_EXIT_TONE=roger          # after speech, queue empty (roger, quindar-out, none, or path)
TTS_CANCEL_TONE=scratch      # on cancel (scratch, reverse-roger, none, or /path/to/custom.wav)
TTS_SHUTDOWN_TIMEOUT=30      # max seconds to wait for current speech on container stop

The standby tone is always the built-in ascending blip. It plays instead of the exit tone when more items are queued.

Tones are generated programmatically at startup (48kHz, 16-bit PCM, -3 dB headroom) in tones.py using numpy. No bundled audio assets.

Voice Identity (Project-Aware Voices)

When multiple Claude Code sessions connect simultaneously, voice identity gives each project a distinct voice via round-robin assignment from a curated English voice pool.

How It Works

1. Client calls speak() or generate_audio() without specifying voice=
2. Project is identified via the project tool parameter, or falls back to MCP Roots (list_roots() with 2s timeout)
3. The next unused voice is assigned from the interleaved pool (alternating gender and accent for maximum contrast)
4. Assignment is persisted to /data/voice-assignments.json — survives server restarts

Explicit voice= parameter always overrides auto-assignment. Voice pools are cached for 5 minutes (picks up blacklist/engine changes).

Note: MCP Roots require stateful Streamable HTTP. With stateless_http=True (current default), roots will timeout — the project parameter is the primary identification method.

Configuration

TTS_VOICE_IDENTITY=true                          # Enable project-aware voice assignment
TTS_VOICE_IDENTITY_PREFIXES=af_,am_,bf_,bm_,ef_,em_  # English voice prefixes
TTS_VOICE_IDENTITY_EXCLUDE=af_nicole              # Available explicitly, excluded from auto-assign (whispery)
TTS_VOICE_IDENTITY_FILE=/data/voice-assignments.json  # Persist across restarts
TTS_ANNOUNCE_PROJECT=false                        # Prefix speech with project name

Pool Interleave Order

Voices are interleaved for perceptual diversity: American female → British male → European female → American male → British female → European male. First 6 projects get maximally distinct voices.

Files

• voice_identity.py — Pool filtering, interleaving, round-robin assignment, JSON persistence

Architecture

• server.py — FastMCP lifespan, tool definitions, engine setup
• queue.py — Producer-consumer speech queue with priority tiers and outcome tracking
• tones.py — Tone WAV generator (entry/exit/standby)
• audio.py — WAV writing and pw-play async wrapper
• settings.py — Pydantic settings from env vars (prefix: TTS_)
• engines/ — TTSEngine implementations (kokoro, piper, orpheus)

speak() Progress Lifecycle

speak() blocks until playback finishes, reporting progress throughout. Each progress notification includes a message field that LLM clients can read to understand the current phase.

Progress	Phase	Message
5%	Entry tone played — audible "I heard you"	Entry tone
30%	Synthesis complete	Synthesizing...
35%	Enqueued for playback	Playing audio — you can continue working
35-99%	Playing — progress tracks elapsed time vs expected duration	Playing audio — you can continue working
100%	Playback finished	Playback complete

MCP-aware clients see a live progress bar. The entry tone fires before synthesis, covering the 1-3s latency gap.

Concurrency guidance

The speak() call completes when playback finishes (10-60s for typical text), but the audio plays through physical speakers — the return value doesn't feed into subsequent reasoning. LLM clients like Claude Code can call speak() alongside other tools in the same message. The server instructions and tool docstring explicitly encourage this. Tool annotations (openWorldHint=True, idempotentHint=True) provide structured signals to clients that support them.

speech_status(speech_id) is still available for checking outcomes after the fact. Possible statuses: completed, playing, queued, unknown.

Outcomes are stored in a bounded ring (last 100 items) — old entries are evicted automatically.

Chunked Synthesis (Pipelined Playback)

Long texts (>= 20 words with sentence boundaries) are automatically split into sentence-sized chunks and pipelined — synthesis of chunk N+1 overlaps playback of chunk N.

Single-shot:  [---12s synthesis---][----------48s playback----------]  First audio at T+12s
Chunked:      [~1s synth][play 1][~1s synth][play 2][~1s synth][play 3]...  First audio at T+1s

Why no gaps

Kokoro synthesizes ~4x realtime on CPU, so synthesis always outpaces playback. pw-play adds ~10ms startup per chunk — imperceptible. Slower engines (Orpheus) may have brief gaps but still beat the full-synthesis-first approach.

Short text bypass

Texts under 20 words or without sentence boundaries (.!? followed by whitespace) take the single-shot path — zero overhead, identical to pre-chunking behavior.

Tone behavior

• Entry tone plays once at the start (before first chunk synthesis)
• Exit/standby tones are suppressed between chunks (suppress_exit_tone flag on _WorkItem)
• Final chunk plays the normal exit tone (roger) or standby tone

Cancellation in chunked mode

• Explicit cancel_speech(speech_id) — cancels the returned speech_id (the final chunk). Already-playing earlier chunks finish naturally.
• MCP disconnect — the synthesis loop stops (remaining chunks aren't synthesized). Already-enqueued chunks play through.

Progress lifecycle (chunked)

Progress	Phase	Message
5%	Entry tone played	Entry tone
5-30%	Synthesis progress across all chunks	Synthesizing chunk N/M...
35%	All chunks enqueued	Playing audio — you can continue working
35-99%	Playing — tracks elapsed time vs total duration	Playing audio — you can continue working
100%	Final chunk playback finished	Playback complete

Files

• server.py — split_text(), _speak_single(), _speak_chunked(), _await_with_progress()
• queue.py — suppress_exit_tone field on _WorkItem

Cancellation

Speech can be cancelled two ways:

1. Explicit cancel_speech(speech_id) — kills pw-play immediately, plays the cancel tone, and the consumer moves to the next item.
2. MCP cancellation — if a client disconnects or sends notifications/cancelled, the speak() handler is cancelled but playback continues. The consumer finishes the current audio naturally. This is intentional — audio is already synthesized and playing, so cutting it mid-sentence would be jarring.

When a currently-playing item is explicitly cancelled, pw-play is killed immediately and the cancel tone plays. When a queued item is cancelled, it's removed from the queue without ever playing. The consumer continues to the next item in both cases.

Graceful Shutdown

On docker compose down or make restart, the server lets the currently-playing speech finish before stopping — no more mid-sentence cutoffs.

How it works: SIGTERM triggers a two-phase shutdown that shares Docker's wall-clock budget:

1. Phase 1 — Handler drain (3s fixed): Uvicorn cancels in-flight speak() handlers. The handlers re-raise CancelledError without touching the consumer — audio keeps playing.
2. Phase 2 — Queue drain (shutdown_timeout, default 30s): Lifespan finalizer calls queue.stop(), which waits for the consumer to finish the current audio. If the timeout expires, it force-cancels.

Docker's stop_grace_period must exceed the total: 3s + shutdown_timeout + 5s safety margin (default 38s). If the max expected audio length exceeds shutdown_timeout, increase both TTS_SHUTDOWN_TIMEOUT and stop_grace_period.

Entry tone timing: The Nextel chirp fires immediately when speak() is called (before synthesis), acting as an audible "I heard you" that covers the 1-3s synthesis latency. Exit/standby tones still play from the consumer (they depend on queue state after playback).

Key Design Decisions

• Speech queue is serialized (one playback at a time) but synthesis is parallel
• speak() blocks until playback finishes with live progress (5% → 30% → 35-99% → 100%)
• Progress uses a background ticker task, NOT asyncio.wait_for polling (see below)
• Entry tone is awaited in speak() before synthesis — covers latency gap
• Explicit cancel_speech() kills pw-play + plays cancel tone; MCP disconnect lets playback finish
• Consumer directly awaits play_audio(); cancel() targets the consumer task with _item_cancelled flag
• speak()'s CancelledError handler does NOT cancel the consumer — only explicit cancel does
• Graceful shutdown is two-phase: 3s handler drain (fixed) + shutdown_timeout queue drain, both sequential within Docker's stop_grace_period
• Tones are non-fatal: if pw-play fails on a tone, speech still plays
• Orpheus uses llama-server (not Ollama) for 15x throughput via continuous batching
• SNAC decoder is lazy-loaded on first Orpheus call to reduce idle memory

Python 3.13 asyncio.wait_for pitfall

Do NOT use asyncio.wait_for(future, timeout) in a polling loop to track progress. In Python 3.13, wait_for cancels its inner task on timeout. When the inner task is awaiting the same asyncio.Future that the consumer will resolve, repeated cancel/re-await cycles cause a stale CancelledError to propagate to the consumer task — killing pw-play mid-playback. Instead, use a background asyncio.create_task ticker for progress and directly await the future for completion.