mcspeak/CLAUDE.md

TTS MCP Server

Multi-engine text-to-speech server exposed via FastMCP 3.0 Streamable HTTP. Engines: Kokoro (ONNX), Piper (Wyoming/Docker), Orpheus (llama-server + SNAC).

Build & Run

make up        # build + start (docker compose)
make logs      # follow logs
make restart   # restart containers
make status    # show running containers + health

Entry & Exit Tones (Beep System)

Queued speech playback (speak()) is bookended by short alert tones. generate_audio() is unaffected (file-only, no playback).

Tone Positions

Position	When	Purpose
Entry tone	Immediately when speak() is called (before synthesis)	"I heard you" acknowledgement — covers synthesis latency
Exit tone	After speech, queue empty	"Over and out" — channel clear
Standby tone	After speech, more queued	"Standby" — more messages coming
Cancel tone	After cancel_speech() or MCP cancellation kills playback	"Nevermind" — speech was aborted

Available Tones

Name	Frequency	Duration	Inspired by
chirp	1800 Hz	~144 ms	Nextel iDEN Talk Permit Tone (TPT) — the 24/24/24/24/48 ms on/off pattern
apollo	2525 Hz	250 ms	NASA quindar intro (key-up) tone used during Apollo missions
roger	1400-1000 Hz	~100 ms	Classic CB radio descending two-tone roger beep
quindar-out	2475 Hz	250 ms	NASA quindar unkey tone (distinct frequency from intro)
standby	1000-1400 Hz	~60 ms	Ascending blip — inverse of roger, signals "more coming"
scratch	2000-300 Hz sweep + noise	~120 ms	Vinyl record scratch — needle yanked off the platter
reverse-roger	1000-1400 Hz	~100 ms	Ascending two-tone — mathematical inverse of roger beep

Configuration

TTS_ENTRY_TONE=chirp        # before speech (chirp, apollo, none, or /path/to/custom.wav)
TTS_EXIT_TONE=roger          # after speech, queue empty (roger, quindar-out, none, or path)
TTS_CANCEL_TONE=scratch      # on cancel (scratch, reverse-roger, none, or /path/to/custom.wav)
TTS_SHUTDOWN_TIMEOUT=30      # max seconds to wait for current speech on container stop

The standby tone is always the built-in ascending blip. It plays instead of the exit tone when more items are queued.

Tones are generated programmatically at startup (48kHz, 16-bit PCM, -3 dB headroom) in tones.py using numpy. No bundled audio assets.

Voice Identity (Project-Aware Voices)

When multiple Claude Code sessions connect simultaneously, voice identity gives each project a distinct voice via round-robin assignment from a curated English voice pool.

How It Works

1. Client calls speak() or generate_audio() without specifying voice=
2. Project is identified via the project tool parameter, or falls back to MCP Roots (list_roots() with 2s timeout)
3. The next unused voice is assigned from the interleaved pool (alternating gender and accent for maximum contrast)
4. Assignment is persisted to /data/voice-assignments.json — survives server restarts

Explicit voice= parameter always overrides auto-assignment. Voice pools are cached for 5 minutes (picks up blacklist/engine changes).

Note: MCP Roots require stateful Streamable HTTP. With stateless_http=True (current default), roots will timeout — the project parameter is the primary identification method.

Configuration

TTS_VOICE_IDENTITY=true                          # Enable project-aware voice assignment
TTS_VOICE_IDENTITY_PREFIXES=af_,am_,bf_,bm_,ef_,em_  # English voice prefixes
TTS_VOICE_IDENTITY_EXCLUDE=af_nicole              # Available explicitly, excluded from auto-assign (whispery)
TTS_VOICE_IDENTITY_FILE=/data/voice-assignments.json  # Persist across restarts
TTS_ANNOUNCE_PROJECT=false                        # Prefix speech with project name

Pool Interleave Order

Voices are interleaved for perceptual diversity: American female → British male → European female → American male → British female → European male. First 6 projects get maximally distinct voices.

Files

• voice_identity.py — Pool filtering, interleaving, round-robin assignment, JSON persistence

Media Ducking

When speak() is called, external audio streams (Firefox, Spotify, etc.) are automatically faded down via PulseAudio before speech begins, then faded back up after the exit tone. This creates a radio-broadcast-interruption effect where the entry tone crossfades over the fading media.

Audio Timeline

0ms     ─ Media starts fading (vol 100%)
150ms   ─ Entry tone starts! Media at ~70%    ← crossfade overlap
294ms   ─ Entry tone ends, media at ~40%
500ms   ─ Media at 0% (ducked)
        ─ [synthesis + playback — media stays silent]
        ─ Exit tone plays (roger/standby)
        ─ Media fades back in over 1000ms

The duck fires in speak() (before synthesis), the unduck fires in the consumer (after exit/cancel tone). Media stays ducked across multiple queued items — only unducks when the queue is empty or on cancel/shutdown.

Configuration

TTS_DUCK_MEDIA=true             # Enable/disable (on by default)
TTS_DUCK_FADE_OUT_MS=500        # Fade-out duration (ms)
TTS_DUCK_FADE_IN_MS=1000        # Fade-in duration (slower = natural)

Docker Requirements

The PulseAudio compatibility socket must be mounted in the container:

volumes:
  - /run/user/1000/pulse:/run/user/1000/pulse

If the socket is missing or pactl fails, ducking silently no-ops — TTS still works normally.

Why pactl, not pulsectl-asyncio? PipeWire's PulseAudio compat layer silently drops sink_input_volume_set operations from pulsectl's native protocol connection, while pactl (using libpulse C library) works reliably. The pulseaudio-utils package is installed in the container for this reason.

Non-fatal Design

All pactl errors are caught and logged. If PulseAudio is unavailable (no socket, wrong permissions, pactl not installed), duck/unduck become no-ops. This ensures TTS never breaks due to ducking failures.

Files

• media_duck.py — MediaDucker class (async pactl subprocess volume control with stepped fades)

Architecture

• server.py — FastMCP lifespan, tool definitions, engine setup
• queue.py — Producer-consumer speech queue with priority tiers and outcome tracking
• tones.py — Tone WAV generator (entry/exit/standby)
• media_duck.py — Async PulseAudio volume control for media ducking
• audio.py — WAV writing and pw-play async wrapper
• settings.py — Pydantic settings from env vars (prefix: TTS_)
• engines/ — TTSEngine implementations (kokoro, piper, orpheus)

speak() Progress Lifecycle

speak() blocks until playback finishes, reporting progress throughout. Clients should call it in parallel with other tools rather than blocking on it alone — the return value is informational (speech_id, duration) and never needed for subsequent reasoning.

Progress notifications

A background ticker emits MCP progress notifications every ~0.5 seconds during playback. Each notification includes progress (0-100), total (100), and a message string. MCP-aware clients render this as a live progress bar.

Progress	Phase	Message
5%	Entry tone played — audible "I heard you"	Entry tone
30%	Synthesis complete	Synthesizing...
35%	Enqueued for playback	Playing audio — you can continue working
35-99%	Playing — progress tracks elapsed time vs expected duration	Playing audio — you can continue working
100%	Playback finished	Playback complete

The entry tone fires before synthesis, covering the 1-3s latency gap. The progress message intentionally says "you can continue working" to signal to LLM clients that they don't need to wait.

Concurrency guidance

speak() blocks for 10-60s for typical text, but audio plays through physical speakers — the return value doesn't feed into subsequent reasoning. The server instructions and tool docstring explicitly tell clients to call speak in parallel with other tools in the same message. Tool annotations provide structured signals:

• openWorldHint=True — this tool interacts with the physical world (speakers)
• idempotentHint=True — safe to retry without side effects beyond replaying audio

speech_status(speech_id) is available for checking outcomes after the fact. Possible statuses: completed, playing, queued, unknown.

Outcomes are stored in a bounded ring (last 100 items) — old entries are evicted automatically.

Chunked Synthesis (Pipelined Playback)

Long texts (>= 20 words with sentence boundaries) are automatically split into sentence-sized chunks and pipelined — synthesis of chunk N+1 overlaps playback of chunk N.

Single-shot:  [---12s synthesis---][----------48s playback----------]  First audio at T+12s
Chunked:      [~1s synth][play 1][~1s synth][play 2][~1s synth][play 3]...  First audio at T+1s

Why no gaps

Kokoro synthesizes ~4x realtime on CPU, so synthesis always outpaces playback. pw-play adds ~10ms startup per chunk — imperceptible. Slower engines (Orpheus) may have brief gaps but still beat the full-synthesis-first approach.

Short text bypass

Texts under 20 words or without sentence boundaries (.!? followed by whitespace) take the single-shot path — zero overhead, identical to pre-chunking behavior.

Tone behavior

• Entry tone plays once at the start (before first chunk synthesis)
• Exit/standby tones are suppressed between chunks (suppress_exit_tone flag on _WorkItem)
• Final chunk plays the normal exit tone (roger) or standby tone

Cancellation in chunked mode

• Explicit cancel_speech(speech_id) — cancels the returned speech_id (the final chunk). Already-playing earlier chunks finish naturally.
• MCP disconnect — the synthesis loop stops (remaining chunks aren't synthesized). Already-enqueued chunks play through.

Progress lifecycle (chunked)

Progress	Phase	Message
5%	Entry tone played	Entry tone
5-30%	Synthesis progress across all chunks	Synthesizing chunk N/M...
35%	All chunks enqueued	Playing audio — you can continue working
35-99%	Playing — tracks elapsed time vs total duration	Playing audio — you can continue working
100%	Final chunk playback finished	Playback complete

Files

• server.py — split_text(), _speak_single(), _speak_chunked(), _await_with_progress()
• queue.py — suppress_exit_tone field on _WorkItem

Cancellation

Speech can be cancelled two ways:

1. Explicit cancel_speech(speech_id) — kills pw-play immediately, plays the cancel tone, and the consumer moves to the next item.
2. MCP cancellation — if a client disconnects or sends notifications/cancelled, the speak() handler is cancelled but playback continues. The consumer finishes the current audio naturally. This is intentional — audio is already synthesized and playing, so cutting it mid-sentence would be jarring.

When a currently-playing item is explicitly cancelled, pw-play is killed immediately and the cancel tone plays. When a queued item is cancelled, it's removed from the queue without ever playing. The consumer continues to the next item in both cases.

Graceful Shutdown

On docker compose down or make restart, the server lets the currently-playing speech finish before stopping — no more mid-sentence cutoffs.

How it works: SIGTERM triggers a two-phase shutdown that shares Docker's wall-clock budget:

1. Phase 1 — Handler drain (3s fixed): Uvicorn cancels in-flight speak() handlers. The handlers re-raise CancelledError without touching the consumer — audio keeps playing.
2. Phase 2 — Queue drain (shutdown_timeout, default 30s): Lifespan finalizer calls queue.stop(), which waits for the consumer to finish the current audio. If the timeout expires, it force-cancels.

Docker's stop_grace_period must exceed the total: 3s + shutdown_timeout + 5s safety margin (default 38s). If the max expected audio length exceeds shutdown_timeout, increase both TTS_SHUTDOWN_TIMEOUT and stop_grace_period.

Entry tone timing: The Nextel chirp fires immediately when speak() is called (before synthesis), acting as an audible "I heard you" that covers the 1-3s synthesis latency. Exit/standby tones still play from the consumer (they depend on queue state after playback).

Key Design Decisions

• Speech queue is serialized (one playback at a time) but synthesis is parallel
• speak() blocks until playback finishes with live progress (5% → 30% → 35-99% → 100%)
• Progress uses a background ticker task, NOT asyncio.wait_for polling (see below)
• Entry tone is awaited in speak() before synthesis — covers latency gap
• Explicit cancel_speech() kills pw-play + plays cancel tone; MCP disconnect lets playback finish
• Consumer directly awaits play_audio(); cancel() targets the consumer task with _item_cancelled flag
• speak()'s CancelledError handler does NOT cancel the consumer — only explicit cancel does
• Graceful shutdown is two-phase: 3s handler drain (fixed) + shutdown_timeout queue drain, both sequential within Docker's stop_grace_period
• Tones are non-fatal: if pw-play fails on a tone, speech still plays
• Orpheus uses llama-server (not Ollama) for 15x throughput via continuous batching
• SNAC decoder is lazy-loaded on first Orpheus call to reduce idle memory

Python 3.13 asyncio.wait_for pitfall

Do NOT use asyncio.wait_for(future, timeout) in a polling loop to track progress. In Python 3.13, wait_for cancels its inner task on timeout. When the inner task is awaiting the same asyncio.Future that the consumer will resolve, repeated cancel/re-await cycles cause a stale CancelledError to propagate to the consumer task — killing pw-play mid-playback. Instead, use a background asyncio.create_task ticker for progress and directly await the future for completion.