mcspeak/CLAUDE.md

McSpeak

Multi-engine text-to-speech MCP server exposed via FastMCP 3.0 Streamable HTTP. Engines: Kokoro (ONNX), Piper (Wyoming/Docker), Orpheus (llama-server + SNAC).

Build & Run

make up        # build + start (docker compose)
make logs      # follow logs
make restart   # restart containers
make status    # show running containers + health

Entry & Exit Tones (Beep System)

Queued speech playback (speak()) is bookended by short alert tones. generate_audio() is unaffected (file-only, no playback).

Tone Positions

Position	When	Purpose
Entry tone	Immediately when speak() is called (before synthesis)	"I heard you" acknowledgement — covers synthesis latency
Exit tone	After speech, queue empty	"Over and out" — channel clear
Standby tone	After speech, more queued	"Standby" — more messages coming
Cancel tone	After cancel_speech() or MCP cancellation kills playback	"Nevermind" — speech was aborted

Available Tones

Name	Frequency	Duration	Inspired by
chirp	1800 Hz	~144 ms	Nextel iDEN Talk Permit Tone (TPT) — the 24/24/24/24/48 ms on/off pattern
apollo	2525 Hz	250 ms	NASA quindar intro (key-up) tone used during Apollo missions
roger	1400-1000 Hz	~100 ms	Classic CB radio descending two-tone roger beep
quindar-out	2475 Hz	250 ms	NASA quindar unkey tone (distinct frequency from intro)
standby	1000-1400 Hz	~60 ms	Ascending blip — inverse of roger, signals "more coming"
scratch	2000-300 Hz sweep + noise	~120 ms	Vinyl record scratch — needle yanked off the platter
reverse-roger	1000-1400 Hz	~100 ms	Ascending two-tone — mathematical inverse of roger beep

Pink Floyd voice kit (current defaults). The bookends now follow a telephone theme:

Name	Used as	Inspired by
heartbeat	speak entry	"Speak to Me" heartbeat (DSotM opener) woven with warm MF telephone tones
soft-chord	speak exit	Mellow resolving A3+E4 dyad, pure sines, ear-friendly on repetition
mf-dial	listen start	The genuine Young Lust R1 MF operator dial — KP, 0-4-4-1-8-3-1, ST (the 44 is the UK country code), real Bell-System MF pairs
machine	listen end	"Welcome to the Machine" pulsing VCS3 throb — a warm "connected / got it"
mf-listen / mf-done	listen (alt)	Gentler stacked-fifths swell / resolving D-major (previous listen defaults)
call-waiting	over ongoing speech	Brief C6 double-blip mixed over the current message when another project queues

There are also softer "natural" tones (bell-soft, chime-tube, water-drop, soft-pulse, hmm-up, …) in tones.py for custom use.

Configuration

TTS_ENTRY_TONE=heartbeat        # before speech (heartbeat, chirp, apollo, none, or /path/to/custom.wav)
TTS_EXIT_TONE=soft-chord         # after speech, queue empty (soft-chord, roger, quindar-out, none, or path)
TTS_CANCEL_TONE=scratch          # on cancel (scratch, reverse-roger, none, or /path/to/custom.wav)
TTS_CALL_WAITING_TONE=call-waiting  # mixed over current speech when a DIFFERENT project queues (once/turn)
TTS_LISTEN_START_TONE=mf-dial    # played once the mic is live (see listen())
TTS_LISTEN_END_TONE=machine      # played after recording stops
TTS_SHUTDOWN_TIMEOUT=30          # max seconds to wait for current speech on container stop

The standby tone is always the built-in ascending blip. It plays instead of the exit tone when more items are queued.

Secretary Announcements & Call-Waiting

When several projects speak concurrently, the queue behaves like a secretary: each message plays whole and in order (never interleaved — see the streaming-utterance model below), and a project is announced by name when the speaker changes or returns after a lull.

• The announcement is a short "<project>." preamble synthesized in a reserved secretary voice (TTS_SECRETARY_VOICE, default bf_emma) — always via Kokoro so it sounds identical regardless of the speaking engine. That voice is excluded from the project auto-assignment pool so no project ever sounds like the secretary.
• The play-or-skip decision is made at play time in the consumer (queue.py:_should_announce), not enqueue time, because urgent reordering means the real speaker order isn't final until then.
• Call-waiting: when a different project's message joins the queue while one is playing, a brief call-waiting blip is mixed over the current audio (a second pw-play stream — PipeWire mixes it). Fires at most once per playing turn (_call_waiting_fired, reset when a new utterance starts) so a burst of queued messages never spams the listener.

TTS_ANNOUNCE_MODE=secretary            # secretary | always | off (legacy TTS_ANNOUNCE_PROJECT=true → always)
TTS_REINTRODUCE_AFTER_SECONDS=120      # same project after this much silence gets re-introduced
TTS_SECRETARY_VOICE=bf_emma            # reserved; excluded from project auto-assignment

listen() — Voice Conversations

listen() captures the host mic (pw-record), transcribes via Parakeet on the gpu.supported.systems gateway, and returns the text. Pair it with speak() for turn-taking: speak a question (let it finish), then listen() for the reply — sequentially, never in parallel, or the mic records the TTS.

• Defaults are conversation-first: wait_for_silence=True (stop when the person stops), duration_seconds=30 cap, vad_aggressiveness=3, silence_threshold_ms=2200. The aggressiveness/threshold defaults were tuned live to stop brief background transients from ending the turn before the real reply.
• No first-word clip: the "go" tone is played after the mic is live (a warmup_ms lead, default 150ms). The beep bleeds harmlessly into the head of the recording — VAD treats a pure tone as non-speech and Parakeet ignores it. See audio.py:record_audio_until_silence.
• Empty transcription (text == "") or a gateway timeout means re-prompt rather than proceed; the recording is saved under /tmp/mcspeak/ and can be retried with transcribe().

Tones are generated programmatically at startup (48kHz, 16-bit PCM, -3 dB headroom) in tones.py using numpy. No bundled audio assets.

Voice Identity (Project-Aware Voices)

When multiple Claude Code sessions connect simultaneously, voice identity gives each project a distinct voice via round-robin assignment from a curated English voice pool.

How It Works

1. Client calls speak() or generate_audio() without specifying voice=
2. Project is identified via the project tool parameter, or falls back to MCP Roots (list_roots() with 2s timeout)
3. The next unused voice is assigned from the interleaved pool (alternating gender and accent for maximum contrast)
4. Assignment is persisted to /data/voice-assignments.json — survives server restarts

Explicit voice= parameter always overrides auto-assignment. Voice pools are cached for 5 minutes (picks up blacklist/engine changes).

Note: MCP Roots require stateful Streamable HTTP. With stateless_http=True (current default), roots will timeout — the project parameter is the primary identification method.

Configuration

TTS_VOICE_IDENTITY=true                          # Enable project-aware voice assignment
TTS_VOICE_IDENTITY_PREFIXES=af_,am_,bf_,bm_,ef_,em_  # English voice prefixes
TTS_VOICE_IDENTITY_EXCLUDE=af_nicole              # Available explicitly, excluded from auto-assign (whispery)
TTS_VOICE_IDENTITY_FILE=/data/voice-assignments.json  # Persist across restarts
TTS_ANNOUNCE_PROJECT=false                        # Prefix speech with project name

Pool Interleave Order

Voices are interleaved for perceptual diversity: American female → British male → European female → American male → British female → European male. First 6 projects get maximally distinct voices.

Files

• voice_identity.py — Pool filtering, interleaving, round-robin assignment, JSON persistence

Media Ducking

When speak() is called, external audio streams (Firefox, Spotify, etc.) are automatically faded down via PulseAudio before speech begins, then faded back up after the exit tone. This creates a radio-broadcast-interruption effect where the entry tone crossfades over the fading media.

Audio Timeline

0ms     ─ Media starts fading (vol 100%)
150ms   ─ Entry tone starts! Media at ~70%    ← crossfade overlap
294ms   ─ Entry tone ends, media at ~40%
500ms   ─ Media at 0% (ducked)
        ─ [synthesis + playback — media stays silent]
        ─ Exit tone plays (roger/standby)
        ─ Media fades back in over 1000ms

The duck fires in speak() (before synthesis), the unduck fires in the consumer (after exit/cancel tone). Media stays ducked across multiple queued items — only unducks when the queue is empty or on cancel/shutdown.

Configuration

TTS_DUCK_MEDIA=true             # Enable/disable (on by default)
TTS_DUCK_FADE_OUT_MS=500        # Fade-out duration (ms)
TTS_DUCK_FADE_IN_MS=1000        # Fade-in duration (slower = natural)

Docker Requirements

The PulseAudio compatibility socket must be mounted in the container:

volumes:
  - /run/user/1000/pulse:/run/user/1000/pulse

If the socket is missing or pactl fails, ducking silently no-ops — TTS still works normally.

Why pactl, not pulsectl-asyncio? PipeWire's PulseAudio compat layer silently drops sink_input_volume_set operations from pulsectl's native protocol connection, while pactl (using libpulse C library) works reliably. The pulseaudio-utils package is installed in the container for this reason.

Non-fatal Design

All pactl errors are caught and logged. If PulseAudio is unavailable (no socket, wrong permissions, pactl not installed), duck/unduck become no-ops. This ensures TTS never breaks due to ducking failures.

Files

• media_duck.py — MediaDucker class (async pactl subprocess volume control with stepped fades)

Architecture

• server.py — FastMCP lifespan, tool definitions, engine setup
• queue.py — Producer-consumer queue of streaming utterances (one per speak()): priority tiers, secretary announcements + reserved voice, call-waiting, outcome tracking. Synthesized WAVs are reaped after playback (no /tmp leak).
• tones.py — Tone WAV generator (speak + listen bookends, call-waiting, natural set)
• media_duck.py — Async PulseAudio volume control for media ducking
• audio.py — WAV writing and pw-play async wrapper
• settings.py — Pydantic settings from env vars (prefix: TTS_)
• engines/ — TTSEngine implementations (kokoro, piper, orpheus)

speak() Progress Lifecycle

speak() blocks until playback finishes, reporting progress throughout. Clients should call it in parallel with other tools rather than blocking on it alone — the return value is informational (speech_id, duration) and never needed for subsequent reasoning.

Progress notifications

A background ticker emits MCP progress notifications every ~0.5 seconds during playback. Each notification includes progress (0-100), total (100), and a message string. MCP-aware clients render this as a live progress bar.

Progress	Phase	Message
5%	Entry tone played — audible "I heard you"	Entry tone
30%	Synthesis complete	Synthesizing...
35%	Enqueued for playback	Playing audio — you can continue working
35-99%	Playing — progress tracks elapsed time vs expected duration	Playing audio — you can continue working
100%	Playback finished	Playback complete

The entry tone fires before synthesis, covering the 1-3s latency gap. The progress message intentionally says "you can continue working" to signal to LLM clients that they don't need to wait.

Concurrency guidance

speak() blocks for 10-60s for typical text, but audio plays through physical speakers — the return value doesn't feed into subsequent reasoning. The server instructions and tool docstring explicitly tell clients to call speak in parallel with other tools in the same message. Tool annotations provide structured signals:

• openWorldHint=True — this tool interacts with the physical world (speakers)
• idempotentHint=True — safe to retry without side effects beyond replaying audio

speech_status(speech_id) is available for checking outcomes after the fact. Possible statuses: completed, playing, queued, unknown.

Outcomes are stored in a bounded ring (last 100 items) — old entries are evicted automatically.

Chunked Synthesis (Pipelined Playback)

Long texts (>= 20 words with sentence boundaries) are automatically split into sentence-sized chunks and pipelined — synthesis of chunk N+1 overlaps playback of chunk N.

Single-shot:  [---12s synthesis---][----------48s playback----------]  First audio at T+12s
Chunked:      [~1s synth][play 1][~1s synth][play 2][~1s synth][play 3]...  First audio at T+1s

Why no gaps

Kokoro synthesizes ~4x realtime on CPU, so synthesis always outpaces playback. pw-play adds ~10ms startup per chunk — imperceptible. Slower engines (Orpheus) may have brief gaps but still beat the full-synthesis-first approach.

Short text bypass

Texts under 20 words or without sentence boundaries (.!? followed by whitespace) take the single-shot path — zero overhead, identical to pre-chunking behavior.

Tone behavior & per-message coherence

Each speak() call is one queue utterance (not one queue item per chunk). The utterance reserves its ordering slot up front and streams its synthesized chunks in through an internal channel closed by an _END sentinel; the consumer stays locked to it from entry tone to exit tone. So when several projects speak at once, a message plays whole and in order — chunks never interleave with another project's audio (the old per-chunk suppress_exit_tone/_WorkItem model is gone).

• Entry tone plays once at the start (before first chunk synthesis)
• Exit/standby tone plays once, at the very end of the utterance
• Pipelining is preserved: synthesis of chunk N+1 overlaps playback of chunk N, but nothing else can be pulled until this utterance finishes

Cancellation in chunked mode

• Explicit cancel_speech(speech_id) — one speech_id now covers the whole utterance; cancelling it flags an abort (synchronously), reaps un-played chunk WAVs, and plays the cancel tone.
• MCP disconnect — the speak() handler is cancelled; its finally closes the utterance channel so the consumer drains the chunks it already has and finishes cleanly.

Progress lifecycle (chunked)

Progress	Phase	Message
5%	Entry tone played	Entry tone
5-30%	Synthesis progress across all chunks	Synthesizing chunk N/M...
35%	All chunks enqueued	Playing audio — you can continue working
35-99%	Playing — tracks elapsed time vs total duration	Playing audio — you can continue working
100%	Final chunk playback finished	Playback complete

Files

• server.py — split_text(), unified _speak() (short + chunked share one path), status-aware _await_with_progress()
• queue.py — _Utterance (streaming chunk channel + _END sentinel), create_utterance(), _play_utterance(), _should_announce(), call-waiting

Cancellation

Speech can be cancelled two ways:

1. Explicit cancel_speech(speech_id) — kills pw-play immediately, plays the cancel tone, and the consumer moves to the next item.
2. MCP cancellation — if a client disconnects or sends notifications/cancelled, the speak() handler is cancelled but playback continues. The consumer finishes the current audio naturally. This is intentional — audio is already synthesized and playing, so cutting it mid-sentence would be jarring.

When a currently-playing item is explicitly cancelled, pw-play is killed immediately and the cancel tone plays. When a queued item is cancelled, it's removed from the queue without ever playing. The consumer continues to the next item in both cases.

Graceful Shutdown

On docker compose down, make up, or make restart, the server stops accepting new speech and plays out what's already queued before stopping — not just the currently-playing message. No dropped messages on a redeploy.

How it works: SIGTERM triggers a two-phase shutdown that shares Docker's wall-clock budget:

1. Phase 1 — Handler drain (3s fixed): Uvicorn cancels in-flight speak() handlers. The handlers re-raise CancelledError without touching the consumer — already-enqueued audio keeps playing.
2. Phase 2 — Queue drain (shutdown_timeout, default 30s): Lifespan finalizer calls queue.stop(), which (a) sets _accepting=False so new speak() calls are rejected immediately, then (b) lets the consumer keep playing queued utterances normally until the queue is empty and nothing's playing, then (c) stops the consumer. Whatever doesn't fit the budget is dropped and reaped. The _accepting flag (reject new) is deliberately separate from _stopped (consumer exit) so the drain window sits between them.

Docker's stop_grace_period must exceed the total: 3s + shutdown_timeout + safety margin (default 38s). Because the whole queue now drains (not just the current message), a deep queue can hit the budget — raise both TTS_SHUTDOWN_TIMEOUT and stop_grace_period if you want longer drains.

Entry tone timing: The Nextel chirp fires immediately when speak() is called (before synthesis), acting as an audible "I heard you" that covers the 1-3s synthesis latency. Exit/standby tones still play from the consumer (they depend on queue state after playback).

Key Design Decisions

• Speech queue is serialized (one playback at a time) but synthesis is parallel; each speak() is one streaming utterance so concurrent projects never interleave
• Secretary behavior: a project is announced by name on speaker-change / after a lull, in a reserved voice excluded from the project pool; a different project queuing mid-playback fires a once-per-turn call-waiting blip mixed over the current audio
• Synthesized WAVs are reaped after playback (and on cancel/shutdown) so /tmp/mcspeak doesn't grow unbounded
• speak() blocks until playback finishes with status-aware progress — a queued item reports "waiting in line", not a false "playing" percentage
• Progress uses a background ticker task, NOT asyncio.wait_for polling (see below)
• Entry tone is awaited in speak() before synthesis — covers latency gap
• Explicit cancel_speech() kills pw-play + plays cancel tone; MCP disconnect lets playback finish
• Consumer directly awaits play_audio(); cancel() targets the consumer task with _item_cancelled flag
• speak()'s CancelledError handler does NOT cancel the consumer — only explicit cancel does
• Graceful shutdown is two-phase: 3s handler drain (fixed) + shutdown_timeout queue drain, both sequential within Docker's stop_grace_period
• Tones are non-fatal: if pw-play fails on a tone, speech still plays
• Orpheus uses llama-server (not Ollama) for 15x throughput via continuous batching
• SNAC decoder is lazy-loaded on first Orpheus call to reduce idle memory

Python 3.13 asyncio.wait_for pitfall

Do NOT use asyncio.wait_for(future, timeout) in a polling loop to track progress. In Python 3.13, wait_for cancels its inner task on timeout. When the inner task is awaiting the same asyncio.Future that the consumer will resolve, repeated cancel/re-await cycles cause a stale CancelledError to propagate to the consumer task — killing pw-play mid-playback. Instead, use a background asyncio.create_task ticker for progress and directly await the future for completion.