The "go" tone was awaited before the VAD read loop started, adding ~1.1s of
dead time before capture effectively began. Now the tone fires as a background
task right after the mic warmup and the read loop starts immediately, so the
person can talk over the dial tone and the turn also ends sooner on silence.
The tone still bleeds into the captured head; VAD skips it (pure tone !=
speech) and the min_speech_ms guard is unchanged. Applied to all three
recorder paths (VAD, fixed-duration Linux, macOS stream); the tone task is
cancelled in the finally so a short recording is never outlived by the tone.
- listen() now retries the Parakeet transcription up to 3x (the gateway
serializes inference on one slot and intermittently times out). The
recording is already saved, so retrying is free — no more manual
transcribe() recovery.
- Add _terminate_quietly() and use it in all three recorder paths so
terminating an already-exited pw-record (disconnected mic) no longer raises
ProcessLookupError. That lets the mic-absent case reach the clear
"No audio captured — is a microphone connected?" message.
Two pure-context wins, no behavior change:
- Tighten the speak() and listen() docstrings, which are permanent per-turn
context. speak 480->197 tok, listen 548->250 tok; all-tools fixed cost
~1783->~1203 tok/turn. Keeps the load-bearing cues (call speak in parallel;
speak-then-listen sequentially) and drops the progress/ducking/chunking
prose the calling model doesn't need.
- Slim the default listen() 'json' result to just {text} + recording/vad
metadata. The gateway returns a null-heavy segments[]/usage/words blob that
only burned context; 'verbose_json' still returns the full segments.
The error return interpolated only str(e), so exceptions that stringify to
"" (e.g. asyncio.TimeoutError) surfaced as "VAD recording failed: " with no
detail — exactly what a disconnected mic produced. Include the exception type
and a fallback hint, and reword the no-audio PlaybackError toward "is a
microphone connected?".
Add a four-touch-point platform audio backend so the speaker/mic plumbing is
swappable per OS without touching the queue/VAD/tone logic. Implements Linux
(PipeWire, unchanged) and macOS (afplay, Swift/AVFoundation recorder verified
capturing, afconvert transcode, documented ducking no-op); Windows is
documented in PLATFORMS.md as the next slot-in. Linux behavior is unchanged.
- server lifespan compiles the Swift recorder on macOS at startup so the first
listen() isn't slowed by swiftc and build errors surface early (Linux/Windows
skip this).
- PLATFORMS.md documents the four-touch-point audio backend, the macOS setup
(built-ins + uv, no Docker), and the Windows plan (winsound + sounddevice).
- convert_audio: use ffmpeg when present (Linux path unchanged); on macOS
without ffmpeg, fall back to the built-in afconvert for m4a, with a clear
"install ffmpeg for mp3/ogg/flac" error for the formats it can't encode.
- media_duck: document why macOS is left as a deliberate no-op — the only
built-in volume control is system-wide and would dim our own afplay voice;
proper per-app ducking needs CoreAudio, not osascript.
Introduce a platform audio backend so the four OS-specific subprocess
touch-points (play, record-stream, transcode, duck) are swappable without
touching the queue/VAD/tone logic. The macOS recorder emits the same
s16/mono/16k PCM-to-stdout wire format as `pw-record -`, so the VAD frame
loop is byte-for-byte identical across platforms.
- platform_audio.py: backend selector (Linux + macOS implemented, Windows
documented). Compiles the Swift recorder once, cached in ~/.cache/mcspeak.
- _macos/mcspeak_record.swift: AVFoundation recorder, built-ins only (no brew).
Verified capturing on the target Mac over SSH — TCC does not block headless
capture.
- audio.py: play_audio uses afplay on macOS (pw-play + volume workaround stay
Linux-only); record_audio_until_silence streams via the backend command;
record_audio gains a stream-and-collect path for stdout-only backends.
Linux behavior is unchanged (commands verified, queue tests green, ruff clean).
Not yet done: afconvert transcode branch (generate_audio non-WAV only),
osascript ducking (currently a graceful no-op), and running the server
natively on the Mac + a live speech test.
Update CLAUDE.md for the streaming-utterance model (drop the stale
_WorkItem/suppress_exit_tone references), the secretary announcements +
reserved voice, call-waiting, the telephone tone kit, and the
conversation-first listen() defaults.
A telephone-themed bookend set, tuned by ear:
- speak: heartbeat entry ("Speak to Me" woven with warm MF tones), soft-chord
exit (pure sines, easy on the ears on repetition)
- listen: mf-dial start (the genuine Young Lust R1 operator routing sequence,
KP 0-4-4-1-8-3-1 ST) and machine end (Welcome to the Machine throb)
- mf-listen/mf-done gentler alternatives, and a call-waiting blip
listen() is now conversation-first: wait_for_silence defaults on with a 30s
cap, and the "go" tone plays only after the mic is live (warmup) so the first
word isn't clipped. VAD defaults tuned (aggressiveness 3, 2200ms silence,
400ms min-speech) after live testing showed the old values cut replies off on
brief background transients. Adds webrtcvad-wheels for the VAD path.
Replace the per-chunk queue model with whole streaming utterances so
concurrent projects never interleave. Each speak() reserves one ordering
slot up front and streams its synthesized chunks through an internal
channel; the consumer stays locked to that utterance from entry tone to
exit tone, preserving synth/playback pipelining without letting another
project's audio wedge between sentences.
Also folded in:
- Reap synthesized WAVs after playback (and on cancel/shutdown/queue-full),
fixing an unbounded /tmp/mcspeak leak — one file per spoken sentence was
never deleted.
- Secretary announcements: announce a project by name on speaker-change or
after a lull, synthesized in a reserved voice (Kokoro) that is excluded
from the project auto-assignment pool. Decision made at play time so
urgent reordering is respected.
- Call-waiting: a brief blip mixed over the current message when a different
project queues up, gated to once per playing turn so it never spams.
- Unify _speak_single/_speak_chunked into one _speak path.
- Status-aware progress: a queued item now reports "waiting in line" instead
of racing to a false "playing 99%".
- Synchronous abort flag closes a cancel-race where a chunk synthesized
during the cancel-tone window could leak past the reap.
mcspeak gains a `listen` MCP tool that records from the host's default
PipeWire source for N seconds and pipes the WAV through the existing
transcribe_audio() pipeline. The container's PipeWire socket bind
mount (already in place for pw-play output) gives pw-record the same
bidirectional access — no new container capabilities needed.
Recording format: 16 kHz mono s16, which is Parakeet's preferred
input, so no resample step. Default source is whatever the host
considers the default mic (`pactl get-default-source`); switching
mics on the host is picked up at next call without restart. Override
per-call via the `source` param if needed.
Optional save_path persists the recording under /output/ (same
scoping discipline as generate_audio). Response carries recorded
(in-container path) and saved_to (host-visible path) so callers can
re-transcribe with different params or play back later.
Forward-compat params (timestamp_granularities, diarize, num_speakers,
punctuation, min_confidence) thread straight through to transcribe_audio
so listen() inherits whatever the gpu-stack Parakeet endpoint gains
later, no client change.
Implementation gotcha: pw-record's exit code on SIGTERM is
non-deterministic (observed 1 on this version, not 0 or -15). The
original allowlist check would have rejected every successful
recording. The fix trusts the WAV file's existence + size instead;
if pw-record genuinely fails the file is missing or empty, which is
the reliable signal.
mcspeak gains a `transcribe` MCP tool that accepts an audio file path
and returns text. Hits the Whisper-API-shaped Parakeet endpoint at
mcspeak.gpu.supported.systems/v1/audio/transcriptions with the shared
bearer key from TTS_PARAKEET_KEY. The mcspeak.* subdomain flows
through to Langfuse as user=mcspeak for tenant attribution.
Input audio_path validates against /output/ and /tmp/mcspeak/ —
symmetric with generate_audio's _resolve_output_path discipline. That
covers the primary round-trip use case (transcribing audio mcspeak
just generated) without extra mounts. Path canonicalization rejects
both absolute paths outside the allowlist and ../-traversal escapes.
The tool exposes forward-compat params (timestamp_granularities,
diarize, num_speakers, vad, punctuation, min_confidence) that the
current phonescribe gateway silently ignores. When the gpu-stack
agent lands word-level alignment or speaker diarization, the same
calls start producing richer responses with no client change. The
shaped-now-instead-of-later approach saves a breaking change later.
response_format dispatch: json/verbose_json parse as JSON dict
directly; text/srt/vtt wrap as {"text": <body>, "format": <fmt>}.
The gateway today rejects srt/vtt with HTTP 400 "use json | text |
verbose_json" — that's a clean error users see, not silent garbage.
transcribe.py: shared httpx.AsyncClient mirrored from orpheus.py's
pattern (explicit timeouts, ConnectError/ReadError/TimeoutException
catches). Closed via lifespan finalizer.
httpx 0.28 gotcha: passing `data=` as list-of-tuples silently routes
the value to `content=`, which wraps as SyncByteStream and crashes
the AsyncClient with "Attempted to send a sync request." `data` must
be a Mapping; list values inside the dict become repeated form
fields automatically.
generate_audio gains output_path + format params (wav, mp3, ogg, flac,
m4a). Non-wav formats route through ffmpeg (added to the Dockerfile).
Files land under /output/ inside the container, bind-mounted from the
host's TTS_OUTPUT_HOST_DIR (defaults to ~/mcspeak-out). Subdirectories
are auto-created. Path validation rejects writes outside /output/ —
both absolute escapes like /etc/passwd and traversal forms like
../../etc/passwd are canonicalized and checked against the mount root.
Extension auto-corrects to match the requested format.
piper-tts service added to compose using rhasspy/wyoming-piper. Default
voice es_MX-ald-medium is pre-warmed at container start and persists in
./piper-data/ across recreates (~60 MB download on first run). Port
10200 published to localhost so host-side scripts can also reach
Wyoming directly; mcspeak inside the stack uses container DNS. Voice
configurable via TTS_PIPER_VOICE — the same env var threads through to
both wyoming-piper's --voice flag and mcspeak's settings.piper_voice,
so they stay in sync.
PiperEngine.__init__ accepts a default_voice override; settings.piper_voice
threads it from the env. list_engines now reports the configured default
rather than the hardcoded en_US-lessac-medium. speak() / generate_audio()
calls with engine=piper and no explicit voice use the configured one.
mcspeak-internal network changed from internal:true to default-bridge.
internal:true was overkill — the goal was per-stack DNS scoping (which
compose provides automatically via the project-prefixed network name),
not internet isolation. The latter broke piper-tts's HuggingFace voice
download with "Name resolution failure" on first start.
.gitignore: add piper-data/ so the downloaded voice models stay out of
the repo.
Stream-restore mute: pw-play streams were being silently restored to
0% volume by PulseAudio's stream-restore module matching the music
role key, producing audible-but-clean-exit playback. audio.py now
bumps each new pw-play sink-input to 100% as a background task; the
new value re-stamps stream-restore on stream end.
Entry tone overlap: when speak() arrived during prior playback, its
entry tone played immediately from the handler and overlapped the
ongoing audio. queue.is_idle() now gates this — tone plays now when
idle (preserves latency-hiding intent), else defers to the consumer
via _WorkItem.entry_tone, which plays it right before the item's
audio. Chunked path defers only on chunk 0.
Docker: llama-server gated behind a `with-orpheus` compose profile;
mcspeak.depends_on uses required:false. `make up` defaults to
kokoro-only with no GPU dependency; `make up-with-orpheus` runs the
full stack. Rename dootie-internal network → mcspeak-internal
(auto-created per-stack via internal:true). Add 127.0.0.1:8371 port
mapping for local MCP clients.
README: fix claude mcp add command (was stdio, must be --transport
http); document kokoro-only default and the with-orpheus path; add
Kokoro model download step. .env.example added so `cp .env.example
.env` matches the documented setup.
Standard src-layout: package name matches import name. All internal
imports are relative so no code changes needed beyond pyproject.toml.
Also fixes 4 pre-existing ruff line-length violations.
Update server instructions and tool docstring to explicitly tell clients
to call speak() in parallel with other tools rather than blocking on it
alone. Expand CLAUDE.md concurrency section with progress notification
details (ticker interval, message fields, annotation semantics).
Duck all PulseAudio sink-inputs (Firefox, Spotify, etc.) before speech,
crossfade the entry tone over the fading media, then restore volumes
after the exit tone. Uses pactl subprocesses for reliable PipeWire
compatibility — pulsectl-asyncio's native protocol writes are silently
dropped by PipeWire's PA compat layer.
New module media_duck.py with MediaDucker class. Unduck is consolidated
into the consumer's finally block to guarantee restoration on all exit
paths (normal, error, cancel, shutdown).
Chunked synthesis splits long texts into sentences and pipelines
synthesis with playback — first audio plays within ~1-2s instead of
waiting for full synthesis. Queue supports suppress_exit_tone between
chunks for seamless playback.
Tool annotations (ToolAnnotations) signal behavioral hints to MCP
clients: speak() is openWorldHint=True (physical speakers),
cancel_speech() is destructiveHint=True, read-only tools marked
accordingly.
Progress notifications now include message= strings that tell LLM
clients they can continue working during playback. Server instructions
corrected from "returns immediately" to accurately describe blocking
behavior with parallel-safe guidance. speak() docstring updated to
encourage concurrent tool calls.
The same shutdown_timeout was used for both uvicorn handler drain AND
queue.stop() consumer drain — sequential phases sharing Docker's
wall-clock budget. With 30s each, worst case was 60s, exceeding
the 35s stop_grace_period and causing SIGKILL.
Fix: uvicorn gets a fixed 3s drain (handlers just re-raise), queue
gets the full shutdown_timeout. Docker grace = 3 + timeout + 5s safety.
Also adds shutdown observability: startup logs the timing chain,
queue.stop() logs remaining audio vs available budget.
speak()'s CancelledError handler was calling queue.cancel() which killed
pw-play mid-sentence during container restarts. Now it just re-raises,
letting the consumer finish naturally. Explicit cancel_speech() still
kills playback immediately.
Also override FastMCP's timeout_graceful_shutdown=0 in uvicorn config
so request handlers get shutdown_timeout seconds before cancellation.
speak() now blocks until playback finishes, reporting progress via SSE
(5% entry tone → 30% synthesis → 35-99% playing → 100% done). Entry
tone fires immediately on call to cover synthesis latency.
Queue shutdown waits for current speech to finish (configurable timeout,
default 30s) before draining pending items — no more mid-sentence
cutoffs on container restart.
Cancellation via cancel_speech() tool or MCP notifications/cancelled
kills pw-play and plays a vinyl scratch tone. Consumer continues to
next item after cancel.
Progress tracking uses a background ticker task instead of
asyncio.wait_for polling — the latter causes stale CancelledError
propagation to the consumer under Python 3.13.
Each project gets a distinct voice from a curated English pool,
assigned via round-robin with gender/accent interleaving for
maximum perceptual contrast between consecutive projects.
- New voice_identity.py: pool filtering, interleaving, persistence
- Round-robin replaces SHA-256 hashing (no collisions until pool
exhaustion at 22 voices)
- Assignments persist to /data/voice-assignments.json across restarts
- speak() and generate_audio() accept optional project= parameter
- MCP roots fallback with 2s timeout for future bidirectional clients
- English-only pool (af_/am_/bf_/bm_/ef_/em_ prefixes)
- af_nicole excluded from auto-assign (whispery), still explicit-ok
- Fix voice blacklist to use full identifiers (am_adam, af_jessica)
Queue-aware exit tones: descending roger beep when queue empties
("over"), ascending standby blip when more items are queued ("standby,
more coming"). Includes quindar-out (2475 Hz) as Apollo-themed
alternative. Configurable via TTS_EXIT_TONE env var.
Also adds CLAUDE.md documenting the full tone system.
Generate 48kHz WAV tones at startup (chirp and apollo/quindar) using
numpy. The queue consumer plays the selected tone before each speech
item via pw-play. Configurable via TTS_ENTRY_TONE env var: chirp
(default), apollo, none, or path to a custom WAV file.
SNAC's convolutional decoder has a receptive field spanning multiple
frames. Batched decode (28 tokens at a time) created boundary
discontinuities that produced muddled audio. Since pw-play waits for
the full WAV anyway, single-pass decode costs only ~2s extra and
produces clean audio.
Build llama.cpp from source with SM 120 CUDA kernels and FORCE_CUBLAS
for RTX 5070 Blackwell. Rewrite OrpheusEngine to stream tokens via SSE
and decode SNAC in overlapping 28-token batches (4 frames), replacing
the blocking requests+stream:false approach.
Performance: 13.5 → 170-213 tok/s. 100s audio generates in ~48s (2x
faster than realtime). Replaces requests with httpx async client.
Also switch MCP transport to stateless_http mode so container restarts
don't invalidate client sessions.
SNAC + torch no longer load at startup — deferred to first Orpheus
call via double-checked locking. Startup drops from ~13s to 0.5s,
idle RAM reduced by ~200MB. OrpheusEngine constructor no longer
takes snac_model; it self-loads on demand.
CPU-only torch image (~180MB vs 873MB CUDA), PipeWire socket
passthrough for audio playback, SNAC HuggingFace cache volume.
Served at voice.l.supported.systems via caddy-docker-proxy.
FastMCP 3.0 Streamable HTTP server exposing Piper (Wyoming/Docker),
Kokoro (ONNX), and Orpheus (Ollama+SNAC) as MCP tools. Includes a
FIFO speech queue so concurrent agents don't talk over each other —
waiting callers get queue position updates via ctx.info().
Tools: speak, generate_audio, list_voices, list_engines
Resource: audio://recent
