Pular para o conteúdo principal

Voice Mode

GarraIA supports end-to-end voice conversation with speech-to-text and text-to-speech.

Overview​

┌─────────────────────────────────────────────────────────────┐
│                    VOICE PIPELINE                             │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  User Audio → STT → LLM → TTS → Audio Response             │
│                                                              │
│  STT: Whisper (local or API)                               │
│  TTS: Chatterbox, Hibiki, OpenAI TTS                       │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Setup​

Prerequisites​

  • FFmpeg installed
  • TTS server (Chatterbox or Hibiki) for TTS
  • Optional: Whisper for local STT

Wizard integration (plan 0126)​

When garraia init runs on a machine with an NVIDIA GPU (nvidia-smi detected) and the user opts into voice mode, the wizard pre-fills the configuration block below and prints the install instructions for both servers — but it does not auto-install the Python TTS/STT stacks. Run those commands yourself once, then start the gateway with garraia start (no --with-voice flag needed when voice.enabled is already true in config.yml):

# TTS — Chatterbox Multilingual (Gradio app) on :7860
# The `chatterbox-tts` wheel is a *library* and ships no CLI — the server
# is the Gradio app that lives in the upstream repository.
git clone https://github.com/resemble-ai/chatterbox
cd chatterbox
pip install -e .
GRADIO_SERVER_NAME=127.0.0.1 GRADIO_SERVER_PORT=7860 python multilingual_app.py

# STT — whisper.cpp server on :9090
git clone https://github.com/ggml-org/whisper.cpp
cd whisper.cpp
cmake -B build && cmake --build build -j --config Release
./models/download-ggml-model.sh base
./build/bin/whisper-server --host 127.0.0.1 --port 9090 -m models/ggml-base.bin

On older whisper.cpp checkouts the server binary is called ./server instead of ./build/bin/whisper-server. Any other server that exposes the OpenAI-compatible POST /v1/audio/transcriptions route works too: the client tries whisper.cpp's POST /inference first and falls back to the OpenAI shape.

The wizard writes voice.tts_endpoint=http://127.0.0.1:7860, voice.stt_endpoint=http://127.0.0.1:9090, voice.tts_provider=chatterbox, and voice.language=pt into the emitted config.yml. CPU-only machines skip the voice prompt entirely.

Configuration​

voice:
  enabled: true
  tts_endpoint: "http://127.0.0.1:7860"  # Chatterbox/Hibiki
  stt_provider: whisper  # whisper or openai
  language: "pt"  # pt, en, es, fr, de, it, hi

Voice is a local service​

Both servers run on the machine that runs the gateway — there is no hosted GarraIA voice endpoint, and no chatterbox.garraia.org or whisper.garraia.org to point at (#1099). Every default in garraia-config is a loopback URL (http://127.0.0.1:7860 / :9090) precisely for that reason. If a config of yours names a public hostname, it came from somewhere other than this repository.

Because the servers are separate processes, "voice mode is on" and "the voice servers are up" are different facts — see Diagnostics for how to tell them apart.

TTS Providers​

Local GPU TTS served by the upstream Gradio app — GarraIA publishes no container image for it, so run it from the source checkout:

git clone https://github.com/resemble-ai/chatterbox
cd chatterbox
pip install -e .
GRADIO_SERVER_NAME=127.0.0.1 GRADIO_SERVER_PORT=7860 python multilingual_app.py

GarraIA talks to it over the Gradio API (POST /gradio_api/call/generate_tts_audio), which is what multilingual_app.py exposes. The chatterbox-tts PyPI wheel is a library only and has no serve subcommand.

The client adapts to the endpoint signature: on the first synthesis it reads GET /gradio_api/info, finds named_endpoints["/generate_tts_audio"].parameters, and builds the data array with exactly that number of values — text, language, reference (null), and for the remaining parameters the parameter_default the server declares, or the stock app's defaults when it does not. The result is cached. The stock app binds 7 inputs (text, language_id, ref_wav, exaggeration=0.5, temperature=0.8, seed_num=0, cfg_weight=0.5); a custom server that exposes only [text, lang, ref, number] (4 values) keeps working. If /gradio_api/info does not answer, the client falls back to the stock 7-value layout.

GPU compatibility: torch pin vs RTX 50xx (sm_120)​

chatterbox-tts pins torch==2.6.0 / torchaudio==2.6.0 (on Python < 3.14), and those default wheels ship no Blackwell (sm_120) kernels — on an RTX 50xx the server crashes with no kernel image is available for execution on the device. Override with the CUDA 12.8 wheels after pip install -e .:

pip install --force-reinstall torch torchaudio \
  --index-url https://download.pytorch.org/whl/cu128

Known pitfalls with the stock app​

1. Health-probe route mismatch (fixed in the gateway, #1538)​

Up to 0.4.7 GarraIA probed the server with GET /gradio_api/config, a route no Gradio version serves — it serves the app config at /config (confirmed on gradio 5.50.0 and 6.8.0: /gradio_api/config -> 404, /config -> 200). A perfectly healthy multilingual_app.py failed the probe and voice mode stayed off:

WARN garraia_voice::tts::chatterbox_client: Chatterbox health check failed:
  error sending request for url (http://127.0.0.1:7860/gradio_api/config)

This was a bug in GarraIA, not in the app, and it is fixed at the source: the probe now tries / -> /config -> /gradio_api/info and stops at the first 2xx, so the stock app is detected as-is. Upgrade and run multilingual_app.py directly — no wrapper needed.

If you are pinned to 0.4.7 or earlier, the local workaround is a small FastAPI wrapper that mounts the Gradio app and redirects the old probe route to /config. Prefer upgrading: the wrapper has to neutralize the demo.launch() that the app runs at import time, which breaks whenever upstream changes how the module is structured.

Once the probe goes green, voice mode enables:

INFO garraia_gateway::server: Voice mode enabled - Chatterbox TTS at http://127.0.0.1:7860
INFO garraia_gateway::health: tts-chatterbox 2ms (green)

2. Default voice prompts are remote URLs​

Every language's default reference voice is a https://storage.googleapis.com/... URL, and multilingual_app.py passes it verbatim to librosa.load, which only opens local files. Synthesis without an uploaded voice fails:

FileNotFoundError: [Errno 2] No such file or directory:
  'https://storage.googleapis.com/chatterbox-demo-samples/mtl_prompts/pt_m1.flac'

Workaround: upload a reference voice per synthesis, or patch resolve_audio_prompt() in multilingual_app.py to download each default prompt once into a local cache and return the local path (fix proposed upstream - see the linked issue).

Features:

  • Multilingual (pt, en, es, fr, de, it, hi)
  • GPU accelerated
  • Low latency

Hibiki​

Alternative GPU TTS. There is no published GarraIA image for it either — follow the upstream project's own instructions and point voice.tts_endpoint at whatever host and port you start it on.

LM Studio (OpenAI-compatible)​

Any server that speaks the OpenAI POST /v1/audio/speech shape, such as LM Studio, is driven by the lmstudio provider. Point tts_endpoint at that server and name the model it serves:

voice:
  enabled: true
  tts_provider: lmstudio
  tts_endpoint: "http://127.0.0.1:1234"
  tts_model: "vieneu-tts-v2-turbo"

tts_model defaults to vieneu-tts-v2-turbo when omitted. GarraIA runs a health check against the endpoint at startup and warns if it is unreachable, but still boots.

Accepted values for tts_provider​

Only three values do anything today:

ValueClientNotes
chatterboxChatterbox over the Gradio APIThe default, and the fallback
hibikiHibiki, at voice.hibiki_endpointNote: its own endpoint key, not tts_endpoint
lmstudioAny OpenAI-compatible /v1/audio/speech serverUses tts_endpoint + tts_model

Any other value silently falls back to Chatterbox. There is no openai provider and no tts_voice key: VoiceConfig has exactly enabled, tts_provider, tts_model, tts_endpoint, stt_endpoint, hibiki_endpoint and language. Earlier versions of this page documented both, and neither was ever read.

ElevenLabs and Kokoro adapters exist in garraia-channels::voice_channel, behind the voice feature, which no crate in the workspace enables. They are not reachable from the gateway and are not options here.

STT Providers​

Local Whisper​

voice:
  stt_provider: whisper
  whisper_model: "base"  # tiny, base, small, medium, large

OpenAI Whisper API​

voice:
  stt_provider: openai
  openai_api_key: "sk-..."

Usage​

Starting with Voice​

garraia start --with-voice

Voice Commands​

  • /voz or /voice - Toggle voice mode for current session
  • Voice responses are automatic when enabled

Telegram Voice​

Send voice messages and receive voice responses automatically when voice mode is enabled.

API Endpoints​

TTS Endpoint​

curl -X POST http://127.0.0.1:3888/api/tts \
  -H "Content-Type: application/json" \
  -d '{"text": "Hello, how can I help you?", "language": "en"}'

Returns audio file (WAV/MP3).

STT Endpoint​

The handler reads multipart form data and only looks at the field named file — a raw --data-binary body is ignored and the request comes back 400 No audio file provided. Send the audio as a multipart part:

curl -X POST http://127.0.0.1:3888/api/stt \
  -F "file=@audio.wav"

Returns transcribed text as JSON ({"text": "..."}).

Health Checks​

Voice services are checked at startup:

garraia health

Output includes TTS and STT status.

Diagnostics​

GET /api/diagnostics (and the Diagnostics page of the Web Console) reports one row per voice server — voice.tts and voice.stt — each probed with a 1.5 s budget:

RowStatusMeaning
voice.tts / voice.sttdisabledVoice mode is off in this process. Nothing is wrong; start with --with-voice. (Was skipped before #1437: disabled says the switch exists and is off, and like skipped it never colours the report's aggregate status.)
okThe configured endpoint answered.
error (unreachable)Configured but did not answer — connection refused or timeout. The row carries the exact start command as next_step.
error (unhealthy)Answered with HTTP 5xx: the server is up but the service is broken. next_step says the server's own logs are the next stop.
error (invalid URL)The configured endpoint is not a URL this gateway may call. next_step is a generic instruction to fix the endpoint in the config — there is no start command for a URL that is not valid.

This is the answer to "voice fails silently" (#1098): an unreachable server used to be a log line nobody read, and POST /api/tts answered 200 with a text fallback. It is now an error row in the console. To get the failure as an HTTP error instead of the fallback, ask for it explicitly:

curl -X POST 'http://127.0.0.1:3888/api/tts?fallback=false' \
  -H 'Content-Type: application/json' \
  -d '{"text": "Hello"}'

Troubleshooting​

TTS not responding​

The stock Chatterbox app (multilingual_app.py) is a Gradio app and does not serve /health — a curl .../health against a perfectly healthy server returns 404. Probe the routes GarraIA itself probes, in order:

curl -sf http://127.0.0.1:7860/ >/dev/null && echo "root ok"
curl -sf http://127.0.0.1:7860/config >/dev/null && echo "/config ok"
curl -sf http://127.0.0.1:7860/gradio_api/info >/dev/null && echo "/gradio_api/info ok"

Any one of the three answering 2xx is enough for garraia health to report ✅ tts-chatterbox (#1538). If none answers, nothing is listening on that port — check that the Chatterbox process is up and that voice.tts_endpoint points at it.

For LM Studio (tts_provider: lmstudio) the probe is /v1/models instead.

When the probe fails, the WARN line names every route it tried and what each answered, so "nothing listening" (connect failed) and "up but on another route" (HTTP 404) are distinguishable in the log.

If /config returns 200 but garraia health still reports ❌ tts-chatterbox, check your version: on 0.4.7 and earlier this is the health-probe route mismatch described under Known pitfalls above, fixed by upgrading.

Audio quality issues​

  • Increase TTS quality setting
  • Check network latency to TTS server
  • Use local TTS (Chatterbox/Hibiki)

STT errors​

  • Check FFmpeg installation
  • Verify audio format (16kHz mono recommended)
  • Try different Whisper model