Skip to content

Voice pipeline and providers

A normal bot selects a reviewed preset. It does not assemble provider clients.

Built-in starting points

Preset STT LLM TTS and voice Turn/audio behavior
starter-default@1 streaming Deepgram Nova 3 streaming Google AI Studio Gemini 3 Flash Preview with low thinking single-utterance SmallestAI Lightning v3.1 Pro playout; one of eight voices en-IN/hi-IN, Silero VAD, BVCTelephony noise
basic-monolingual@1 batch Deepgram Nova 3 after endpointing streaming Google AI Studio Gemini 2.5 Flash streaming Google Chirp 3 HD, one en-IN voice runtime turn handling; no noise profile
birla-prod-v42@14 streaming Deepgram Nova 3 streaming Vertex Gemini 3 Flash with low thinking; Google AI Studio fallback single-utterance SmallestAI Lightning v3.1 Pro playout with production voice rhea en-IN/hi-IN, Silero VAD, self-host-compatible no-noise profile, best-effort GCS recording

The source default birla-prod-v42@14 is published in rc43. Historical rc42 installs remain paired with birla-prod-v42@13.

The @14 Birla revision is self-host compatible: its noise.profile = "none" does not attempt the LiveKit Cloud-only noise-cancellation adapter. Its operation_deadlines.provider_operation_seconds = 25 gives provider operations a larger bounded window than the earlier 15-second revision. This field applies to provider operations beyond reasoning too, so it can increase failure-tail latency; it is a mitigation for the measured Gemini tail, not a guarantee that model requests finish within the deadline.

Development calls using this preset still select one voice deterministically from the allow-list and keep it for the entire call. Production locks pin the default rhea voice for the A/B; this is controlled configuration, not a cosmetic change or a mid-call voice change.

Credential names

basic-monolingual@1 requires:

DEEPGRAM_API_KEY=...
GOOGLE_AI_STUDIO_API_KEY=...
GOOGLE_SPEECH_SERVICE_ACCOUNT_JSON={"type":"service_account",...}

starter-default@1 requires:

DEEPGRAM_API_KEY=...
GOOGLE_AI_STUDIO_API_KEY=...
SMALLEST_API_KEY=...

birla-prod-v42@14 requires:

DEEPGRAM_API_KEY=...
VERTEX_SERVICE_ACCOUNT_JSON={"type":"service_account",...}
GOOGLE_AI_STUDIO_API_KEY=...
SMALLEST_API_KEY=...

The Birla @14 preset's enabled measurement policy is BEST_EFFORT: a recording-start or finalization failure is retained as recording evidence while the customer call continues. Install the recording-gcs extra and bind the approved recording Adapter before assistant.check(); this preset does not permit a recording Adapter to be silently omitted.

Values belong in .env locally or a production secret manager. Never put them in TOML, locks, prompts, browser bundles, or logs. Installing an extra makes an integration available; only the selected preset/deployment activates it.

What happens on each turn

  1. LiveKit sends participant audio into the preset's noise and VAD path.
  2. Endpointing decides when a participant utterance is ready.
  3. Deepgram produces participant text, in batch or streaming mode according to the preset.
  4. Gemini streams safe response text and tool decisions.
  5. TTS converts the accepted response into audio. Backends differ in when audio can begin.
  6. VAANI publishes participant and assistant transcripts and records latency, provider, tool, usage, cost, and lifecycle evidence.

Qualifying participant speech over interruptible assistant audio commits a barge-in and stops obsolete playout. Short noise can remain an interruption candidate without becoming a participant turn. Silence timing comes from the conversation profile; the bot supplies its nudge and ending wording.

Metrics are recorded when the preset enables them. A Judge runs only when the preset has a Judge slot and the bot supplies a rubric and result schema. Judge output annotates the final evidence; it cannot change the base call outcome. CallResult carries the normalized evidence; the consuming application turns it into business analytics without changing the result.

SmallestAI streaming continuity

The pinned SmallestAI LiveKit plugin sends text only when its stream is flushed. Flushing every sentence starts a separate serialized provider segment and creates audible gaps. VAANI therefore sends one complete assistant utterance per SmallestAI stream. Audio frames stream after that request starts, but first audio waits for the LLM response to finish. Participant STT and LLM generation still stream internally; the browser must not promise token-level assistant transcript updates for this route.

This is a deliberate continuity-versus-first-audio tradeoff, not a general promise made by the VAANI interface. Measure it with the intended provider/model combination before production rollout.

Text emitted with a non-fulfilling tool call

On the LiveKit streaming path, non-empty model text emitted with a tool call that does not fulfil or terminate the participant response is an acknowledgement, not disposable tool metadata. VAANI seals, synthesizes, and completes that segment before it invokes the tool and starts the tool-result continuation. The participant therefore hears the acknowledgement before the concise final answer, and the CallResult records both agent transcript items in that order.

An empty tool-only result remains silent. If acknowledgement synthesis fails before audio starts, VAANI aborts that segment rather than recording it as a committed spoken response. A qualifying barge-in interrupts the segment and fences the obsolete tool continuation.

Fallbacks

Fallback order is part of a reviewed preset or Production Deployment Profile, not browser input. VAANI may move an operation only before its safe boundary: before accepted STT text, before model output or an admitted tool effect, and before TTS audio is audible or its delivery is uncertain. This avoids duplicate turns, tools, and speech.

birla-prod-v42@14 declares a Google AI Studio reasoning fallback. The packaged basic-production@1 deployment demonstrates independent STT, LLM, and TTS fallback targets. A bot builder chooses a reviewed profile; operations teams monitor which route actually served each operation.

Supported integrations

VAANI packages integrations for Deepgram, Google, Sarvam, SmallestAI, ElevenLabs, Cartesia, OpenAI, OpenRouter, Groq, Silero, LiveKit turn detection, LiveKit noise cancellation, GCS recording, and PostgreSQL result publication. Not every integration belongs to a built-in preset.

Low-level factories remain compatibility APIs. New bots should use Assistant and a reviewed preset. Exact provider policy, credential, deadline, tool, and measurement fields are in the configuration reference.

Next: test the live path.