> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pyai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Voices, languages, and controls

> Choose a compatible PyAI configuration, distinguish requested from served behavior, and copy a safe Omni, Hear, or Speak starting point.

Choose a configuration in this order: **product → language → accent → voice →
delivery → controls**. Product compatibility is not implied by a voice name or
by a successful request. Read the catalog, then verify the effective
configuration returned by the runtime.

<Info>
  Download the dated [JSON reference pack](/reference/pyai-reference-packs-2026-09-08.json).
  It is designed for code generation and local validation; it is not a substitute
  for `GET /v1/voices`, account entitlement checks, or the Omni `configured`
  acknowledgement.
</Info>

## Pick a product before a language

| Product | How language is selected | Runtime evidence |
| - | - | - |
| Hear streaming STT | `language=auto`, omitted, or an explicit supported code | Transcript events; no language-ID field is returned |
| Speak TTS | The selected voice | Catalog `available_on`, response headers, and successful synthesis |
| Omni | Agent or `configure.language`, plus a compatible voice | `configured.language_active`, `language_fallback`, canonical `voice_id`, and `voice_tier` |
| Cast | Query Cast capabilities | The capabilities response, not the Speak or Omni catalog |

Hear supports automatic detection only inside its published language set. Omni
accepts `en`, `es`, `fr`, `de`, and `hi`, while its served set is currently
`en`, `es`, `fr`, and `hi`; German is an English fallback. Speak catalog groups
are broader, so a Speak voice must never be presented as proof that the same
language works end to end in Omni.

## English is a language; accent is a voice choice

Selecting `en-IN` does not transform an American voice into Indian English. Use
an accent-specific voice ID, then listen to the exact configuration before you
ship it.

| Customer choice | Omni language | Example voice | Accent |
| - | - | - | - |
| English — US | `en` | `stock_dorit_en_us` | American |
| English — UK | `en` | `stock_amelia_en_gb` | British |
| English — Australia | `en` | `stock_elias_en_au` | Australian |
| English — India | `en` | `stock_arjun_en_in` | Indian |

Evaluate names, numbers, dates, phone numbers, interruptions, and short
conversational replies. Evaluate Hindi and Hinglish separately from Indian
English; a catalog identifier is not a quality result.

## Copy a complete Omni configuration

This is the body of a tagged `0x03` `configure` frame, not a REST request or a
complete browser client:

```json theme={null}
{
  "type": "configure",
  "language": "en",
  "voice_id": "stock_elias_en_au",
  "voice_instruct": "Speak calmly and clearly. Leave space between steps.",
  "persona": "You are a concise support assistant. Ask one question at a time.",
  "greeting": "Hi, thanks for calling. How can I help?"
}
```

`voice_instruct` is a delivery request, not an acoustic guarantee or an emotion
enum. It is limited to 200 characters and works only on compatible tiers. Speak
uses its own conditional `emotion` field; Cast exposes a separate capabilities
response with emotion and intensity. Do not copy controls between these products.

For Omni, check `configured.voice_instruct_supported === true` when delivery
direction is a requirement. Standard Elise and Gwen do not support this control;
English Natural does. An absent field means the server has not reported the
capability. A later `voice_capabilities` event can revoke support after synthesis
fallback. Keep prompt guidance for wording separate from this acoustic capability.

After sending `configure`, require the session to confirm any capabilities your
application needs:

```js theme={null}
function onConfigured(event, requested) {
  if (event.event !== "configured") return;
  if (event.language_fallback || event.language_active !== requested.language) {
    throw new Error("Required language was not served; do not start the call.");
  }
  console.info({ requested, served: event });
}
```

## Conversation timing and silence

Omni has a session `endpointing_ms` control and an Agent/session `idle_check_in`
preset (`auto`, `patient`, `off`). The deprecated `barge_sensitivity` and
`ack_mode` Agent fields are stored for compatibility and do not tune the runtime.
Do not map a responsiveness slider to an invented scale or present a saved value
as evidence of application. See the [controls contract and effective
acknowledgement](/realtime/omni-protocol#conversation-controls-and-effective-configuration)
and [machine-readable matrix](/reference/omni-conversation-controls-v1.json).

## Delivery is an observed behavior

| Surface | Transport | What the client receives | Common integration error |
| - | - | - | - |
| Hear streaming | WebSocket | JSON transcript events | Appending `speech_final` and `final` as two rows |
| Speak streaming | HTTP response body | Audio bytes | Waiting for the full body before beginning playback |
| Omni | WebSocket | Tagged audio, text, and control frames | Playing a control frame as audio or retaining audio after `flush` |

Speak `stream: true` means the request accepts streamed delivery. If the server
returns `x-pyai-stream: buffered`, it completed successfully but audio was
buffered before delivery. Record that mode when measuring time to first audible
audio.

## Launch check

1. Query `GET /v1/voices` and retain the canonical ID, aliases, `available_on`,
   and delivery metadata you selected.
2. Mint a short-lived, origin-bound browser session token on your backend; do
   not expose a PyAI API key in the browser.
3. Configure Omni, wait for `configured`, and compare requested versus served
   language, voice, and tier.
4. Start microphone capture and playback only after required configuration is
   active. On `flush`, stop local playback and clear queued audio immediately.
5. Test microphone denial, cancellation while connecting, duplicate
   start/stop, language fallback, failed configuration, interruptions, tool
   timeout, and duplicate tool delivery.

For a customer-owned write such as booking, retain the conversation ID, tool
invocation `call_id`, and callback-delivery identity separately. Authorize the
customer tenant in the customer backend and return a durable prior result when
the same invocation is delivered again. See [Omni tools](/guides/omni-tools).

<CardGroup cols={2}>
  <Card title="Language support" href="/reference/language-support">Accepted, served, and fallback language behavior.</Card>
  <Card title="Omni protocol" href="/realtime/omni-protocol">Tagged frames and the configured acknowledgement.</Card>
  <Card title="Browser voice agent" href="/guides/browser-voice-agent">A complete microphone-to-playback client.</Card>
  <Card title="Speak" href="/guides/speak-overview">Voice catalog, delivery modes, and audio formats.</Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.