> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pyai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Omni realtime runtime

> Build a complete realtime voice agent through one WebSocket instead of wiring together speech-to-text, an LLM, text-to-speech, VAD, turn detection, and interruption handling.

Omni is the realtime runtime behind **PyAI Agents**, exposed through one WebSocket. Speech,
reasoning, tools, and turn-taking run in one loop at human conversational
pace (about 390 ms median voice-to-voice, in-region; early measurement, not
an SLA). You describe the agent in a `configure` frame. You do not assemble
STT, an LLM, and TTS.

```
wss://api.pyai.com/v1/omni
```

<Warning>
  **Migrating an older Omni client?** The former chat URL was discontinued on
  August 13, 2026. Follow the
  [Omni endpoint migration guide](/guides/migrate-omni-v2-chat) for the exact
  URL, auth, query, configure, framing, and error-handling changes.
</Warning>

Send audio in. Receive spoken audio and events back. Omni handles speech
recognition, reasoning, speech generation, voice activity detection, turn
detection, and interruption handling inside the session.

## Same runtime for UI and API agents

Create a saved agent in the [console](/agents/getting-started) or with
[`POST /v1/agents`](/guides/create-agents-api), then pass its returned `agent_id`
as `session_label` when opening `/v1/omni`. The saved voice, instructions,
knowledge and tools are loaded for the call. You can also configure sessions
inline without saving a profile. The endpoint and `omni:session` scope stay the same.

## One endpoint instead of a voice cascade

A traditional voice agent makes you assemble and operate this:

```
microphone → VAD → speech-to-text → LLM → text-to-speech → speaker
```

That diagram leaves out the hard parts: deciding when a turn is complete,
stopping speech when the caller interrupts, carrying context between
components, retrying failures, and keeping each hop responsive.

With Omni, your application connects to one realtime endpoint. You describe the
agent in a `configure` frame, then stream audio:

```
microphone → Omni → speaker
                 ↘ tools and knowledge
```

You still control the persona, voice, language, greeting, tools, and knowledge
sources. PyAI runs the realtime conversation loop.

Managed Agent profiles can also store an opt-in `vocabulary` list for up to
five distinctive terms. The list is fixed at session start. It does not inherit
organization Hear vocabulary, and inline Omni sessions do not accept a
vocabulary override in this release. See
[Create agents via API](/guides/create-agents-api).

English agents can select Natural `en1` (canonical id `stock_aria_en`).
The `configured` acknowledgement reports the canonical voice id and
`voice_tier: "natural"` while that tier is active. Omni Hindi uses the
Standard voices `hi1`–`hi4`; there is no Hindi Natural tier (the former
`hi5`–`hi8` aliases are retired).

## Connect

There is no agent resource you must create before opening a session. Authorize
the WebSocket with an API key, then send the agent configuration:

```ts theme={null}
import WebSocket from "ws";

function controlFrame(body: object) {
  return Buffer.concat([
    Buffer.from([0x03]),
    Buffer.from(JSON.stringify(body), "utf8"),
  ]);
}

const ws = new WebSocket(
  "wss://api.pyai.com/v1/omni?format=pcm16&rate=24000",
  ["pyai.v1", `pyai-key.${process.env.PYAI_API_KEY}`],
);

ws.on("open", () => {
  ws.send(controlFrame({
    type: "configure",
    voice_id: "stock_dorit_en_us",
    persona: "You are a helpful appointment scheduler.",
    tools: ["datetime"],
  }));
});

ws.on("message", (data, isBinary) => {
  if (!isBinary) return;
  const frame = Buffer.from(data as Buffer);
  if (frame[0] !== 0x03) return;
  const event = JSON.parse(frame.subarray(1).toString("utf8"));
  if (event.event === "configured") {
    console.log(event);
    // { "event": "configured", "voice_id": "stock_dorit_en_us", "tools": 1 }
  }
});
```

Success is a `configured` control frame within about two seconds. `tools` is
the accepted count. After that, send PCM16 as `0x01 || audio_bytes`. Server audio uses
`0x01`; plain UTF-8 caller-transcript deltas use `0x02`; control JSON uses
`0x03`. Every server JSON body is keyed on `event`; the transcript body is not
JSON. See the browser tutorial for capture, playback, and frame handling.

## Choose how to build

<CardGroup cols={2}>
  <Card title="Choose your path" href="/choose-your-path">
    API primitives, console Agent, or a framework adapter.
  </Card>

  <Card title="Launch an Agent" href="/agents/getting-started">
    Create, test, and put a Call Now button on your site without a token broker.
  </Card>

  <Card title="Create agents via API" href="/guides/create-agents-api">
    Store a profile, bind knowledge and tools, open Omni with session\_label.
  </Card>

  <Card title="Build the browser client" href="/guides/browser-voice-agent">
    Capture the microphone, mint a short-lived token, handle frames, and play audio.
  </Card>

  <Card title="Put Omni on a phone call" href="/guides/twilio-voice-agent">
    Bridge Twilio media, barge-in, DTMF, and call-control events.
  </Card>

  <Card title="Read the wire protocol" href="/realtime/omni-protocol">
    Binary audio framing, lifecycle events, tools, and session controls.
  </Card>
</CardGroup>

## When to use a focused API instead

Use [Hear](/guides/streaming-stt) when you only need transcripts or live
captions. Use Speak when you already have text and only need audio. If you want
to operate each component yourself, combine Hear with your own reasoning and
speech output where Hear's published sync/streaming language set fits the call.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.