> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pyai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Build a browser voice agent

> Capture the mic, stream PCM16 to Omni over WebSocket, and play the agent's voice back, a full talking agent in the browser in about 10 minutes.

By the end of this guide you'll have a single HTML page that opens your mic,
streams audio to an [Omni](/realtime/omni-protocol) agent in real time, plays the
agent's reply through your speakers, and renders the live transcript as you both
talk. No framework, no build step, just the Web Audio API and a WebSocket.

<Note>
  **Run the complete example.** Scaffold this guide's full, CI-tested code in one
  command, no clone:

  ```bash theme={null}
  npm create pyai-app@latest omni-browser-widget
  ```

  Or browse it: [`omni-browser-widget`](https://github.com/atomsai/pyai-examples/tree/main/omni-browser-widget).
</Note>

## How it fits together

```mermaid theme={null}
flowchart LR
  mic[getUserMedia] --> cap[AudioWorklet capture]
  cap -->|Float32 to PCM16| up[24 kHz PCM16 frames]
  up -->|binary WS| omni[Omni wss/v1/omni]
  omni -->|binary WS: agent audio| play[AudioWorklet playback]
  omni -->|binary WS: JSON events + transcript| ui[Transcript UI]
  play --> spk[Speakers]
```

One binary envelope shares the socket: byte `0x01` carries PCM16 audio, `0x02`
carries plain UTF-8 caller-transcript deltas, and `0x03` carries UTF-8 JSON control
payloads. Caller audio must also include the leading `0x01`; outbound control
frames use `0x03` plus JSON keyed by `type`. See the wire protocol before
implementing a transport.

<Info>
  This guide is correct on transport, codecs, resampling, and the event *behaviors*
  you build against. The exact JSON payloads for each event are defined by the
  [Omni wire protocol](/realtime/omni-protocol), which is the source of truth, we
  isolate event handling behind a single function so you can fill in field names
  once you've checked the reference.
</Info>

<Warning>
  **Protected opening invariant.** Browser echo cancellation is not a startup
  gate. Resume the playback graph before sending `configure`, buffer any server
  PCM that arrives before it is ready, and send real-time digital silence
  instead of microphone or speaker energy while the turn-0 consent line and
  greeting play.
  Rearm microphone audio only after that queue drains. Later replies should keep
  normal barge-in after a short echo-cancellation warm-up.
</Warning>

## Prerequisites

<Steps>
  <Step title="A key, that's it">
    Omni is **zero-state**: there's nothing to create first. The session is
    authorized by your key's org, and the agent's behavior travels in a
    `configure` frame after connect. Grab a `pyai_test_` sandbox key to build
    against, it works instantly with hard daily caps and no billing.
  </Step>

  <Step title="A local static server">
    `getUserMedia` requires a secure context, which includes `http://localhost`.
    Any static server works: `npx serve`, `python3 -m http.server`, etc.
  </Step>
</Steps>

<Warning>
  A `pyai_test_` key is fine for a local experiment. **Never ship any long-lived
  key or publishable token in a production voice agent.** Use the hosted website
  widget from Agent → Connect, or have your backend call
  `POST /v1/omni/sessions` and hand the browser only the returned short-lived,
  origin-locked session token.
</Warning>

## Production browser token broker

Keep the live API key on your server. Your authenticated application endpoint
mints one short-lived token for one allowed browser origin:

```js theme={null}
import express from "express"; // npm install express

const app = express();
app.post("/api/omni-session", async (_req, res) => {
  // Authenticate your own user before minting. Never reflect an arbitrary
  // request Origin; allow-list the deployed application origin server-side.
  const upstream = await fetch("https://api.pyai.com/v1/omni/sessions", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.PYAI_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      allowed_origins: [process.env.APP_ORIGIN],
      ttl_seconds: 60,
    }),
  });
  res.status(upstream.status).json(await upstream.json());
});

app.listen(3000);
```

The browser asks your backend for the token, then connects to the returned URL:

```js theme={null}
const session = await fetch("/api/omni-session", { method: "POST" }).then((r) => r.json());
const ws = new WebSocket(session.url, ["pyai.v1", `pyai-key.${session.token}`]);
```

The token carries only `omni:session`, is limited to one concurrent session,
and is locked to `allowed_origins`. The default lifetime is 60 seconds and the
maximum is 600 seconds. Mint it immediately before connecting; do not store it
in local storage, analytics, logs, or a reusable configuration file.

## Pick one sample rate and stick to it

The single biggest source of "it sounds like chipmunks/robots" bugs is a sample
rate mismatch. Omni speaks **PCM16 little-endian**; for browser/WebRTC use
**24 kHz**, declared on the URL as `?format=pcm16&rate=24000`.

The clean trick: ask the browser for a 24 kHz `AudioContext` so the mic capture,
the worklets, and the wire all agree, **no resampling needed**.

```js theme={null}
const ctx = new AudioContext({ sampleRate: 24000 });
```

Most browsers honor this. If yours pins the context to its hardware rate
(commonly 48 kHz), you have two options: keep the 24 kHz context and let the
worklet see 24 kHz directly, or run at 48 kHz and **decimate 2:1** on capture
(take every other sample) / **upsample 2:1** on playback (duplicate each sample).
The math is exact because 48000 / 24000 = 2. We use the 24 kHz context path; the
fallback is one `if` away and called out in the code.

## Build it

<Steps>
  <Step title="Capture worklet: mic Float32 → PCM16">
    An `AudioWorkletProcessor` runs on the audio thread and hands you 128-sample
    blocks. Buffer them into \~20 ms frames (480 samples @ 24 kHz), convert
    Float32 `[-1, 1]` to little-endian Int16, and post the bytes to the main
    thread.

    ```js capture-processor.js theme={null}
    // 20 ms @ 24 kHz = 480 samples per frame.
    const FRAME = 480;

    class CaptureProcessor extends AudioWorkletProcessor {
      constructor() {
        super();
        this._buf = new Int16Array(FRAME);
        this._n = 0;
      }

      process(inputs) {
        const ch = inputs[0]?.[0];
        if (!ch) return true;
        for (let i = 0; i < ch.length; i++) {
          // clamp then scale: negatives use 0x8000, positives 0x7FFF
          const s = Math.max(-1, Math.min(1, ch[i]));
          this._buf[this._n++] = s < 0 ? s * 0x8000 : s * 0x7fff;
          if (this._n === FRAME) {
            // transfer the ArrayBuffer to avoid a copy
            const out = this._buf.slice();
            this.port.postMessage(out.buffer, [out.buffer]);
            this._n = 0;
          }
        }
        return true;
      }
    }

    registerProcessor("capture-processor", CaptureProcessor);
    ```
  </Step>

  <Step title="Playback worklet: a ring buffer of agent audio">
    Agent audio arrives in bursts; speakers consume it at a steady 24 kHz. Bridge
    the two with a ring buffer. Crucially, expose a `clear` message, that's how
    barge-in stays snappy (next step).

    ```js playback-processor.js theme={null}
    const CAP = 24000 * 10; // up to 10 s of buffered audio

    class PlaybackProcessor extends AudioWorkletProcessor {
      constructor() {
        super();
        this._ring = new Float32Array(CAP);
        this._r = 0;
        this._w = 0;
        this._hadAudio = false;
        this.port.onmessage = (e) => {
          if (e.data === "clear") {
            this._r = this._w = 0; // drop everything queued (barge-in)
            this._hadAudio = false;
            return;
          }
          const pcm = new Int16Array(e.data);
          for (let i = 0; i < pcm.length; i++) {
            this._ring[this._w] = pcm[i] / 0x8000; // Int16 → Float32
            this._w = (this._w + 1) % CAP;
          }
          this._hadAudio ||= pcm.length > 0;
        };
      }

      process(_inputs, outputs) {
        const out = outputs[0][0];
        for (let i = 0; i < out.length; i++) {
          out[i] = this._r === this._w ? 0 : this._ring[this._r];
          if (this._r !== this._w) this._r = (this._r + 1) % CAP;
        }
        if (this._hadAudio && this._r === this._w) {
          this._hadAudio = false;
          this.port.postMessage("drained");
        }
        return true;
      }
    }

    registerProcessor("playback-processor", PlaybackProcessor);
    ```
  </Step>

  <Step title="Connect to Omni and wire the audio loop">
    Open the WebSocket with the key as a subprotocol (browser-safe, browsers
    can't set headers on the upgrade), forward capture frames as binary, and feed
    binary replies into the playback ring. The official `@pyai/sdk` ships
    `realtimeURL` / `realtimeSubprotocol` helpers so you don't hand-build the URL.

    <CodeGroup>
      ```js Vanilla JS theme={null}
      // This is the short-lived, origin-bound session token returned by your
      // server-side token broker. It is never a PyAI API key.
      const SESSION_TOKEN = await fetch("/api/omni-session", { method: "POST" })
        .then((response) => response.json())
        .then(({ token }) => token);
      const encoder = new TextEncoder();
      const decoder = new TextDecoder("utf-8", { fatal: true });
      const OPENING_WAIT_MS = 4000;
      const REPLY_WARMUP_MS = 350;
      const OPENING_DRAIN_GRACE_MS = 500;
      const requestedConfiguration = {
        language: "en",
        voiceId: "stock_dorit_en_us",
      };
      let openingPhase = "waiting"; // waiting → playing → complete
      let connectedAt = 0;
      let responseActive = false;
      let responseStartedAt = 0;
      let drainTimer;

      function tagged(tag, payload) {
        const body = typeof payload === "string"
          ? encoder.encode(payload)
          : new Uint8Array(payload);
        const frame = new Uint8Array(body.byteLength + 1);
        frame[0] = tag;
        frame.set(body, 1);
        return frame;
      }

      function sendControl(ws, body) {
        ws.send(tagged(0x03, JSON.stringify(body)));
      }

      function protectedCallerAudio(pcm) {
        const now = performance.now();
        const protectOpening =
          openingPhase === "playing" ||
          (openingPhase === "waiting" && now - connectedAt < OPENING_WAIT_MS);
        const protectReplyWarmup =
          responseActive && now - responseStartedAt < REPLY_WARMUP_MS;
        return protectOpening || protectReplyWarmup
          ? new Uint8Array(new Uint8Array(pcm).byteLength) // digital silence
          : pcm;
      }

      // Omni is zero-state, no agent to create. Optionally add
      // &session_label=<tag> to brand the session in your kb_endpoint.
      const url = "wss://api.pyai.com/v1/omni?format=pcm16&rate=24000";

      async function start() {
        const ctx = new AudioContext({ sampleRate: 24000 });
        await ctx.resume(); // call start() from the user's click handler
        await ctx.audioWorklet.addModule("capture-processor.js");
        await ctx.audioWorklet.addModule("playback-processor.js");

        const stream = await navigator.mediaDevices.getUserMedia({
          audio: { channelCount: 1, echoCancellation: true, noiseSuppression: true },
        });

        const src = ctx.createMediaStreamSource(stream);
        const capture = new AudioWorkletNode(ctx, "capture-processor");
        const playback = new AudioWorkletNode(ctx, "playback-processor");
        src.connect(capture);          // mic → capture (no audible monitor)
        playback.connect(ctx.destination); // playback → speakers

        const ws = new WebSocket(url, ["pyai.v1", `pyai-key.${SESSION_TOKEN}`]);
        ws.binaryType = "arraybuffer";

        // set the agent's voice + behavior on connect (zero-state configure)
        ws.onopen = () => {
          connectedAt = performance.now();
          sendControl(ws, {
            type: "configure",
            language: requestedConfiguration.language,
            voice_id: requestedConfiguration.voiceId,
            persona: "You are a friendly receptionist.",
            greeting: "Hi, thanks for calling. How can I help?",
          });
        };

        // mic frames → server
        capture.port.onmessage = (e) => {
          if (ws.readyState === WebSocket.OPEN) {
            ws.send(tagged(0x01, protectedCallerAudio(e.data)));
          }
        };
        playback.port.onmessage = (e) => {
          if (e.data !== "drained") return;
          clearTimeout(drainTimer);
          drainTimer = setTimeout(() => {
            openingPhase = "complete";
            responseActive = false;
            responseStartedAt = 0;
          }, OPENING_DRAIN_GRACE_MS);
        };

        ws.onmessage = (e) => {
          const frame = new Uint8Array(e.data);
          const tag = frame[0];
          const body = frame.subarray(1);
          if (tag === 0x01) {
            clearTimeout(drainTimer);
            if (!responseActive) {
              responseActive = true;
              responseStartedAt = performance.now();
              if (openingPhase === "waiting") {
                openingPhase =
                  responseStartedAt - connectedAt < OPENING_WAIT_MS
                    ? "playing"
                    : "complete";
              }
            }
            playback.port.postMessage(body.slice().buffer);
          } else if (tag === 0x02) {
            renderTranscript({
              role: "user",
              text: decoder.decode(body), // append this delta to the current row
              final: false,
            });
          } else if (tag === 0x03) {
            handleEvent(JSON.parse(decoder.decode(body)), playback, () => {
              clearTimeout(drainTimer);
              openingPhase = "complete";
              responseActive = false;
              responseStartedAt = 0;
            });
          }
        };

        ws.onclose = (e) => console.log("closed", e.code, e.reason);
        window.stop = () => { ws.close(); stream.getTracks().forEach((t) => t.stop()); };
      }
      ```

      ```ts @pyai/sdk theme={null}
      import PyAI from "@pyai/sdk";

      // The SDK client is initialized with the same short-lived session token
      // from your server-side broker; do not put a long-lived API key in this file.
      const SESSION_TOKEN = await fetch("/api/omni-session", { method: "POST" })
        .then((response) => response.json())
        .then(({ token }) => token);
      const pyai = new PyAI({ apiKey: SESSION_TOKEN });
      // omni.connect() handles the frame-key asymmetry for you: it sends a
      // type-keyed `configure` on open and parses event-keyed server frames into
      // callbacks, so you can't trip the "configured but zero turns" bug.
      const omni = pyai.omni.connect({
        rate: 24000, // zero-state, sessionLabel is optional, not required
        configure: {
          voice_id: "stock_dorit_en_us",
          persona: "You are a friendly receptionist.",
          greeting: "Hi, thanks for calling. How can I help?",
        },
        onAudio: (chunk) => playback.port.postMessage(chunk), // agent audio → speakers
        onBargeIn: () => playback.port.postMessage("clear"),  // drop stale audio
        onTranscript: (f) => renderTranscript(f),
        onError: (e) => console.error(e),
      });
      // mic frames → server
      // Reuse the protectedCallerAudio state above; the SDK owns framing, while
      // your media layer still owns opening isolation and playback drain.
      capture.port.onmessage = (e) => omni.sendAudio(protectedCallerAudio(e.data));
      ```
    </CodeGroup>

    <Warning>
      **Frame-key asymmetry, the #1 Omni bug.** Your **outbound** control frames
      (`configure`, `dtmf`) are keyed on **`type`**; the **inbound** server frames
      (`hello`, `session_started`, `configured`, `flush`, …) are keyed on
      **`event`**. The edge is transparent, so a mis-keyed `{"event":"configure"}`
      is **acked but silently dropped**, you get a connected session that never
      speaks, with no error. Send `{"type":"configure",…}`; parse inbound on
      `event`. The `@pyai/sdk` `omni.connect()` helper does both for you.
    </Warning>
  </Step>

  <Step title="Handle events: transcript + barge-in">
    `0x02` carries plain UTF-8 caller text; `0x03` carries JSON keyed on `event`.
    Coalesce successive transcript deltas into the current caller row. Keep all
    control-event handling in one function so payload details live in exactly
    one place.

    ```js theme={null}
    function handleEvent(msg, playback, completeOpening) {
      // ⚠️ Inbound server frames are keyed on `event`, NOT `type`. (Your
      // outbound control frames, e.g. configure, are keyed on `type`.) Switching
      // on `msg.type` here is the #1 Omni bug: every frame falls through.
      switch (msg.event) {
        case "hello":            // connection accepted, before session_started
        case "session_started":  // socket is live; wait for configured
          break;

        case "configured":
          if (msg.language_fallback || msg.language_active !== requestedConfiguration.language) {
            throw new Error("Required language was not served; close this session.");
          }
          if (msg.voice_id !== requestedConfiguration.voiceId) {
            throw new Error("Required voice was not served; close this session.");
          }
          console.info("effective configuration", msg);
          break;

        case "flush":
          // Barge-in: you started speaking, so the agent's queued audio is
          // now stale. Drop it immediately for a snappy turn-handoff.
          playback.port.postMessage("clear");
          completeOpening(); // a real flush must never leave capture muted
          break;

        case "transfer_to_human": // agent decided to escalate
        case "session_end":       // server is closing the session
          break;

        default:
          // Ignore unknown additive events.
          break;
      }
    }
    ```

    <Info>
      The exact fields on each event are defined in the
      [Omni wire protocol](/realtime/omni-protocol). Server JSON never uses `type`
      as its discriminator.
    </Info>
  </Step>
</Steps>

## Run it

Drop `capture-processor.js`, `playback-processor.js`, and an `index.html` (a
**Start** button calling `start()`) in one folder and serve it:

```bash theme={null}
npx serve .   # then open http://localhost:3000
```

Click **Start**, allow the mic, and say hello. You should hear the agent reply
within a few hundred milliseconds and see the transcript fill in. Talk over it, the agent's audio should cut out as soon as you speak (that's the `flush` →
`clear` path).

## Barge-in, latency & quality

* **Barge-in** is the difference between a demo and a product. The server detects
  your speech and sends `flush`; your only job is to **stop playing queued agent
  audio right then**, that's the single `playback.port.postMessage("clear")`
  call. Don't wait for the socket to drain.
* **Keep frames small (\~20 ms).** Smaller frames lower latency; much smaller and
  you pay per-message overhead. 480 samples @ 24 kHz is a good default.
* **Use AEC, but do not treat it as an opening gate.** Keep
  `echoCancellation: true`, and independently replace mic samples with digital
  silence through the consent/greeting drain. AEC can need hundreds of
  milliseconds to converge and cannot guarantee the opening survives.
* **Don't add your own jitter buffer on top of the worklet ring**, the ring is
  already the buffer. Extra queuing only adds latency.
* **Resume the AudioContext from a user gesture.** Browsers start it suspended;
  call `ctx.resume()` inside the click handler if playback is silent.

## Troubleshooting

| Symptom | Likely cause | Fix |
| - | - | - |
| **Connects and `configured` acks, but the agent never speaks (zero turns, no error)** | **`configure` keyed on `event` instead of `type`**, the engine drops the unknown key and runs with no persona | Send `{"type":"configure",…}` (not `"event"`). Parse inbound on `event`. The `@pyai/sdk` `omni.connect()` helper handles both sides. |
| Agent sounds high/low-pitched or sped up | Sample-rate mismatch | Ensure the `AudioContext`, worklets, and `?rate=` all agree on 24 kHz; if the context forced 48 kHz, decimate/upsample 2:1 |
| Connection closes immediately | Bad credential, missing scope, or a malformed `session_label` | Check the close code against [Errors & limits](/errors-and-limits); `4401` = bad key, `4403` = missing `omni:session` scope, `400 invalid_session_label` = malformed `session_label` |
| `403 origin_not_allowed` on close | Publishable token origin not allow-listed | Add your origin in the console (production tokens only) |
| No mic prompt / `getUserMedia` throws | Insecure context | Serve over `http://localhost` or HTTPS, not `file://` or a LAN IP |
| Opening greeting clips or mutes itself | Mic/speaker energy reached Omni before turn-0 playback drained | Send digital silence through the protected opening, then rearm mic audio after drain; keep AEC enabled |
| Agent hears itself on later replies | AEC off, or playback feeding back | Set `echoCancellation: true`; never audibly monitor `capture`; keep the short reply warm-up |
| Choppy or robotic playback | Ring buffer starved or overrun | Confirm 20 ms frames; check the socket isn't backpressured |
| `429 concurrency_limit_exceeded` | Too many live sessions | Close old sockets; raise the limit on your plan |

## Make the agent speak first (greeting)

By default the **caller speaks first**, Omni listens, then replies. To open the
call with a greeting ("Hi, thanks for calling, how can I help?"), set the native
`greeting` field in your `configure` frame, it's **live**, and the engine speaks
it as turn 0. The **pre-roll pattern** below is an alternative for when you'd
rather render the opening audio client-side:

<Steps>
  <Step title="Synthesize the greeting once with Speak">
    Call `POST /v1/audio/speech` with `response_format: "pcm"` at your session
    rate and cache the bytes (it's the same line every call).
  </Step>

  <Step title="Play it the moment the session is live">
    On `session_started`, push the cached PCM into your playback ring while the
    uplink carries real-time digital silence. Rearm mic samples only after the
    ring reports drained. The agent appears to speak first; normal turn-taking +
    barge-in take over from there.
  </Step>
</Steps>

<Note>
  The native `greeting` field in the `configure` frame is **live**: set it and the
  engine speaks your opening line as turn 0 with no extra hop, so the pre-roll
  above is optional. For agent profiles, set **Greeting message** in the
  [console Agents builder](/guides/agent-greeting) instead of sending `greeting` every
  connect. The connect URL's `session_label` is still just an opaque correlation tag
  when you are not using an agent profile. Native and pre-rendered openings use
  the same client invariant: playback ready first, silence on the mic uplink
  through drain, then normal later barge-in.
</Note>

## Next steps

<CardGroup cols={2}>
  <Card title="Omni wire protocol" href="/realtime/omni-protocol">Exact event payloads, close codes, and golden frames.</Card>
  <Card title="Phone agent with Twilio" href="/guides/twilio-voice-agent">Bridge the same Omni session to a real phone number.</Card>
  <Card title="FreeSWITCH integration" href="/guides/freeswitch-voice-agent">Fork SIP/PSTN audio into Omni.</Card>
  <Card title="Errors & limits" href="/errors-and-limits">Close codes, rate limits, and concurrency.</Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.