Skip to main content
Build a website voice receptionist that visitors talk to through their browser microphone and speakers. No phone number, carrier account or phone call is required. An Omni session carries the conversation; the browser renders its live transcript and plays spoken replies. The browser client is a single HTML page using Web Audio and a WebSocket. A separate authenticated backend issues short-lived session tokens. The backend must enforce access and usage limits before a visitor can start a session.

Choose the website path

  • Website or in-app receptionist: follow this guide. Visitors grant microphone access, start a session and can interrupt later replies. Keep microphone audio gated during the opening greeting as described below.
  • Product questions: connect your knowledge base. Confirm that the documents are indexed before claiming the receptionist knows them.
  • Booking and other actions: use backend tools and your application’s permissions. Saving an Agent profile alone does not connect your systems.
  • Answer incoming phone calls: use the separate phone receptionist guide.
Omni sessions consume usage even though no carrier call is placed. Inspect the server key’s project and billing context with GET /v1/me, agree an application budget, and use the current pricing. Token expiry is not a spending cap. This guide does not establish a latency or accuracy benchmark.
Run the complete example. Scaffold this guide’s full, CI-tested code in one command, no clone:
Or browse it: omni-browser-widget.

How it fits together

One binary envelope shares the socket: byte 0x01 carries PCM16 audio, 0x02 carries plain UTF-8 caller-transcript deltas, and 0x03 carries UTF-8 JSON control payloads. Caller audio must also include the leading 0x01; outbound control frames use 0x03 plus JSON keyed by type. See the wire protocol before implementing a transport.
This guide is correct on transport, codecs, resampling, and the event behaviors you build against. The exact JSON payloads for each event are defined by the Omni wire protocol, which is the source of truth, we isolate event handling behind a single function so you can fill in field names once you’ve checked the reference.
Protected opening invariant. Browser echo cancellation is not a startup gate. Resume the playback graph before sending configure, buffer any server PCM that arrives before it is ready, and send real-time digital silence instead of microphone or speaker energy while the turn-0 consent line and greeting play. Rearm microphone audio only after that queue drains. Later replies should keep normal barge-in after a short echo-cancellation warm-up.

Prerequisites

1

A server-side key and confirmed project

Omni is zero-state: there’s nothing to create first. The session is authorized by your key’s org, and the agent’s behavior travels in a configure frame after connect. Keep the key on your backend and verify its environment, scopes, limits and billing using GET /v1/me. Do not assume an existing key or saved profile belongs to a test project.
2

A browser page and an authenticated backend

getUserMedia requires a secure context, which includes http://localhost. Serve the page over HTTPS in production. A static server can host the page locally, but it does not replace the token broker or its authentication and quota policy.
Keep test and live API keys on your backend, including during development. Use the hosted website widget from Agent → Connect, or have your backend call POST /v1/omni/sessions and hand the browser only the returned short-lived, origin-locked session token.

Fail-closed browser token broker

For a runnable fictional product Q&A → qualification → one backend action → interruption example, see the synthetic website demonstration. Its 18 November 2026 result separates the real browser run from mocked credit failures and the untested managed-widget scenario. It binds only to localhost; retain the production authentication and quota requirements below for an internet-facing broker. This integration template returns 503 and never mints a token until your application supplies all three policy functions. It is not a ready-to-deploy public token endpoint. Use your existing authenticated session (for example a secure, HttpOnly cookie), authorize that user for the configured project, and atomically consume a per-user quota in your shared store. Do not trust a user ID supplied in the request body or an unverified header. Origin checks are an additional browser boundary, not user authentication. The broker checks /v1/me with the same server key on every request, before consuming quota or minting a token. It refuses a different project, inactive identity or failed lookup. PYAI_PROJECT_ID is an expected identity, not a way to select the project for a key. A key rotated to another project must not use the previous project’s authorization or quota policy. Keep APP_ORIGIN an exact HTTPS origin, with no path or trailing slash; localhost HTTP is permitted for development. The policy must include application/session spending limits; token expiry alone does not cap usage. The minting endpoint accepts only an active server API key for that exact project. A browser token cannot mint or renew another token. A server key with a positive monthly spending cap is rejected with 403 key_budget_delegation_unsupported: session tokens do not share the parent key’s spending counter. Do not remove the cap just to make this work. Keep the bounded server-side workflow, or establish an approved application budget and enforcement policy before choosing browser sessions.
The browser requests a session from its same-origin backend and stops on any error. It never receives the server key:
The token carries only omni:session, is limited to one concurrent session, and is locked to allowed_origins. The default lifetime is 60 seconds and the maximum is 600 seconds. Mint it immediately before connecting; do not store it in local storage, analytics, logs, or a reusable configuration file. Token expiry is not a spending cap or maximum call duration. Apply application session limits and check the selected project billing context before starting audio.

Pick one sample rate and stick to it

The single biggest source of “it sounds like chipmunks/robots” bugs is a sample rate mismatch. Omni speaks PCM16 little-endian; for browser/WebRTC use 24 kHz, declared on the URL as ?format=pcm16&rate=24000. The clean trick: ask the browser for a 24 kHz AudioContext so the mic capture, the worklets, and the wire all agree, no resampling needed.
Most browsers honor this. If yours pins the context to its hardware rate (commonly 48 kHz), you have two options: keep the 24 kHz context and let the worklet see 24 kHz directly, or run at 48 kHz and decimate 2:1 on capture (take every other sample) / upsample 2:1 on playback (duplicate each sample). The math is exact because 48000 / 24000 = 2. We use the 24 kHz context path; the fallback is one if away and called out in the code.

Build it

1

Capture worklet: mic Float32 → PCM16

An AudioWorkletProcessor runs on the audio thread and hands you 128-sample blocks. Buffer them into ~20 ms frames (480 samples @ 24 kHz), convert Float32 [-1, 1] to little-endian Int16, and post the bytes to the main thread.
capture-processor.js
2

Playback worklet: a ring buffer of agent audio

Agent audio arrives in bursts; speakers consume it at a steady 24 kHz. Bridge the two with a ring buffer. Crucially, expose a clear message, that’s how barge-in stays snappy (next step).
playback-processor.js
3

Connect to Omni and wire the audio loop

Open the WebSocket with the short-lived session token as a subprotocol, forward capture frames as binary, and feed binary replies into the playback ring. The official @pyai/sdk ships realtimeURL / realtimeSubprotocol helpers so you don’t hand-build the URL.
Frame-key asymmetry, the #1 Omni bug. Your outbound control frames (configure, dtmf) are keyed on type; the inbound server frames (hello, session_started, configured, flush, …) are keyed on event. The edge is transparent, so a mis-keyed {"event":"configure"} is acked but silently dropped, you get a connected session that never speaks, with no error. Send {"type":"configure",…}; parse inbound on event. The @pyai/sdk omni.connect() helper does both for you.
4

Handle events: transcript + barge-in

0x02 carries plain UTF-8 caller text; 0x03 carries JSON keyed on event. Coalesce successive transcript deltas into the current caller row. Keep all control-event handling in one function so payload details live in exactly one place.
The exact fields on each event are defined in the Omni wire protocol. Server JSON never uses type as its discriminator.

Run it

Drop capture-processor.js, playback-processor.js, and an index.html (a Start button calling start()) in one folder and serve it:
Click Start, allow the mic, and say hello. You should hear the agent reply within a few hundred milliseconds and see the transcript fill in. Talk over it, the agent’s audio should cut out as soon as you speak (that’s the flush → clear path).

Barge-in, latency & quality

  • Barge-in is the difference between a demo and a product. The server detects your speech and sends flush; your only job is to stop playing queued agent audio right then, that’s the single playback.port.postMessage("clear") call. Don’t wait for the socket to drain.
  • Keep frames small (~20 ms). Smaller frames lower latency; much smaller and you pay per-message overhead. 480 samples @ 24 kHz is a good default.
  • Use AEC, but do not treat it as an opening gate. Keep echoCancellation: true, and independently replace mic samples with digital silence through the consent/greeting drain. AEC can need hundreds of milliseconds to converge and cannot guarantee the opening survives.
  • Don’t add your own jitter buffer on top of the worklet ring, the ring is already the buffer. Extra queuing only adds latency.
  • Resume the AudioContext from a user gesture. Browsers start it suspended; call ctx.resume() inside the click handler if playback is silent.

Troubleshooting

Make the agent speak first (greeting)

Use the native greeting field on a saved agent or inline configure. Greetings and recording disclosure is the setup guide. The browser must have playback ready and send real-time digital silence on the microphone uplink until the opening queue drains; then restore normal capture and interruption handling. Keep this opening gate even with echo cancellation.
If your application already owns opening audio, synthesize it once with Speak, cache PCM at the session output rate, and play it after the session is ready. Keep the uplink silent until playback drains. Do not also configure a native greeting, or the caller can hear two openings.

Next steps

Omni wire protocol

Exact event payloads, close codes, and golden frames.

Phone agent with Twilio

Bridge the same Omni session to a real phone number.

FreeSWITCH integration

Fork SIP/PSTN audio into Omni.

Errors & limits

Close codes, rate limits, and concurrency.