Choose the website path
- Website or in-app receptionist: follow this guide. Visitors grant microphone access, start a session and can interrupt later replies. Keep microphone audio gated during the opening greeting as described below.
- Product questions: connect your knowledge base. Confirm that the documents are indexed before claiming the receptionist knows them.
- Booking and other actions: use backend tools and your application’s permissions. Saving an Agent profile alone does not connect your systems.
- Answer incoming phone calls: use the separate phone receptionist guide.
GET /v1/me, agree an application
budget, and use the current pricing. Token expiry is
not a spending cap. This guide does not establish a latency or accuracy benchmark.
Run the complete example. Scaffold this guide’s full, CI-tested code in one
command, no clone:Or browse it:
omni-browser-widget.How it fits together
One binary envelope shares the socket: byte0x01 carries PCM16 audio, 0x02
carries plain UTF-8 caller-transcript deltas, and 0x03 carries UTF-8 JSON control
payloads. Caller audio must also include the leading 0x01; outbound control
frames use 0x03 plus JSON keyed by type. See the wire protocol before
implementing a transport.
This guide is correct on transport, codecs, resampling, and the event behaviors
you build against. The exact JSON payloads for each event are defined by the
Omni wire protocol, which is the source of truth, we
isolate event handling behind a single function so you can fill in field names
once you’ve checked the reference.
Prerequisites
1
A server-side key and confirmed project
Omni is zero-state: there’s nothing to create first. The session is
authorized by your key’s org, and the agent’s behavior travels in a
configure frame after connect. Keep the key on your backend and verify its
environment, scopes, limits and billing using GET /v1/me. Do not assume an
existing key or saved profile belongs to a test project.2
A browser page and an authenticated backend
getUserMedia requires a secure context, which includes http://localhost.
Serve the page over HTTPS in production. A static server can host the page
locally, but it does not replace the token broker or its authentication and quota policy.Fail-closed browser token broker
For a runnable fictional product Q&A → qualification → one backend action → interruption example, see the synthetic website demonstration. Its 18 November 2026 result separates the real browser run from mocked credit failures and the untested managed-widget scenario. It binds only to localhost; retain the production authentication and quota requirements below for an internet-facing broker. This integration template returns 503 and never mints a token until your application supplies all three policy functions. It is not a ready-to-deploy public token endpoint. Use your existing authenticated session (for example a secure, HttpOnly cookie), authorize that user for the configured project, and atomically consume a per-user quota in your shared store. Do not trust a user ID supplied in the request body or an unverified header. Origin checks are an additional browser boundary, not user authentication. The broker checks/v1/me with the same server key on every request, before
consuming quota or minting a token. It refuses a different project, inactive
identity or failed lookup. PYAI_PROJECT_ID is an expected identity, not a way
to select the project for a key. A key rotated to another project must not use
the previous project’s authorization or quota policy.
Keep APP_ORIGIN an exact HTTPS origin, with no path or trailing
slash; localhost HTTP is permitted for development. The policy must include
application/session spending limits; token expiry alone does not cap usage.
The minting endpoint accepts only an active server API key for that exact
project. A browser token cannot mint or renew another token. A server key with
a positive monthly spending cap is rejected with
403 key_budget_delegation_unsupported: session tokens do not share the
parent key’s spending counter. Do not remove the cap just to make this work.
Keep the bounded server-side workflow, or establish an approved application
budget and enforcement policy before choosing browser sessions.
omni:session, is limited to one concurrent session,
and is locked to allowed_origins. The default lifetime is 60 seconds and the
maximum is 600 seconds. Mint it immediately before connecting; do not store it
in local storage, analytics, logs, or a reusable configuration file. Token expiry
is not a spending cap or maximum call duration. Apply application session limits
and check the selected project billing context before starting audio.
Pick one sample rate and stick to it
The single biggest source of “it sounds like chipmunks/robots” bugs is a sample rate mismatch. Omni speaks PCM16 little-endian; for browser/WebRTC use 24 kHz, declared on the URL as?format=pcm16&rate=24000.
The clean trick: ask the browser for a 24 kHz AudioContext so the mic capture,
the worklets, and the wire all agree, no resampling needed.
if away and called out in the code.
Build it
1
Capture worklet: mic Float32 → PCM16
An
AudioWorkletProcessor runs on the audio thread and hands you 128-sample
blocks. Buffer them into ~20 ms frames (480 samples @ 24 kHz), convert
Float32 [-1, 1] to little-endian Int16, and post the bytes to the main
thread.capture-processor.js
2
Playback worklet: a ring buffer of agent audio
Agent audio arrives in bursts; speakers consume it at a steady 24 kHz. Bridge
the two with a ring buffer. Crucially, expose a
clear message, that’s how
barge-in stays snappy (next step).playback-processor.js
3
Connect to Omni and wire the audio loop
Open the WebSocket with the short-lived session token as a subprotocol,
forward capture frames as binary, and feed
binary replies into the playback ring. The official
@pyai/sdk ships
realtimeURL / realtimeSubprotocol helpers so you don’t hand-build the URL.4
Handle events: transcript + barge-in
0x02 carries plain UTF-8 caller text; 0x03 carries JSON keyed on event.
Coalesce successive transcript deltas into the current caller row. Keep all
control-event handling in one function so payload details live in exactly
one place.The exact fields on each event are defined in the
Omni wire protocol. Server JSON never uses
type
as its discriminator.Run it
Dropcapture-processor.js, playback-processor.js, and an index.html (a
Start button calling start()) in one folder and serve it:
flush →
clear path).
Barge-in, latency & quality
- Barge-in is the difference between a demo and a product. The server detects
your speech and sends
flush; your only job is to stop playing queued agent audio right then, that’s the singleplayback.port.postMessage("clear")call. Don’t wait for the socket to drain. - Keep frames small (~20 ms). Smaller frames lower latency; much smaller and you pay per-message overhead. 480 samples @ 24 kHz is a good default.
- Use AEC, but do not treat it as an opening gate. Keep
echoCancellation: true, and independently replace mic samples with digital silence through the consent/greeting drain. AEC can need hundreds of milliseconds to converge and cannot guarantee the opening survives. - Don’t add your own jitter buffer on top of the worklet ring, the ring is already the buffer. Extra queuing only adds latency.
- Resume the AudioContext from a user gesture. Browsers start it suspended;
call
ctx.resume()inside the click handler if playback is silent.
Troubleshooting
Make the agent speak first (greeting)
Use the nativegreeting field on a saved agent or inline configure.
Greetings and recording disclosure is the setup guide.
The browser must have playback ready and send real-time digital silence on the
microphone uplink until the opening queue drains; then restore normal capture
and interruption handling. Keep this opening gate even with echo cancellation.
Alternative: a pre-rendered client greeting
Alternative: a pre-rendered client greeting
If your application already owns opening audio, synthesize it once with Speak,
cache PCM at the session output rate, and play it after the session is ready.
Keep the uplink silent until playback drains. Do not also configure a native
greeting, or the caller can hear two openings.
Next steps
Omni wire protocol
Exact event payloads, close codes, and golden frames.
Phone agent with Twilio
Bridge the same Omni session to a real phone number.
FreeSWITCH integration
Fork SIP/PSTN audio into Omni.
Errors & limits
Close codes, rate limits, and concurrency.