partial text in your UI; commit the corrected final text for each utterance.
Use this for live captions, voice search, and agent assistance. Your key needs
the hear:stream scope.
Run the existing microphone-caption example. It keeps the key on a local
backend, verifies your test project, requests and checks 16 kHz capture, and
permits one session of at most 60 seconds. Scaffold it without a clone:Or browse it:
browser-hear-live-captions.Endpoint:
wss://api.pyai.com/v1/audio/transcriptions/stream?protocol=pyai-hear-v1. The full wire
protocol, query params, the JSON frame schema, the {"type":"commit"} flush, and
close codes, is published in the API reference
(GET /v1/audio/transcriptions/stream, generated from
https://api.pyai.com/openapi.json) and summarized in
Wire protocol below.Transcribe audio from an existing telephone app
Use Hear streaming for incoming call audio; you do not need to create a new call. Decode G.711 μ-law to PCM16 mono on your backend and declaresample_rate=8000.
Do not send base64 carrier envelopes or compressed μ-law bytes directly to Hear.
The microphone example above captures 16 kHz audio; it is not a carrier adapter.
Keep credentials on your authenticated backend: Omni browser session tokens do
not authorize Hear. For speech output in the same telephone app, use the existing
bounded streaming TTS example,
which returns 8 kHz G.711 μ-law. Neither example places a call.
How it fits together
One socket carries two things: you send binary PCM16 audio frames up, and receive text JSON result frames down. Results come in two flavors, fast-but-revisable partials and stable finals, and your UI’s whole job is to show the former without committing to them, then lock in the latter.Streaming vs. batch, pick the right tool
Audio format: PCM16 at 8 or 16 kHz
Hear streaming consumes PCM16 little-endian mono at 8 or 16 kHz. Setsample_rate to the actual rate of the bytes you send; the default is
16000. Hear converts 8 kHz input before recognition and endpointing.
Other sample rates and compressed encodings are rejected before the WebSocket
opens with 400 unsupported_audio_format.
- Browser mic: request a 16 kHz
AudioContextand inspect its actualsampleRate. If it differs, resample with an anti-aliasing filter before sending. Declaring 16 kHz does not convert 48 kHz audio. - Telephony is usually 8 kHz. Decode μ-law to PCM16 first, then send it with
sample_rate=8000, or resample it to 16 kHz and declaresample_rate=16000. - Frame size: ~20 ms per message (160 samples at 8 kHz; 320 at 16 kHz) keeps latency low without paying per-message overhead.
capture-processor.js, Float32 → PCM16 @ 16 kHz
The two-pass display pattern
This is the heart of a good live-transcription UI. Treat the transcript as a list of committed final lines plus one pending partial at the end:- A partial arrives → render it greyed/italic as the “current” line. Partials are revisable, replace, never append.
- A newer partial arrives → overwrite the same pending line.
- A final arrives → commit it as solid text and clear the pending line.
- Repeat for the next phrase.
transcript_gap identifies an
utterance whose text was withheld. Replace any pending caption for that ID with
a missing-transcript indicator.
Two-pass transcript renderer
Wire protocol
Connect. Open a WebSocket to the streaming transcription endpoint and pick your audio format with query params:protocol=pyai-hear-v1 selects the published bare frame names used throughout
this guide. It is the only Hear streaming protocol; omitting the parameter uses
the same pyai-hear-v1 default.
From a trusted server, authenticate the upgrade with your key as a subprotocol:
Sec-WebSocket-Protocol: pyai.v1, pyai-key.<API_KEY>, server-side clients may use
?api_key= instead. The key needs the hear:stream scope. Never put a long-lived key in a browser;
proxy browser audio through your authenticated backend. Omni session tokens
carry omni:session and cannot authorize Hear.
With an explicit English hint, final transcripts apply number formatting by default (phones,
currency, dates, ordinals). Send
numerals=false to keep spoken form, or
numerals=true to force digits. Interim partials are never rewritten. Automatic mode may retain spoken number forms.Add &smart_format=true (or smartFormat: true in the SDK) for punctuation
and sentence capitalization on finals only. Optional extras, also finals-only
and off by default: dictation (spoken period / comma / new paragraph /
question mark), drop_fillers (um / uh / er; do not enable on legal
or compliance audio by default), and vocabulary for known terms. Send terms
for one session or explicitly enable the stored hear_stream profile. See
Format Hear transcripts.
These English cleanup flags are conservative/no-op where unsupported on
non-English transcripts. Formatting flags do not select the recognition language; an explicit language
hint constrains it, while omitted language or auto uses automatic detection.sample_rate=8000 or sample_rate=16000), ~20 ms each. To force-finalize the
current utterance, e.g. when your own VAD detects end-of-turn, send a JSON text
frame {"type":"commit"}. Closing the socket also flushes a final.
Server → client. JSON text frames, emitted with bare type names:
utterance_id groups the partials and finals of one phrase. t_ms is elapsed
server time when the event is created; it is not a word timestamp. audio_ms
is the utterance’s active-speech length used for metering.
endpoint_reason is peak_te_early when the high-confidence path fires or
silence_backstop when the bounded fallback closes the turn. Log it when tuning
conversational pacing.
Word timestamps and two-speaker calls
For a stereo call, open one mono stream per speaker. Start both streams at the same position in the recording and send each channel’s silence. If one stream begins later, add its known offset to its timestamps. Order sentences by utterancestart. A later sentence can arrive first.
Display final.text for the sentence. On accounts enabled for per-word
coverage, the normal connection keeps delivering captions when a word’s time
is unavailable. In this example the application can highlight “Hello” at
12.12 seconds and still display the full sentence.
words in the order received. Highlight a word only when both offsets
are numbers and end > start. Times are seconds from the start of audio on
that socket. If every word is unmeasured, the final reports
word_timing_status: "unavailable". You can still position the sentence
using its utterance bounds. Some streams return text-only finals.
Number formatting can group several words under one measured span. With
redaction enabled, both the sentence and the word entries contain protected
text. A multiword mask can appear more than once in words.
All policies attempt timing recovery. A recognition interruption or a redaction
gap still makes the session incomplete.
With redaction enabled, the first speech draft can produce a
partial with
redaction_status: "pending" and the fixed text [REDACTED]. This signals
that speech is being checked. Replace it with the next caption for the same
utterance_id. Longer turns can also produce a verified prefix. Save the
corrected final and remove the pending indicator on a transcript_gap.
See live redaction and transcript gaps.
Close codes. 1000 normal · 1008 auth/policy (bad key or missing
hear:stream scope) · 1011 engine error · 1013 temporarily busy · 4429 over
the concurrency cap. For a busy connection, follow the capacity retry guidance.
An error frame with code: recognition_interrupted, followed by close 1011,
means accepted audio could not be fully recovered. Treat the session as
incomplete and start a new stream. An earlier final completes its own
utterance; it does not prove that later audio was recognized. Do not treat an
error close as success because an earlier final exists.
PyAI tries internal recovery before reporting an interrupted session. If recovery
cannot complete, no further transcript results or automatic Recap are produced
from that session. Reconnecting creates a new audio timeline and does not recover
audio from the previous socket.
Wire it up
Open the socket, stream capture frames as binary, and route text frames through one handler. Authenticate the upgrade with the key as a subprotocol, the server-side authentication pattern. For a browser, connect to your own authenticated WebSocket proxy and keep the PyAI credential on that backend.Connect + route frames
partial as revisable UI. Commit final. speech_final arrives first
with endpoint_reason; treat it as a stable preview of the same utterance, not
a second committed line.
Force-finalize and handle frames
The endpoint path, the frame schema, and the
{"type":"commit"} control message
are part of the published contract, see the API reference
(GET /v1/audio/transcriptions/stream) or the Wire protocol
summary above. Keep all framing inside handleFrame so adding a field later is a
one-place change.speech_final and final describe the same completed utterance.
speech_final is the immediate stable result; final follows with the
full-context correction. If your UI commits both, key rows by utterance_id and
replace the first value. Do not append both as separate transcript lines.Latency expectations
- Display partial captions when they arrive. Protected streams can deliver an entire utterance as a final without sending a preview.
- A pause or a commit triggers finalization. Requested redaction adds a verification step. Keep reading after the speaker stops and after sending the end-of-input frame so the last protected final has time to arrive.
- Keep frames small and steady. 20 ms frames sent as they’re produced minimize end-to-end latency; don’t batch several seconds of audio into one message.
- Don’t add your own buffering on top. Send frames straight from the capture worklet; extra queues only add delay.
Sizing concurrent calls
GET /v1/me reports two ceilings. limits.concurrency applies to the current
API key. limits.org_concurrency applies across the organization’s keys.
Available shared realtime capacity can be lower than either ceiling at a given
moment.
A two-speaker client using one mono socket per speaker needs two available
streams for each call. A target of 100 simultaneous calls therefore requires
200 available streams. Validate your expected peak before sizing a rollout.
If a connection attempt returns HTTP 429, wait at least the number of seconds
in Retry-After when the WebSocket client exposes the handshake response.
A busy server can also accept the upgrade and immediately close with 1013.
Start with a short wait if the client cannot read Retry-After. On repeated rejections, increase the delay
exponentially and add random jitter so callers reconnect at different times.
If both streams are required to start a call, release a lone open socket before
retrying the pair. Keep any audio you need to transcribe while waiting.
Troubleshooting
Next steps
Telephony audio (8 kHz μ-law)
Stream phone-call audio into Hear: native μ-law and the exact resample ratios.
Timestamped recording jobs
When the audio is finished, batch-transcribe with word/segment offsets and diarization.
Browser voice agent
The same PCM16 capture pipeline, driving a full-duplex Omni agent.
Errors & limits
Close codes, rate limits, and concurrency.