Skip to main content
PyAI Hear streaming transcribes audio as it arrives over a WebSocket. Send binary PCM16 frames and receive JSON transcript events. Replace revisable partial text in your UI; commit the corrected final text for each utterance. Use this for live captions, voice search, and agent assistance. Your key needs the hear:stream scope.
Run the existing microphone-caption example. It keeps the key on a local backend, verifies your test project, requests and checks 16 kHz capture, and permits one session of at most 60 seconds. Scaffold it without a clone:
Or browse it: browser-hear-live-captions.
Endpoint: wss://api.pyai.com/v1/audio/transcriptions/stream?protocol=pyai-hear-v1. The full wire protocol, query params, the JSON frame schema, the {"type":"commit"} flush, and close codes, is published in the API reference (GET /v1/audio/transcriptions/stream, generated from https://api.pyai.com/openapi.json) and summarized in Wire protocol below.

Transcribe audio from an existing telephone app

Use Hear streaming for incoming call audio; you do not need to create a new call. Decode G.711 μ-law to PCM16 mono on your backend and declare sample_rate=8000. Do not send base64 carrier envelopes or compressed μ-law bytes directly to Hear. The microphone example above captures 16 kHz audio; it is not a carrier adapter. Keep credentials on your authenticated backend: Omni browser session tokens do not authorize Hear. For speech output in the same telephone app, use the existing bounded streaming TTS example, which returns 8 kHz G.711 μ-law. Neither example places a call.

How it fits together

One socket carries two things: you send binary PCM16 audio frames up, and receive text JSON result frames down. Results come in two flavors, fast-but-revisable partials and stable finals, and your UI’s whole job is to show the former without committing to them, then lock in the latter.

Streaming vs. batch, pick the right tool

If the audio has already finished (a recording, a voicemail, a call you’ve hung up on), use timestamped batch jobs: they provide diarization and avoid maintaining a live socket. Compare current metering on the pricing page.

Audio format: PCM16 at 8 or 16 kHz

Hear streaming consumes PCM16 little-endian mono at 8 or 16 kHz. Set sample_rate to the actual rate of the bytes you send; the default is 16000. Hear converts 8 kHz input before recognition and endpointing. Other sample rates and compressed encodings are rejected before the WebSocket opens with 400 unsupported_audio_format.
  • Browser mic: request a 16 kHz AudioContext and inspect its actual sampleRate. If it differs, resample with an anti-aliasing filter before sending. Declaring 16 kHz does not convert 48 kHz audio.
  • Telephony is usually 8 kHz. Decode μ-law to PCM16 first, then send it with sample_rate=8000, or resample it to 16 kHz and declare sample_rate=16000.
  • Frame size: ~20 ms per message (160 samples at 8 kHz; 320 at 16 kHz) keeps latency low without paying per-message overhead.
The processor below converts samples to PCM16 and frames them; it does not resample. Run it in a context already operating at 16 kHz, or resample upstream.
capture-processor.js, Float32 → PCM16 @ 16 kHz

The two-pass display pattern

This is the heart of a good live-transcription UI. Treat the transcript as a list of committed final lines plus one pending partial at the end:
  1. A partial arrives → render it greyed/italic as the “current” line. Partials are revisable, replace, never append.
  2. A newer partial arrives → overwrite the same pending line.
  3. A final arrives → commit it as solid text and clear the pending line.
  4. Repeat for the next phrase.
Protected streams can go straight to a final. A transcript_gap identifies an utterance whose text was withheld. Replace any pending caption for that ID with a missing-transcript indicator.
Two-pass transcript renderer

Wire protocol

Connect. Open a WebSocket to the streaming transcription endpoint and pick your audio format with query params:
protocol=pyai-hear-v1 selects the published bare frame names used throughout this guide. It is the only Hear streaming protocol; omitting the parameter uses the same pyai-hear-v1 default.
Omit language or set language=auto to detect the spoken language automatically. To pin recognition deliberately, use en, es, fr, de, hi, it, pt, or nl. Unsupported explicit values return 400 unsupported_language. This endpoint does not currently return language-identification or decoder-confidence fields.
From a trusted server, authenticate the upgrade with your key as a subprotocol: Sec-WebSocket-Protocol: pyai.v1, pyai-key.<API_KEY>, server-side clients may use ?api_key= instead. The key needs the hear:stream scope. Never put a long-lived key in a browser; proxy browser audio through your authenticated backend. Omni session tokens carry omni:session and cannot authorize Hear.
With an explicit English hint, final transcripts apply number formatting by default (phones, currency, dates, ordinals). Send numerals=false to keep spoken form, or numerals=true to force digits. Interim partials are never rewritten. Automatic mode may retain spoken number forms.Add &smart_format=true (or smartFormat: true in the SDK) for punctuation and sentence capitalization on finals only. Optional extras, also finals-only and off by default: dictation (spoken period / comma / new paragraph / question mark), drop_fillers (um / uh / er; do not enable on legal or compliance audio by default), and vocabulary for known terms. Send terms for one session or explicitly enable the stored hear_stream profile. See Format Hear transcripts. These English cleanup flags are conservative/no-op where unsupported on non-English transcripts. Formatting flags do not select the recognition language; an explicit language hint constrains it, while omitted language or auto uses automatic detection.
Client → server. Stream binary audio frames continuously (PCM16 little-endian mono at sample_rate=8000 or sample_rate=16000), ~20 ms each. To force-finalize the current utterance, e.g. when your own VAD detects end-of-turn, send a JSON text frame {"type":"commit"}. Closing the socket also flushes a final.
The only explicit finalization control is the JSON frame {"type":"commit"}. Do not send a plain-text sentinel or another control name.
Tune automatic endpointing. Add endpointing_ms to the connect URL, for example &endpointing_ms=800. This is the minimum trailing-pause length before an utterance may end, not the exact pause that forces a final. Turn detection may wait longer. An utterance always completes within max(endpointing_ms, 1500 ms) of streamed silence, with exactly one endpoint for each continuous silence run.Keep sending audio frames, including silence, while the caller pauses. The timer advances from incoming frames; if audio stops, the pause timer stops too. Change the floor without reconnecting by sending {"type":"config","endpointing_ms":800}.Both forms return config_ack with the applied floor, ceiling, score interval, and warnings. Assert that warnings is empty. clamped_to_range, not_a_number, or unknown_config_field means the setting did not apply exactly as sent, but the session stays open. You can always force immediate end-of-turn with {"type":"commit"}.
Server → client. JSON text frames, emitted with bare type names: utterance_id groups the partials and finals of one phrase. t_ms is elapsed server time when the event is created; it is not a word timestamp. audio_ms is the utterance’s active-speech length used for metering. endpoint_reason is peak_te_early when the high-confidence path fires or silence_backstop when the bounded fallback closes the turn. Log it when tuning conversational pacing.

Word timestamps and two-speaker calls

For a stereo call, open one mono stream per speaker. Start both streams at the same position in the recording and send each channel’s silence. If one stream begins later, add its known offset to its timestamps. Order sentences by utterance start. A later sentence can arrive first. Display final.text for the sentence. On accounts enabled for per-word coverage, the normal connection keeps delivering captions when a word’s time is unavailable. In this example the application can highlight “Hello” at 12.12 seconds and still display the full sentence.
Keep words in the order received. Highlight a word only when both offsets are numbers and end > start. Times are seconds from the start of audio on that socket. If every word is unmeasured, the final reports word_timing_status: "unavailable". You can still position the sentence using its utterance bounds. Some streams return text-only finals. Number formatting can group several words under one measured span. With redaction enabled, both the sentence and the word entries contain protected text. A multiword mask can appear more than once in words. All policies attempt timing recovery. A recognition interruption or a redaction gap still makes the session incomplete. With redaction enabled, the first speech draft can produce a partial with redaction_status: "pending" and the fixed text [REDACTED]. This signals that speech is being checked. Replace it with the next caption for the same utterance_id. Longer turns can also produce a verified prefix. Save the corrected final and remove the pending indicator on a transcript_gap. See live redaction and transcript gaps.
Cue is unavailable. The Hear stream does not accept Cue grounding configuration, return grounding results, or emit Cue usage. Do not build against Cue fields.
Close codes. 1000 normal · 1008 auth/policy (bad key or missing hear:stream scope) · 1011 engine error · 1013 temporarily busy · 4429 over the concurrency cap. For a busy connection, follow the capacity retry guidance. An error frame with code: recognition_interrupted, followed by close 1011, means accepted audio could not be fully recovered. Treat the session as incomplete and start a new stream. An earlier final completes its own utterance; it does not prove that later audio was recognized. Do not treat an error close as success because an earlier final exists. PyAI tries internal recovery before reporting an interrupted session. If recovery cannot complete, no further transcript results or automatic Recap are produced from that session. Reconnecting creates a new audio timeline and does not recover audio from the previous socket.

Wire it up

Open the socket, stream capture frames as binary, and route text frames through one handler. Authenticate the upgrade with the key as a subprotocol, the server-side authentication pattern. For a browser, connect to your own authenticated WebSocket proxy and keep the PyAI credential on that backend.
Connect + route frames
Frames you will see while someone is talking:
Render partial as revisable UI. Commit final. speech_final arrives first with endpoint_reason; treat it as a stable preview of the same utterance, not a second committed line.
Force-finalize and handle frames
The endpoint path, the frame schema, and the {"type":"commit"} control message are part of the published contract, see the API reference (GET /v1/audio/transcriptions/stream) or the Wire protocol summary above. Keep all framing inside handleFrame so adding a field later is a one-place change.
speech_final and final describe the same completed utterance. speech_final is the immediate stable result; final follows with the full-context correction. If your UI commits both, key rows by utterance_id and replace the first value. Do not append both as separate transcript lines.

Latency expectations

  • Display partial captions when they arrive. Protected streams can deliver an entire utterance as a final without sending a preview.
  • A pause or a commit triggers finalization. Requested redaction adds a verification step. Keep reading after the speaker stops and after sending the end-of-input frame so the last protected final has time to arrive.
  • Keep frames small and steady. 20 ms frames sent as they’re produced minimize end-to-end latency; don’t batch several seconds of audio into one message.
  • Don’t add your own buffering on top. Send frames straight from the capture worklet; extra queues only add delay.

Sizing concurrent calls

GET /v1/me reports two ceilings. limits.concurrency applies to the current API key. limits.org_concurrency applies across the organization’s keys. Available shared realtime capacity can be lower than either ceiling at a given moment. A two-speaker client using one mono socket per speaker needs two available streams for each call. A target of 100 simultaneous calls therefore requires 200 available streams. Validate your expected peak before sizing a rollout. If a connection attempt returns HTTP 429, wait at least the number of seconds in Retry-After when the WebSocket client exposes the handshake response. A busy server can also accept the upgrade and immediately close with 1013. Start with a short wait if the client cannot read Retry-After. On repeated rejections, increase the delay exponentially and add random jitter so callers reconnect at different times. If both streams are required to start a call, release a lone open socket before retrying the pair. Keep any audio you need to transcribe while waiting.

Troubleshooting

Next steps

Telephony audio (8 kHz μ-law)

Stream phone-call audio into Hear: native μ-law and the exact resample ratios.

Timestamped recording jobs

When the audio is finished, batch-transcribe with word/segment offsets and diarization.

Browser voice agent

The same PCM16 capture pipeline, driving a full-duplex Omni agent.

Errors & limits

Close codes, rate limits, and concurrency.