> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pyai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Stream speech-to-text in real time

> Build live captions on Hear's streaming WebSocket: send PCM16 audio, render greyed interim partials, force-finalize with commit, and lock in finals on the fly.

When you need words on screen *while someone is still talking*, live captions,
a voice search box, agent-assist, meeting notes as they happen, you want
**streaming** speech-to-text, not a batch job. This guide covers the architecture
and the UX pattern for **Hear streaming**: a WebSocket that emits **interim
("partial")** results within a few hundred milliseconds and **final** results
when a phrase settles.

<Note>
  **Run the complete example.** A browser live-captions demo, scaffolded in one
  command, no clone:

  ```bash theme={null}
  npm create pyai-app@latest browser-hear-live-captions
  ```

  Or browse it: [`browser-hear-live-captions`](https://github.com/atomsai/pyai-examples/tree/main/browser-hear-live-captions).
</Note>

<Info>
  **Endpoint:** `wss://api.pyai.com/v1/audio/transcriptions/stream?protocol=pyai-hear-v1`. The full wire
  protocol, query params, the JSON frame schema, the `{"type":"commit"}` flush, and
  close codes, is published in the **API reference**
  (`GET /v1/audio/transcriptions/stream`, generated from
  `https://api.pyai.com/openapi.json`) and summarized in
  [Wire protocol](#wire-protocol) below. *Last verified 2026-06-16 against
  `api.pyai.com`.*
</Info>

## How it fits together

```mermaid theme={null}
flowchart LR
  mic[Mic / call audio] --> cap[Capture → PCM16 frames]
  cap -->|binary WS: PCM16 16 kHz| hear[Hear streaming WS]
  hear -->|text WS: partial results| ui1[Render greyed partial]
  hear -->|text WS: final results| ui2[Commit final line]
  ui1 -.->|replaced by| ui2
```

One socket carries two things: you send **binary PCM16 audio frames** up, and
receive **text JSON result frames** down. Results come in two flavors, fast-but-revisable **partials** and stable **finals**, and your UI's whole job
is to show the former without committing to them, then lock in the latter.

## Streaming vs. batch, pick the right tool

| | **Hear streaming (this guide)** | **Batch jobs** (`/v1/transcription/jobs`) |
| - | - | - |
| Latency | Sub-second first partial; live | Seconds-to-minutes; asynchronous |
| Output | Interim partials + finals as you speak | One complete transcript when done |
| Best for | Live captions, voice UI, agent-assist | Recordings, archives, post-call analytics |
| Diarization | Speaker labeling is a batch strength | `channel` / `diarize` (exact + model-based) |
| Pricing | [View current pricing](https://pyai.com/pricing) | [View current pricing](https://pyai.com/pricing) |

<Tip>
  If the audio has already finished (a recording, a voicemail, a call you've
  hung up on), use [timestamped batch jobs](/guides/async-transcription-jobs):
  they provide diarization and avoid maintaining a live socket. Compare current
  metering on the [pricing page](https://pyai.com/pricing).
</Tip>

## Audio format: PCM16 at 8 or 16 kHz

Hear streaming consumes **PCM16 little-endian** mono at **8 or 16 kHz**.
Set `sample_rate` to the actual rate of the bytes you send; the default is
`16000`. Hear converts 8 kHz input before recognition and endpointing.
Other sample rates and compressed encodings are rejected before the WebSocket
opens with `400 unsupported_audio_format`.

* **Browser mic** typically runs the `AudioContext` at 48 kHz. Either request a
  16 kHz context, or **decimate 3:1** on capture (48000 / 16000 = 3) before
  sending.
* **Telephony** is usually 8 kHz. Decode μ-law to PCM16 first, then send it with
  `sample_rate=8000`, or resample it to 16 kHz and declare `sample_rate=16000`.
* **Frame size:** \~20 ms per message (160 samples at 8 kHz; 320 at 16 kHz) keeps latency low
  without paying per-message overhead.

```js capture-processor.js, Float32 → PCM16 @ 16 kHz theme={null}
const FRAME = 320; // 20 ms @ 16 kHz

class CaptureProcessor extends AudioWorkletProcessor {
  constructor() {
    super();
    this._buf = new Int16Array(FRAME);
    this._n = 0;
  }
  process(inputs) {
    const ch = inputs[0]?.[0];
    if (!ch) return true;
    for (let i = 0; i < ch.length; i++) {
      const s = Math.max(-1, Math.min(1, ch[i]));
      this._buf[this._n++] = s < 0 ? s * 0x8000 : s * 0x7fff;
      if (this._n === FRAME) {
        const out = this._buf.slice();
        this.port.postMessage(out.buffer, [out.buffer]);
        this._n = 0;
      }
    }
    return true;
  }
}
registerProcessor("capture-processor", CaptureProcessor);
```

## The two-pass display pattern

This is the heart of a good live-transcription UI. Treat the transcript as a
list of **committed final lines** plus **one pending partial** at the end:

1. A **partial** arrives → render it greyed/italic as the "current" line.
   Partials are revisable, replace, never append.
2. A newer partial arrives → **overwrite** the same pending line.
3. A **final** arrives → **commit** it as solid text and clear the pending line.
4. Repeat for the next phrase.

The effect users love: text appears almost instantly, wobbles a little as the
recognizer reconsiders, then snaps to clean final text, exactly like live
captions on a video call.

```js Two-pass transcript renderer theme={null}
const finalsByUtterance = new Map(); // committed lines keyed by utterance_id
let pending = "";      // current partial (greyed)

function onPartial(text) {
  pending = text;      // replace, don't append
  paint();
}

function onFinal(utteranceId, text) {
  finalsByUtterance.set(utteranceId, text); // replace the earlier speech_final
  pending = "";        // clear the greyed line
  paint();
}

function paint() {
  els.committed.textContent = [...finalsByUtterance.values()].join(" ");
  els.pending.textContent = pending;      // styled grey/italic in CSS
}
```

## Wire protocol

**Connect.** Open a WebSocket to the streaming transcription endpoint and pick
your audio format with query params:

```
wss://api.pyai.com/v1/audio/transcriptions/stream
  ?protocol=pyai-hear-v1&model=pyai-hear&language=auto&sample_rate=16000&encoding=pcm16&interim_results=true
```

`protocol=pyai-hear-v1` selects the published bare frame names used throughout
this guide. It is the only Hear streaming protocol; omitting the parameter uses
the same `pyai-hear-v1` default.

<Warning>
  Omit `language` or set `language=auto` to detect the spoken language automatically.
  To pin recognition deliberately, use `en`, `es`, `fr`, `de`, `hi`, `it`, `pt`, or `nl`.
  Unsupported explicit values return `400 unsupported_language`.
  This endpoint does not currently return language-identification or
  decoder-confidence fields.
</Warning>

Authenticate the upgrade with your key as a subprotocol (browser-safe):
`Sec-WebSocket-Protocol: pyai.v1, pyai-key.<API_KEY>`, server-side clients may use
`?api_key=` instead. The key needs the `hear:stream` scope.

<Note title="Numbers, punctuation, finals only">
  With an explicit English hint, **final** transcripts apply number formatting by default (phones,
  currency, dates, ordinals). Send `numerals=false` to keep spoken form, or
  `numerals=true` to force digits. Interim partials are never rewritten. Automatic mode may retain spoken number forms.

  Add `&smart_format=true` (or `smartFormat: true` in the SDK) for punctuation
  and sentence capitalization on finals only. Optional extras, also finals-only
  and off by default: `dictation` (spoken `period` / `comma` / `new paragraph` /
  `question mark`), `drop_fillers` (`um` / `uh` / `er`; do not enable on legal
  or compliance audio by default), and `vocabulary` for known terms. Send terms
  for one session or explicitly enable the stored `hear_stream` profile. See
  [Format Hear transcripts](/guides/hear-transcript-formatting).
  These English cleanup flags are conservative/no-op where unsupported on
  non-English transcripts. Formatting flags do not select the recognition language; an explicit language
  hint constrains it, while omitted language or `auto` uses automatic detection.
</Note>

**Client → server.** Stream **binary** audio frames continuously (PCM16
little-endian mono at `sample_rate=8000` or `sample_rate=16000`), \~20 ms each. To force-finalize the
current utterance, e.g. when your own VAD detects end-of-turn, send a JSON text
frame `{"type":"commit"}`. Closing the socket also flushes a final.

<Warning>
  The only explicit finalization control is the JSON frame
  **`{"type":"commit"}`**. Do not send a plain-text sentinel or another control
  name.
</Warning>

<Tip>
  **Tune automatic endpointing.** Add `endpointing_ms` to the connect URL, for
  example `&endpointing_ms=800`. This is the minimum trailing-pause length before
  an utterance may end, not the exact pause that forces a final. Turn detection
  may wait longer. An utterance always completes within
  `max(endpointing_ms, 1500 ms)` of streamed silence, with exactly one endpoint
  for each continuous silence run.

  Keep sending audio frames, including silence, while the caller pauses. The timer
  advances from incoming frames; if audio stops, the pause timer stops too. Change
  the floor without reconnecting by sending
  `{"type":"config","endpointing_ms":800}`.

  Both forms return `config_ack` with the applied floor, ceiling, score interval,
  and `warnings`. Assert that `warnings` is empty. `clamped_to_range`,
  `not_a_number`, or `unknown_config_field` means the setting did not apply exactly
  as sent, but the session stays open. You can always force immediate end-of-turn
  with `{"type":"commit"}`.
</Tip>

**Server → client.** JSON text frames, emitted with bare type names:

| `type` | When | Payload |
| - | - | - |
| `config_ack` | after connect-time or mid-session endpointing config | `{ endpointing_ms, effective_floor_ms, effective_ceiling_ms, score_interval_ms, warnings }`, the applied setting; assert `warnings` is empty |
| `partial` | every eager tick | `{ text, utterance_id, t_ms }` (may also carry `stable_text` / `active_text`), the live hypothesis; replace it per `utterance_id` |
| `partial_stable` | when a prefix locks in | `{ text, utterance_id, t_ms }`, the portion the recognizer no longer expects to revise |
| `speech_final` | on endpoint / after `commit` | `{ text, utterance_id, t_ms, audio_ms, endpoint_reason }`, end of an utterance, stable |
| `final` | follows `speech_final` | `{ text, utterance_id, t_ms, audio_ms, endpoint_reason }`, corrected full-context transcript |
| `usage` | just before a graceful close | `{ product, meter, audio_seconds, minutes }`, the session's billed active-audio for in-band reconciliation (realtime sockets have no `x-pyai-units` header). Best-effort; omitted if there was no billable audio or the close was abnormal. |
| `error` | on fault | `{ code, message }` |

`utterance_id` groups the partials and finals of one phrase; `t_ms` is the audio
position of the hypothesis; `audio_ms` is the utterance's active-speech length.
`endpoint_reason` is `peak_te_early` when the high-confidence path fires or
`silence_backstop` when the bounded fallback closes the turn. Log it when tuning
conversational pacing.

<Warning>
  **Cue is unavailable.** The Hear stream does not accept Cue grounding
  configuration, return grounding results, or emit Cue usage. Do not build against
  Cue fields.
</Warning>

**Close codes.** `1000` normal · `1008` auth/policy (bad key or missing
`hear:stream` scope) · `1011` engine error · `4429` over the concurrency cap.

An `error` frame with `code: recognition_interrupted`, followed by close `1011`,
means accepted audio could not be fully recovered. Treat the session as
incomplete and start a new stream. An earlier `final` completes its own
utterance; it does not prove that later audio was recognized. Do not treat an
error close as success because an earlier final exists.

Connection retries are limited to before caller PCM is accepted. Hear does not
silently replay missing audio or produce further transcript results or automatic
Recap from an interrupted session.

## Wire it up

Open the socket, stream capture frames as binary, and route text frames through
**one** handler. Authenticate the upgrade with the key as a subprotocol, the
browser-safe pattern that's stable across PyAI's realtime surfaces (server-side
clients may use `?api_key=` instead).

```js Connect + route frames theme={null}
const HEAR_STREAM_URL =
  "wss://api.pyai.com/v1/audio/transcriptions/stream" +
  "?protocol=pyai-hear-v1&model=pyai-hear&language=auto&sample_rate=16000&encoding=pcm16&interim_results=true&endpointing_ms=800";

function connectHearStream(apiKey, capture) {
  const ws = new WebSocket(HEAR_STREAM_URL, ["pyai.v1", `pyai-key.${apiKey}`]);
  ws.binaryType = "arraybuffer";

  // mic/call PCM16 frames → server (binary)
  capture.port.onmessage = (e) => {
    if (ws.readyState === WebSocket.OPEN) ws.send(e.data);
  };

  // text frames are JSON results, all framing lives in handleFrame()
  ws.onmessage = (e) => {
    if (typeof e.data === "string") handleFrame(JSON.parse(e.data));
  };
  return ws;
}

```

Frames you will see while someone is talking:

```json theme={null}
{ "type": "partial", "text": "your appointment is", "utterance_id": "utt_1", "t_ms": 820 }
```

```json theme={null}
{
  "type": "final",
  "text": "Your appointment is confirmed for Thursday at two.",
  "utterance_id": "utt_1",
  "t_ms": 2410,
  "audio_ms": 2100,
  "endpoint_reason": "silence_backstop"
}
```

Render `partial` as revisable UI. Commit `final`. `speech_final` arrives first
with `endpoint_reason`; treat it as a stable preview of the same utterance, not
a second committed line.

```js Force-finalize and handle frames theme={null}
// Force-finalize the current utterance (e.g. on your own end-of-turn signal).
// The explicit finalization frame is JSON: {"type":"commit"}.
function commit(ws) {
  if (ws.readyState === WebSocket.OPEN) ws.send(JSON.stringify({ type: "commit" }));
}

// Change the minimum trailing-pause floor without reconnecting.
function setEndpointing(ws, endpointingMs) {
  if (ws.readyState === WebSocket.OPEN) {
    ws.send(JSON.stringify({ type: "config", endpointing_ms: endpointingMs }));
  }
}

// The one place that knows the frame schema. The engine emits bare types:
// partial / partial_stable (interim) and speech_final / final (stable).
function handleFrame(msg) {
  switch (msg.type) {
    case "config_ack":
      if (msg.warnings.length > 0) {
        console.error("endpointing config warning:", msg.warnings);
      }
      break;
    case "partial":
    case "partial_stable":
      onPartial(msg.text);
      break;
    case "speech_final":
      console.info("endpoint reason:", msg.endpoint_reason);
      // Stable utterance result, followed by the corrected full-context `final`.
      // Show it immediately, but do not append it as a second committed row.
      onPartial(msg.text);
      break;
    case "final":
      onFinal(msg.utterance_id, msg.text);
      break;
    case "error":
      console.error("hear-stream error:", msg.code, msg.message);
      break;
  }
}
```

<Info>
  The endpoint path, the frame schema, and the `{"type":"commit"}` control message
  are part of the published contract, see the **API reference**
  (`GET /v1/audio/transcriptions/stream`) or the [Wire protocol](#wire-protocol)
  summary above. Keep all framing inside `handleFrame` so adding a field later is a
  one-place change.
</Info>

<Note>
  `speech_final` and `final` describe the same completed utterance.
  `speech_final` is the immediate stable result; `final` follows with the
  full-context correction. If your UI commits both, key rows by `utterance_id` and
  replace the first value. Do not append both as separate transcript lines.
</Note>

## Latency expectations

* **First partial: sub-second.** Once audio is flowing you should see an initial
  partial within a few hundred milliseconds, that immediacy is the entire point
  of streaming.
* **Finals lag partials slightly.** A phrase finalizes once the recognizer is
  confident (typically at a pause or end of utterance). This is normal, show
  partials so the UI never feels stalled while waiting for a final.
* **Keep frames small and steady.** 20 ms frames sent as they're produced
  minimize end-to-end latency; don't batch several seconds of audio into one
  message.
* **Don't add your own buffering on top.** Send frames straight from the capture
  worklet; extra queues only add delay.

## Troubleshooting

| Symptom | Likely cause | Fix |
| - | - | - |
| Transcription is gibberish | Sample-rate mismatch | Capture, resample, and send all at one rate (16 kHz); decimate 3:1 from a 48 kHz context |
| No partials appear | Audio not flowing, or frames too large | Confirm binary PCM16 frames are being sent (\~20 ms each) from the capture worklet |
| Text "jumps around" before settling | Treating partials as final | Render partials as a single replaceable greyed line; only append on a final |
| Connection closes immediately | Bad credential / wrong subprotocol | Check the close code in [Errors & limits](/errors-and-limits); auth via `pyai-key.<key>` subprotocol |
| `429 concurrency_limit_exceeded` | Too many live sessions | Close idle sockets or raise the limit on your plan |
| High latency / choppy partials | Frames batched or buffered before send | Send each \~20 ms frame immediately; remove extra jitter buffers |
| Transcript never finalizes | Silence frames stopped, or the wrong control frame was sent | Keep sending audio frames through pauses for automatic endpointing. To force a final, send `{"type":"commit"}`, or close the socket |

## Next steps

<CardGroup cols={2}>
  <Card title="Telephony audio (8 kHz μ-law)" href="/reference/telephony-audio">Stream phone-call audio into Hear: native μ-law and the exact resample ratios.</Card>
  <Card title="Timestamped recording jobs" href="/guides/async-transcription-jobs">When the audio is finished, batch-transcribe with word/segment offsets and diarization.</Card>
  <Card title="Browser voice agent" href="/guides/browser-voice-agent">The same PCM16 capture pipeline, driving a full-duplex Omni agent.</Card>
  <Card title="Errors & limits" href="/errors-and-limits">Close codes, rate limits, and concurrency.</Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.