> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pyai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# PyAI changelog: API, SDK, and Omni protocol releases

> Release notes for the PyAI API, SDKs, and docs covering Hear, Speak, and Omni protocol updates, telephony audio, and developer-experience improvements.

<Update label="2026-09-04" description="Hear streaming: session start no longer depends on the billing plane; explicit capacity signalling">
  * **Streams start even when our billing plane is unavailable.** Opening a
    Hear stream used to wait on an internal usage reservation before any
    transcription began, and a hiccup in that service refused the stream.
    The reservation is now recorded locally on the streaming node and
    reconciled in the background, so a billing-plane blip no longer affects
    session start or end. Metering is unchanged.
  * **Capacity is signalled up front.** The streaming edge now reads the
    speech engine's live stream count and, when the engine is at capacity,
    refuses a new stream immediately with error code `capacity` and close
    code 1013 (retry) rather than accepting it into a degraded state. This
    is the same retryable signal introduced on 2026-09-03 for mid-call
    shedding.
</Update>

<Update label="2026-09-03" description="Docs: first-call loops, SDKs page, expected responses">
  * **Quickstart now closes every first call.** `GET /v1/me` shows a sandbox
    identity (including scopes the mint does *not* grant). Hear has curl /
    Python / Node tabs and a sample `{ "text": "..." }`. Omni documents the
    `configured` ack. Cast preview is runnable with `stock_dorit_en_us`. Recap
    has a submit-and-read path with a complete `recap.record.v1` sample.
  * **Clone examples match the shipped SDKs.** Use `pyai.clones`, not
    `pyai.voice.clones`. Poll `GET /v1/voice/clones` (there is no GET-by-id).
  * **New [SDKs](/guides/sdks) page** in Get started. Auth and Errors now show
    the first-call `401 invalid_api_key` / `403 insufficient_scope` /
    `402 credit_exhausted` envelopes.
</Update>

<Update label="2026-09-03" description="Hear streaming: finals on long turns, and no more mid-call drops under load">
  * **A `final` now arrives at least every 15 seconds of audio.** Two-party
    call audio that never pauses for `endpointing_ms` used to run one open
    utterance for minutes and, on a dropped socket, never produce a `final`
    at all. The engine now closes such an utterance at 15 s with
    `endpoint_reason: "max_utterance"` and starts a fresh `utterance_id`;
    pipelines that store only finals get every word. Silence endpoints and
    `commit` behave exactly as before.
  * **Mid-call engine drops no longer end your session.** If the speech
    engine behind the stream closes or stalls, the stream re-establishes its
    engine leg and resumes; you may see a gap of up to one chunk, not a
    closed socket. When the platform is genuinely out of capacity you get
    error code `capacity` and close code 1013 (retry), instead of
    `internal: stream terminated unexpectedly`.
  * Concurrency: streams are now decoded together on the GPU rather than one
    at a time, and each engine sheds load with a retryable 1013 instead of
    degrading every active stream. Measured limits are published with the
    rollout record.
</Update>

<Update label="2026-09-03" description="Hindi word-level timestamps">
  * **Hindi joins word-level timestamps on async transcription.** `words[]`
    now carries per-word times for `hi` alongside `en`, `es`, `fr`, `de`,
    `it`, and `pt` — native Devanagari, no romanization. Hindi timing
    precision is a step below the Latin-script languages on heavily
    code-switched telephony (roughly four in five word starts within 120 ms);
    `segments` remain robust either way. `nl` keeps turn-level segments with
    an empty `words` array.
</Update>

<Update label="2026-09-03" description="Word-level timestamps on async transcription">
  * **`words[]` is back on async transcription jobs — now word-granular.** A
    forced-alignment pass restores per-word `start`/`end` times for `en`,
    `es`, `fr`, `de`, `it`, and `pt` (median word-start accuracy measured at
    20-50 ms on real telephony). `segments` are now built from word timings
    on these languages; `hi` and `nl` keep turn-level segments with an empty
    `words` array for now.
  * Word timings survive number formatting: tokens rewritten by digit
    formatting carry their span's true start/end rather than a guess.
  * No request change needed — jobs that always requested timestamps simply
    start receiving them.
</Update>

<Update label="2026-09-03" description="Async Trace live: deterministic PII scan on transcription jobs">
  * **`trace: true` on async transcription jobs now runs a real scan.** The
    deterministic detector set from the Trace inline gate — SSN, card numbers
    (grouped and contiguous), CVV in context, email, US phone — runs over the
    final transcript; findings are redacted as `[REDACTED_<TYPE>]` and the
    completed job carries the redacted transcript plus a `trace` summary
    (verdict, PII count). Requires the Trace entitlement; bills one Trace
    call per job. `rule_pack.entities` narrows the scan to a subset.
  * Patterns run on the formatted transcript: keep the default `numerals`
    (digits) for full effect — verbatim transcripts can leave spoken-out
    numbers unredacted. Not yet combined with `diarize`/`channel`.
  * This replaces yesterday's temporary `400` on async `trace`.
</Update>

<Update label="2026-09-03" description="API reference: Clone, Hear, and Speak">
  * **The generated API reference matches the docs sidebar.** OpenAPI tags are
    now **Speech To Text (Hear)**, **Text To Speech (Speak)**, and **Clone**.
    Clone list/create/delete live under Clone, not Speak. Voice design stays
    on Speak.
</Update>

<Update label="2026-09-03" description="Docs nav: Clone, Hear, and Speak">
  * **Clone is its own docs category.** Voice enrollment no longer lives under
    Speak. Sidebar groups are now **Speech To Text (Hear)**, **Text To Speech
    (Speak)**, **Clone**, and **Cast**.
  * **Voice design is documented as live.** The Clone guide no longer calls
    `/v1/voice/design` a rolling-out surface.
</Update>

<Update label="2026-09-02" description="Async diarization restored; async Trace paused; Cast voice roster">
  * **Channel and diarize segments restored on async jobs.** Between 2026-08-26
    and 2026-09-02, `channel` jobs returned one block per speaker instead of
    conversational turns, and mono `diarize` jobs fell back to a flat
    transcript. Turn segmentation is now derived from the audio itself and no
    longer depends on engine word timings; `segments` carry per-turn speakers
    and timing again. The `words` array remains empty on async jobs until
    word-level alignment returns to the batch engine.
  * **Async `trace` is temporarily rejected.** The current batch engine cannot
    run the Trace scan, and a job that requests PII scanning must never
    complete without one — `trace: true` on async submission now returns `400`
    instead of silently skipping the scan. The synchronous Trace surface is
    unaffected.
  * **Cast voices are the capabilities roster.** `GET /v1/cast/capabilities`
    is the authoritative voice list, as documented; legacy undocumented names
    from before the 2026-08-31 engine migration (e.g. `sarah`) now return
    `422 unsupported_voice`.
</Update>

<Update label="2026-08-28" description="New Hear recognition engine">
  * **Hear is served by a new speech-recognition engine** across streaming and
    async batch. In a blind side-by-side judged on real calls, the new engine's
    transcripts were preferred in **87.3%** of disagreements with the prior
    engine.
  * Async batch jobs now transcribe all eight Hear languages, with the spoken
    language auto-detected per call. The detected language code is not yet
    returned in the result.
</Update>

<Update label="2026-08-27" description="Vocabulary biasing across Hear and Omni">
  * **Per-job vocabulary:** async transcription jobs accept an optional
    `vocabulary` list of up to five distinctive terms, sanitized and applied to
    that job only.
  * **Organization vocabulary:** stored Hear vocabulary settings with explicit
    batch and streaming activation; streaming biasing uses the same bounded
    sanitizer.
  * **Managed Agent vocabulary:** Omni Agent profiles accept an opt-in
    `vocabulary` list, frozen at session start and never inherited from
    organization settings.
  * Vocabulary biases recognition toward your terms; it does not choose the
    transcription language or steer the output script.
</Update>

<Update label="2026-08-26" description="200 ms first partial on streaming">
  * **Hear streaming publishes \~200 ms to first partial** measured on real
    speech in-region. Partials remain revisable; finals are the stable text to
    commit.
</Update>

<Update label="2026-08-25" description="Eight-language sync and streaming STT">
  * **Hear sync and streaming serve eight languages:** `en`, `es`, `fr`, `de`,
    `hi`, `it`, `pt`, and `nl` via the explicit `language` parameter. Omission
    means English; codes outside the set are rejected with
    `400 unsupported_language`, never silently mis-transcribed.
</Update>

<Update label="2026-08-24" description="Certified browser widget v11">
  * **Widget v11 is the certified browser widget** for opening playback and
    Omni protocol compatibility. New embeds should load
    `https://cdn.pyai.com/widget/v11/pyai-widget.js`.
  * Versioned widget URLs are immutable. Sites pinned to v1-v10 should update
    their script URL to v11.
</Update>

<Update label="2026-08-19" description="Website knowledge crawl">
  * **Hosted knowledge bases can crawl a public website.**
    `POST /v1/knowledgebases/{id}/crawls` discovers same-origin pages, ranks
    them, and registers each selected URL as a normal document. Poll until
    `indexed` before you treat the site as ready. The agent wizard crawls the
    first website you add.
</Update>

<Update label="2026-08-19" description="Docs paths and website widget">
  * **Choose your path** is a real docs page. The navbar button no longer
    jumps straight to console signup.
  * **Create agents via API** collects profile CRUD, hosted KB bind, tools,
    and `session_label` in one guide.
  * **Frameworks** (LiveKit, Pipecat, Cursor) sit in their own nav group,
    not under CRM integrations. Omni v2 migration moved to the bottom of Omni.
  * Hosted knowledge bases are the default grounding path. `kb_endpoint`
    remains for teams that already run retrieval.
  * Website widget docs match the console tab name: **Website & phone**.
</Update>

<Update label="2026-08-13" description="Omni v2 chat URL discontinued">
  * **Hindi is live on Omni** with four Standard-tier voices. Choose `hi1`
    through `hi4`; omit `voice_id` for the default. See the
    [pricing page](https://pyai.com/pricing) for current tier treatment.
  * **Short multilingual voice aliases:** `es1`–`es3`, `fr1`, `de1`, and
    `hi1`–`hi4`. Aliases are permanent convenience inputs; the
    `configured.voice_id` ack reports the canonical stock id actually served.
  * **The old Omni v2 chat URL has been discontinued.** It now returns `404`.
    Connect to
    `wss://api.pyai.com/v1/omni?format=pcm16&rate=24000`.
  * **Query migration:** move the value of `agent` or `agent_id` to
    `session_label`. The label is optional. Remove `model`. Replace an
    `access_token` query parameter with the `pyai-key.<key>` WebSocket
    subprotocol or `api_key=<key>`.
  * **Auth:** browser clients should send
    `Sec-WebSocket-Protocol: pyai-key.pyai_live_...`. Server clients can use
    `Authorization: Bearer <key>`, `x-api-key: <key>`, or the `api_key` query
    fallback. The key needs `omni:session`.
  * **Configure:** after the upgrade, send
    `0x03 || {"type":"configure",...}` as a binary frame. Client control JSON
    uses `type`; server JSON uses `event`.
  * **Audio and control framing:** prefix caller and agent PCM16 with `0x01`,
    transcripts with `0x02`, and control frames with `0x03`. Bare PCM and text
    control frames are not accepted.
  * **Behavioral differences:** the discontinued URL returns HTTP `404`.
    Canonical query and auth failures can reject the HTTP upgrade with
    `400`/`401`/`403`; post-upgrade closes include `4401`, `4403`, `4429`, and
    `1011`.
  * Follow the [complete migration guide](/guides/migrate-omni-v2-chat) for
    before-and-after URLs, code, close handling, and troubleshooting.
</Update>

<Update label="2026-07-01" description="AMD, answering machine detection">
  * **PyAI AMD is live**, answering-machine detection that tells your dialer *who
    or what* answered a call (human, voicemail, IVR, live receptionist,
    iPhone/Google screening, dead number, fax) in a fraction of Twilio's dead-air
    dwell, with the word-level **reason** it decided.
  * **One-line-TwiML Twilio drop-in:** point a `<Start><Stream>` at
    `wss://api.pyai.com/v1/amd/stream` and drop `machineDetection`. `<Start>`
    forks the audio, so your call flow continues untouched. The
    `answered_by_twilio` field maps to Twilio's exact `AnsweredBy` enum, so your
    routing logic doesn't change.
  * **One `aggressiveness` dial** (0-1, per account via `POST /v1/amd/config` or
    per call via a TwiML `<Parameter>`): human-safe ↔ machine-aggressive.
  * **SDKs:** `pyai.amd.config` / `pyai.amd.calls` (+ `amd.stream()` in TS);
    MCP tools `configure_amd`, `list_amd_calls`, `get_amd_call`. New guide:
    [Answering machine detection](/guides/amd-answering-machine-detection).
  * **Metering:** AMD records answered-call usage. See the
    [pricing page](https://pyai.com/pricing) for current terms.
</Update>

<Update label="2026-06-22" description="Post-call data capture + Omni tools GA">
  * **Post-call data capture**, declare an `extraction_schema` + `extraction_webhook_url`
    on an agent and PyAI extracts structured JSON from each call's transcript and
    delivers it signed (`omni.call.extracted`). No client or engine code. New guide:
    [Post-call data capture](/guides/post-call-extraction); configure it in the console
    Agents → **Data capture** tab.
  * **`npm create pyai-app@latest`** scaffolds any runnable example in one command, no clone, no setup. Browse them at the [examples repo](https://github.com/atomsai/pyai-examples).
  * **New examples**, each linked from its guide and exercised in nightly CI: **OpenAI drop-in**,
    **voice cloning**, **call analytics**, and offline **`omni-mock`**.
  * **Machine-readable Omni contract:** AsyncAPI 3.0 at `contracts/omni-asyncapi.yaml`, see
    [Omni wire protocol](/realtime/omni-protocol).
  * **Omni function calling (tools) is GA**, client-loop `tool_call` / `tool_result` on the WebSocket.
    New guide: [Omni tools](/guides/omni-tools). REST: [`/v1/tools`](/api-reference),
    [`/v1/agents/{id}/tools`](/api-reference).
  * **`tools[]`** moved from roadmap → live in the protocol reference (\~5 s default timeout).
</Update>

<Update label="2026-06-19" description="One docs home + drift guards">
  * **`docs.pyai.com` is now the single home** for everything: quickstart, guides,
    and the generated [API reference](/api-reference). `api.pyai.com/docs` redirects
    here; `api.pyai.com/openapi.json` remains the machine-readable contract.
  * **Quickstart verifies your key first.** A new `GET /v1/me` step gives you a
    self-diagnosing first `200` in seconds before any audio call.
  * **Docs can't drift to dead ids.** CI now fails if any sample uses a voice or
    model id that isn't in the live catalog, every snippet stays copy-paste-runnable.
  * Example voice standardized to `stock_dorit_en_us` across the quickstart, guides,
    and SDK READMEs.
</Update>

<Update label="2026-06-16" description="Docs accuracy & unified API reference">
  * **REST reference unified in-site.** The full OpenAPI reference now lives
    inside these docs under the **API reference** tab (generated from
    [`https://api.pyai.com/openapi.json`](https://api.pyai.com/openapi.json)), no more bouncing to a separate page.
  * **Hear streaming contract:** `pyai-hear-v1` emits the canonical bare frames
    (`partial`, `partial_stable`, `speech_final`, `final`, `usage`, `error`) and
    force-finalizes with the JSON frame `{"type":"commit"}`. Cue is unavailable
    on the serving route. See
    [Stream speech-to-text](/guides/streaming-stt) for current behavior.
  * **Telephony output on Speak:** request `g711_ulaw` / `g711_alaw` for native
    8 kHz μ-law/A-law, no client-side resampling. New reference:
    [Telephony audio](/reference/telephony-audio) with the exact integer resample
    ratios for the cases where you still convert yourself.
</Update>

<Update label="2026-06" description="Developer experience">
  * New guides: [browser voice agent](/guides/browser-voice-agent), [Twilio phone agent](/guides/twilio-voice-agent), and [FreeSWITCH](/guides/freeswitch-voice-agent).
  * Official SDKs: `@pyai/sdk` (TypeScript/JavaScript) and `pyai` (Python), plus the `pyai` CLI smoke tester.
  * `Idempotency-Key` support and cursor pagination on transcription jobs.
  * Machine-readable error codes documented end to end (see [Errors & limits](/errors-and-limits)).
  * Sandbox tier: `pyai_test_` keys work instantly with no billing gate.
</Update>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.