Skip to main content
PyAI Hear and Speak listed in Pipecat
PyAI’s community integration is published in Pipecat’s STT and TTS references. Keep your existing transport, LLM, and tools while adding Hear and Speak. Start with the Pipecat guide for pipecat-pyai==0.1.5, the existing interruption recording, and the complete WebRTC example.
Product docs: voice agent settings and clearer navigation
  • A new voice agent settings guide maps console tabs to API fields, supported controls, and customization guides.
  • Agent docs are grouped by customization, connections, and protocol. Existing page URLs are preserved.
  • The product directory links directly to settings for voice cloning, Cast, dubbing, Live Translation, Recap, Trace, and AMD.
  • Repeated setup and webhook instructions now link to their canonical guides. Console labels and Omni voice-control guidance match the current contract.
STT and TTS documentation: clearer first requests and current async contracts
  • The quickstart now generates a speech file with TTS and transcribes it with STT, with expected responses and first-request troubleshooting.
  • Speech guides and navigation use STT and TTS names alongside Hear and Speak.
  • Async transcription docs now cover transcription_language, original or Romanized output, JSON and multipart idempotency, completion callbacks, and fallback input retention.
  • Language and audio-format guidance now matches the current API contract. Existing documentation URLs are preserved.
Hear streaming: session start no longer depends on the billing plane; explicit capacity signalling
  • Streams start even when our billing plane is unavailable. Opening a Hear stream used to wait on an internal usage reservation before any transcription began, and a hiccup in that service refused the stream. The reservation is now recorded locally on the streaming node and reconciled in the background, so a billing-plane blip no longer affects session start or end. Metering is unchanged.
  • Capacity is signalled up front. The streaming edge now reads the speech engine’s live stream count and, when the engine is at capacity, refuses a new stream immediately with error code capacity and close code 1013 (retry) rather than accepting it into a degraded state. This is the same retryable signal introduced on 2026-09-03 for mid-call shedding.
Docs: first-call loops, SDKs page, expected responses
  • Quickstart now closes every first call. GET /v1/me shows a sandbox identity (including scopes the mint does not grant). Hear has curl / Python / Node tabs and a sample { "text": "..." }. Omni documents the configured ack. Cast preview is runnable with stock_dorit_en_us. Recap has a submit-and-read path with a complete recap.record.v1 sample.
  • Clone examples match the shipped SDKs. Use pyai.clones, not pyai.voice.clones. Poll GET /v1/voice/clones (there is no GET-by-id).
  • New SDKs page in Get started. Auth and Errors now show the first-call 401 invalid_api_key / 403 insufficient_scope / 402 credit_exhausted envelopes.
Hear streaming: finals on long turns, and no more mid-call drops under load
  • A final now arrives at least every 15 seconds of audio. Two-party call audio that never pauses for endpointing_ms used to run one open utterance for minutes and, on a dropped socket, never produce a final at all. The engine now closes such an utterance at 15 s with endpoint_reason: "max_utterance" and starts a fresh utterance_id; pipelines that store only finals get every word. Silence endpoints and commit behave exactly as before.
  • Mid-call engine drops no longer end your session. If the speech engine behind the stream closes or stalls, the stream re-establishes its engine leg and resumes; you may see a gap of up to one chunk, not a closed socket. When the platform is genuinely out of capacity you get error code capacity and close code 1013 (retry), instead of internal: stream terminated unexpectedly.
  • Concurrency: streams are now decoded together on the GPU rather than one at a time, and each engine sheds load with a retryable 1013 instead of degrading every active stream. Measured limits are published with the rollout record.
Hindi word-level timestamps
  • Hindi joins word-level timestamps on async transcription. words[] now carries per-word times for hi alongside en, es, fr, de, it, and pt — native Devanagari, no romanization. Hindi timing precision is a step below the Latin-script languages on heavily code-switched telephony (roughly four in five word starts within 120 ms); segments remain robust either way. nl keeps turn-level segments with an empty words array.
Word-level timestamps on async transcription
  • words[] is back on async transcription jobs — now word-granular. A forced-alignment pass restores per-word start/end times for en, es, fr, de, it, and pt (median word-start accuracy measured at 20-50 ms on real telephony). segments are now built from word timings on these languages; hi and nl keep turn-level segments with an empty words array for now.
  • Word timings survive number formatting: tokens rewritten by digit formatting carry their span’s true start/end rather than a guess.
  • No request change needed — jobs that always requested timestamps simply start receiving them.
Async Trace live: deterministic PII scan on transcription jobs
  • trace: true on async transcription jobs now runs a real scan. The deterministic detector set from the Trace inline gate — SSN, card numbers (grouped and contiguous), CVV in context, email, US phone — runs over the final transcript; findings are redacted as [REDACTED_<TYPE>] and the completed job carries the redacted transcript plus a trace summary (verdict, PII count). Requires the Trace entitlement; bills one Trace call per job. rule_pack.entities narrows the scan to a subset.
  • Patterns run on the formatted transcript: keep the default numerals (digits) for full effect — verbatim transcripts can leave spoken-out numbers unredacted. Not yet combined with diarize/channel.
  • This replaces yesterday’s temporary 400 on async trace.
API reference: Clone, Hear, and Speak
  • The generated API reference matches the docs sidebar. OpenAPI tags are now Speech To Text (Hear), Text To Speech (Speak), and Clone. Clone list/create/delete live under Clone, not Speak. Voice design stays on Speak.
Docs nav: Clone, Hear, and Speak
  • Clone is its own docs category. Voice enrollment no longer lives under Speak. Sidebar groups are now Speech To Text (Hear), Text To Speech (Speak), Clone, and Cast.
  • Voice design is documented as live. The Clone guide no longer calls /v1/voice/design a rolling-out surface.
Async diarization restored; async Trace paused; Cast voice roster
  • Channel and diarize segments restored on async jobs. Between 2026-08-26 and 2026-09-02, channel jobs returned one block per speaker instead of conversational turns, and mono diarize jobs fell back to a flat transcript. Turn segmentation is now derived from the audio itself and no longer depends on engine word timings; segments carry per-turn speakers and timing again. The words array remains empty on async jobs until word-level alignment returns to the batch engine.
  • Async trace is temporarily rejected. The current batch engine cannot run the Trace scan, and a job that requests PII scanning must never complete without one — trace: true on async submission now returns 400 instead of silently skipping the scan. The synchronous Trace surface is unaffected.
  • Cast voices are the capabilities roster. GET /v1/cast/capabilities is the authoritative voice list, as documented; legacy undocumented names from before the 2026-08-31 engine migration (e.g. sarah) now return 422 unsupported_voice.
New Hear recognition engine
  • Hear is served by a new speech-recognition engine across streaming and async batch. In a blind side-by-side judged on real calls, the new engine’s transcripts were preferred in 87.3% of disagreements with the prior engine.
  • Async batch jobs now transcribe all eight Hear languages, with the spoken language auto-detected per call. The detected language code is not yet returned in the result.
Vocabulary biasing across Hear and Omni
  • Per-job vocabulary: async transcription jobs accept an optional vocabulary list of up to five distinctive terms, sanitized and applied to that job only.
  • Organization vocabulary: stored Hear vocabulary settings with explicit batch and streaming activation; streaming biasing uses the same bounded sanitizer.
  • Managed Agent vocabulary: Omni Agent profiles accept an opt-in vocabulary list, frozen at session start and never inherited from organization settings.
  • Vocabulary biases recognition toward your terms; it does not choose the transcription language or steer the output script.
200 ms first partial on streaming
  • Hear streaming publishes ~200 ms to first partial measured on real speech in-region. Partials remain revisable; finals are the stable text to commit.
Eight-language sync and streaming STT
  • Hear sync and streaming serve eight languages: en, es, fr, de, hi, it, pt, and nl via the explicit language parameter. Omission means English; codes outside the set are rejected with 400 unsupported_language, never silently mis-transcribed.
Certified browser widget v11
  • Widget v11 is the certified browser widget for opening playback and Omni protocol compatibility. New embeds should load https://cdn.pyai.com/widget/v11/pyai-widget.js.
  • Versioned widget URLs are immutable. Sites pinned to v1-v10 should update their script URL to v11.
Website knowledge crawl
  • Hosted knowledge bases can crawl a public website. POST /v1/knowledgebases/{id}/crawls discovers same-origin pages, ranks them, and registers each selected URL as a normal document. Poll until indexed before you treat the site as ready. The agent wizard crawls the first website you add.
Docs paths and website widget
  • Choose your path is a real docs page. The navbar button no longer jumps straight to console signup.
  • Create agents via API collects profile CRUD, hosted KB bind, tools, and session_label in one guide.
  • Frameworks (LiveKit, Pipecat, Cursor) sit in their own nav group, not under CRM integrations. Omni v2 migration moved to the bottom of Omni.
  • Hosted knowledge bases are the default grounding path. kb_endpoint remains for teams that already run retrieval.
  • Website widget docs match the console tab name: Website & phone.
Omni v2 chat URL discontinued
  • Hindi is live on Omni with four Standard-tier voices. Choose hi1 through hi4; omit voice_id for the default. See the pricing page for current tier treatment.
  • Short multilingual voice aliases: es1–es3, fr1, de1, and hi1–hi4. Aliases are permanent convenience inputs; the configured.voice_id ack reports the canonical stock id actually served.
  • The old Omni v2 chat URL has been discontinued. It now returns 404. Connect to wss://api.pyai.com/v1/omni?format=pcm16&rate=24000.
  • Query migration: move the value of agent or agent_id to session_label. The label is optional. Remove model. Replace an access_token query parameter with the pyai-key.<key> WebSocket subprotocol or api_key=<key>.
  • Auth: browser clients should send Sec-WebSocket-Protocol: pyai-key.pyai_live_.... Server clients can use Authorization: Bearer <key>, x-api-key: <key>, or the api_key query fallback. The key needs omni:session.
  • Configure: after the upgrade, send 0x03 || {"type":"configure",...} as a binary frame. Client control JSON uses type; server JSON uses event.
  • Audio and control framing: prefix caller and agent PCM16 with 0x01, transcripts with 0x02, and control frames with 0x03. Bare PCM and text control frames are not accepted.
  • Behavioral differences: the discontinued URL returns HTTP 404. Canonical query and auth failures can reject the HTTP upgrade with 400/401/403; post-upgrade closes include 4401, 4403, 4429, and 1011.
  • Follow the complete migration guide for before-and-after URLs, code, close handling, and troubleshooting.
AMD, answering machine detection
  • PyAI AMD is live, answering-machine detection that tells your dialer who or what answered a call (human, voicemail, IVR, live receptionist, iPhone/Google screening, dead number, fax) in a fraction of Twilio’s dead-air dwell, with the word-level reason it decided.
  • One-line-TwiML Twilio drop-in: point a <Start><Stream> at wss://api.pyai.com/v1/amd/stream and drop machineDetection. <Start> forks the audio, so your call flow continues untouched. The answered_by_twilio field maps to Twilio’s exact AnsweredBy enum, so your routing logic doesn’t change.
  • One aggressiveness dial (0-1, per account via POST /v1/amd/config or per call via a TwiML <Parameter>): human-safe ↔ machine-aggressive.
  • SDKs: pyai.amd.config / pyai.amd.calls (+ amd.stream() in TS); MCP tools configure_amd, list_amd_calls, get_amd_call. New guide: Answering machine detection.
  • Metering: AMD records answered-call usage. See the pricing page for current terms.
Post-call data capture + Omni tools GA
  • Post-call data capture, declare an extraction_schema + extraction_webhook_url on an agent and PyAI extracts structured JSON from each call’s transcript and delivers it signed (omni.call.extracted). No client or engine code. New guide: Post-call data capture; configure it in the console Agents → Data capture tab.
  • npm create pyai-app@latest scaffolds any runnable example in one command, no clone, no setup. Browse them at the examples repo.
  • New examples, each linked from its guide and exercised in nightly CI: OpenAI drop-in, voice cloning, call analytics, and offline omni-mock.
  • Machine-readable Omni contract: AsyncAPI 3.0 at contracts/omni-asyncapi.yaml, see Omni wire protocol.
  • Omni function calling (tools) is GA, client-loop tool_call / tool_result on the WebSocket. New guide: Omni tools. REST: /v1/tools, /v1/agents/{id}/tools.
  • tools[] moved from roadmap → live in the protocol reference (~5 s default timeout).
One docs home + drift guards
  • docs.pyai.com is now the single home for everything: quickstart, guides, and the generated API reference. api.pyai.com/docs redirects here; api.pyai.com/openapi.json remains the machine-readable contract.
  • Quickstart verifies your key first. A new GET /v1/me step gives you a self-diagnosing first 200 in seconds before any audio call.
  • Docs can’t drift to dead ids. CI now fails if any sample uses a voice or model id that isn’t in the live catalog, every snippet stays copy-paste-runnable.
  • Example voice standardized to stock_dorit_en_us across the quickstart, guides, and SDK READMEs.
Docs accuracy & unified API reference
  • REST reference unified in-site. The full OpenAPI reference now lives inside these docs under the API reference tab (generated from https://api.pyai.com/openapi.json), no more bouncing to a separate page.
  • Hear streaming contract: pyai-hear-v1 emits the canonical bare frames (partial, partial_stable, speech_final, final, usage, error) and force-finalizes with the JSON frame {"type":"commit"}. Cue is unavailable on the serving route. See Stream speech-to-text for current behavior.
  • Telephony output on Speak: request g711_ulaw / g711_alaw for native 8 kHz μ-law/A-law, no client-side resampling. New reference: Telephony audio with the exact integer resample ratios for the cases where you still convert yourself.
Developer experience
  • New guides: browser voice agent, Twilio phone agent, and FreeSWITCH.
  • Official SDKs: @pyai/sdk (TypeScript/JavaScript) and pyai (Python), plus the pyai CLI smoke tester.
  • Idempotency-Key support and cursor pagination on transcription jobs.
  • Machine-readable error codes documented end to end (see Errors & limits).
  • Sandbox tier: pyai_test_ keys work instantly with no billing gate.