Using OpenClaw? Install the live PyAI skills for task-based speech and calling from ClawHub. See the OpenClaw setup guide for install commands, permissions and example prompts.
The
pipecat-pyai package is maintained by PyAI. It does not replace your
transport, LLM, context aggregator, or application processors.Listed in Pipecat’s provider directory
PyAI’s community integration is listed in Pipecat’s STT reference and TTS reference. Thepipecat-pyai package and examples are maintained by PyAI.
Watch the recorded interruption demo
This 34-second recording uses prerecorded synthetic input, live Hear and Speak, scripted replies, and VAD-driven interruption. The first reply stops when the second speaker begins; the next reply says “The blue lantern is ready for tomorrow.” The saved recording passed transcription readback and a human playback check. It is evidence for that run, not a human-microphone or latency benchmark.Run the complete WebRTC example
The existing voice-agent example includes the WebRTC transport, VAD, an LLM, Hear, Speak, and disconnect cleanup. Use Python 3.12 and the pinned dependencies in its README. SetPYAI_API_KEY and
your LLM credential on the server, then start examples/bot.py from the package
directory. The runner prints the local URL for connecting your microphone.
Before starting, verify the intended project and limits with GET /v1/me.
The example’s 60-second default limits session duration, not spending; PyAI,
your LLM, and hosting have separate billing. Its Pipecat 1.14.0 setup passed
non-audio startup and cleanup checks. The older two-turn audio qualification
below does not establish live speech compatibility for every newer framework pin.
Download the existing starter
The Pipecat starter includes pinned dependencies, source, environment setup and run commands. It is a text-to-WAV pipeline, not a microphone or transport agent. For a complete conversation, continue below with your transport, LLM and tools. CheckGET /v1/me for the intended project, billing and usage limits before running;
the starter generates real audio. Machine-readable download instructions.
Try Hear and Speak before integrating
Open the live voice demo: record a question about PyAI, see Hear’s transcript, and listen to Speak’s reply. You can also type. The product answers are scripted; transcription and speech use real PyAI APIs. Three free questions, no signup required; daily demo limits apply. The hosted demo is a standalone Hear + Speak reference app. To use these APIs in your own application, get a free API key and follow the setup below. The demo also includes a signup link.What stays under your control
Pipecat still owns frame order and routing. You can place custom processors before, between, or after the PyAI services and keep the deployment model you already use.Prerequisites
- Python 3.11 or newer
- A Pipecat application with a transport and LLM service
- A PyAI key with
hear:streamandspeak:synthesizescopes - A voice id from
GET https://api.pyai.com/v1/voices
Version compatibility and test setup
The two-turn media → Hear → LLM → Speak path was exercised withpipecat-ai==1.8.1, pyai-sdk==0.5.0, and Python 3.12. Keep your transport,
VAD, and LLM dependencies compatible with that framework version. Your LLM
and media hosting have their own credentials and costs.
Before running, use GET /v1/me to verify the expected organization, project,
env: test, and effective limits. Set explicit audio duration, request and
runtime bounds; environment alone does not guarantee a daily spending cap.
Shared speech capacity can return HTTP 429 even below the key’s concurrency
ceiling. Respect Retry-After and keep retries bounded.
Install
Install the verified public release from PyPI:Add PyAI to the pipeline
Put Hear after the transport input and Speak before the transport output:transport, user_aggregator, assistant_aggregator, and llm
refer to the objects your Pipecat application already creates. Your VAD and
turn configuration remain in effect. For a transport without built-in VAD, add
your configured VADProcessor after transport.input() and before STT; the
STT adapter uses the VAD stop event to request a committed final transcript.
Complete task shape
Configuration
Hear speech-to-text
InterimTranscriptionFrame plus finalized TranscriptionFrame objects. Hear
streaming detects the language automatically when you omit language.
Hear supports 29 recognition languages. To pin recognition, use a supported
code from the language table. Language support
is separate from measured accuracy on your audio; test your accents, noise and
terminology before rollout.
Speak text-to-speech
TTSAudioRawFrame objects at the configured rate. Use a streaming-capable
stock, cloned, or designed voice id available to the same PyAI organization.
Share a TTS HTTP session
For applications that already manage anaiohttp.ClientSession, pass it to
the TTS service:
Run and verify
1
Start your Pipecat bot
Run the same transport and deployment command you use today.
2
Send a two-turn conversation
Speak one short request, wait for agent audio, then interrupt or ask a
follow-up. This verifies transcript finalization, context flow, TTS, and
interruption behavior.
3
Inspect pipeline metrics
Confirm the Pipecat task receives transcript frames and TTS usage metrics.
Log the transport region, worker region, LLM, VAD settings, voice, and PyAI
region with any latency result.
4
Exercise failure handling
Test an invalid key, missing scope, and exhausted concurrency so your
pipeline presents a safe fallback instead of silently stopping.
Production notes
- Keep your Pipecat worker, transport media server, and PyAI speech endpoints in nearby regions.
- A running Hear service counts against the PyAI key’s realtime concurrency.
- Hear and Speak meter against your PyAI account; see the pricing page for current terms. Pipecat Cloud, transport, your LLM, and worker hosting remain separate.
- Keep a bounded queue and timeout around custom processors so one integration cannot stall caller audio indefinitely.
- Use Omni instead when you want PyAI to own turn-taking, reasoning, tools, caller continuity when an Agent profile has continuity enabled and a caller key is available, and optional managed telephony as one system.
Troubleshooting
Which path should I choose?
Choose Pipecat + PyAI for custom frame processors, transport flexibility, branching, and per-stage observability. Choose Omni when one managed speech-to-speech contract is more valuable than operating the pipeline.Pipecat documentation
Review Pipecat’s pipeline, transport, and deployment model.
PyAI Omni overview
Compare the managed speech-to-speech path.
Speech To Text (Hear)
Streaming behavior, language support, and transcript formatting.
Text To Speech (Speak)
Voice selection, streaming modes, and PCM output.
Telephony audio
Phone codecs, PCM rates, and resampling boundaries.
Clone
Enroll and use a consented custom voice.
Measuring a first saved artifact
The downloadable Pipecat starter validates the closed WAV before sending an optional metadata-only saved-file receipt. It sends only the original Speak request ID and a fixed starter source with the same key; no audio, text, filename or local path is uploaded for analytics. SetPYAI_REPORT_ARTIFACTS=0 to disable reporting. Reporting failure never retries
speech generation.
The integration growth report matches receipts to successful Speak usage and
separates the first saved artifact, another saved artifact on a later UTC day,
and later-day API usage from the signup cohort. These are client-reported saves
corroborated by server usage, not proof of audible playback. The complete WebRTC
example streams audio and does not save a WAV or emit this receipt. Existing
package installs and older starter downloads are not retroactively instrumented.