Skip to main content
Using OpenClaw? Install the live PyAI skills for task-based speech and calling from ClawHub. See the OpenClaw setup guide for install commands, permissions and example prompts.
Use Pipecat when you want an explicit Python pipeline where your application controls transports, processors, context, the LLM, tools, and turn policy. Add PyAI Hear and PyAI Speak as two services in that pipeline. If you want PyAI to operate the complete conversation loop, use Omni.
The pipecat-pyai package is maintained by PyAI. It does not replace your transport, LLM, context aggregator, or application processors.

Listed in Pipecat’s provider directory

PyAI’s community integration is listed in Pipecat’s STT reference and TTS reference. The pipecat-pyai package and examples are maintained by PyAI.

Watch the recorded interruption demo

This 34-second recording uses prerecorded synthetic input, live Hear and Speak, scripted replies, and VAD-driven interruption. The first reply stops when the second speaker begins; the next reply says “The blue lantern is ready for tomorrow.” The saved recording passed transcription readback and a human playback check. It is evidence for that run, not a human-microphone or latency benchmark.

Run the complete WebRTC example

The existing voice-agent example includes the WebRTC transport, VAD, an LLM, Hear, Speak, and disconnect cleanup. Use Python 3.12 and the pinned dependencies in its README. Set PYAI_API_KEY and your LLM credential on the server, then start examples/bot.py from the package directory. The runner prints the local URL for connecting your microphone. Before starting, verify the intended project and limits with GET /v1/me. The example’s 60-second default limits session duration, not spending; PyAI, your LLM, and hosting have separate billing. Its Pipecat 1.14.0 setup passed non-audio startup and cleanup checks. The older two-turn audio qualification below does not establish live speech compatibility for every newer framework pin.

Download the existing starter

The Pipecat starter includes pinned dependencies, source, environment setup and run commands. It is a text-to-WAV pipeline, not a microphone or transport agent. For a complete conversation, continue below with your transport, LLM and tools. Check GET /v1/me for the intended project, billing and usage limits before running; the starter generates real audio. Machine-readable download instructions.

Try Hear and Speak before integrating

Open the live voice demo: record a question about PyAI, see Hear’s transcript, and listen to Speak’s reply. You can also type. The product answers are scripted; transcription and speech use real PyAI APIs. Three free questions, no signup required; daily demo limits apply. The hosted demo is a standalone Hear + Speak reference app. To use these APIs in your own application, get a free API key and follow the setup below. The demo also includes a signup link.

What stays under your control

Pipecat still owns frame order and routing. You can place custom processors before, between, or after the PyAI services and keep the deployment model you already use.

Prerequisites

  • Python 3.11 or newer
  • A Pipecat application with a transport and LLM service
  • A PyAI key with hear:stream and speak:synthesize scopes
  • A voice id from GET https://api.pyai.com/v1/voices
A pyai_test_ sandbox key includes the required Hear and Speak scopes, starts working immediately, and does not require billing. Keep the key on the server that runs the Pipecat pipeline.

Version compatibility and test setup

The two-turn media → Hear → LLM → Speak path was exercised with pipecat-ai==1.8.1, pyai-sdk==0.5.0, and Python 3.12. Keep your transport, VAD, and LLM dependencies compatible with that framework version. Your LLM and media hosting have their own credentials and costs. Before running, use GET /v1/me to verify the expected organization, project, env: test, and effective limits. Set explicit audio duration, request and runtime bounds; environment alone does not guarantee a daily spending cap. Shared speech capacity can return HTTP 429 even below the key’s concurrency ceiling. Respect Retry-After and keep retries bounded.

Install

Install the verified public release from PyPI:
Keep any Pipecat transport, VAD, and LLM extras your application already installs. Then set the PyAI key:

Add PyAI to the pipeline

Put Hear after the transport input and Speak before the transport output:
The names transport, user_aggregator, assistant_aggregator, and llm refer to the objects your Pipecat application already creates. Your VAD and turn configuration remain in effect. For a transport without built-in VAD, add your configured VADProcessor after transport.input() and before STT; the STT adapter uses the VAD stop event to request a committed final transcript.

Complete task shape

The transport remains responsible for joining the room or call and queuing the initial frames required by your application.

Configuration

Hear speech-to-text

The service opens the canonical PyAI Hear WebSocket, sends raw PCM audio frames, sends a commit when Pipecat VAD reports the end of speech, and emits InterimTranscriptionFrame plus finalized TranscriptionFrame objects. Hear streaming detects the language automatically when you omit language. Hear supports 29 recognition languages. To pin recognition, use a supported code from the language table. Language support is separate from measured accuracy on your audio; test your accents, noise and terminology before rollout.

Speak text-to-speech

The service sends Pipecat text frames to Speak and yields raw mono TTSAudioRawFrame objects at the configured rate. Use a streaming-capable stock, cloned, or designed voice id available to the same PyAI organization.

Share a TTS HTTP session

For applications that already manage an aiohttp.ClientSession, pass it to the TTS service:
The TTS service will reuse that session and leave its lifecycle under your application’s control.

Run and verify

1

Start your Pipecat bot

Run the same transport and deployment command you use today.
2

Send a two-turn conversation

Speak one short request, wait for agent audio, then interrupt or ask a follow-up. This verifies transcript finalization, context flow, TTS, and interruption behavior.
3

Inspect pipeline metrics

Confirm the Pipecat task receives transcript frames and TTS usage metrics. Log the transport region, worker region, LLM, VAD settings, voice, and PyAI region with any latency result.
4

Exercise failure handling

Test an invalid key, missing scope, and exhausted concurrency so your pipeline presents a safe fallback instead of silently stopping.

Production notes

  • Keep your Pipecat worker, transport media server, and PyAI speech endpoints in nearby regions.
  • A running Hear service counts against the PyAI key’s realtime concurrency.
  • Hear and Speak meter against your PyAI account; see the pricing page for current terms. Pipecat Cloud, transport, your LLM, and worker hosting remain separate.
  • Keep a bounded queue and timeout around custom processors so one integration cannot stall caller audio indefinitely.
  • Use Omni instead when you want PyAI to own turn-taking, reasoning, tools, caller continuity when an Agent profile has continuity enabled and a caller key is available, and optional managed telephony as one system.

Troubleshooting

Which path should I choose?

Choose Pipecat + PyAI for custom frame processors, transport flexibility, branching, and per-stage observability. Choose Omni when one managed speech-to-speech contract is more valuable than operating the pipeline.

Pipecat documentation

Review Pipecat’s pipeline, transport, and deployment model.

PyAI Omni overview

Compare the managed speech-to-speech path.

Speech To Text (Hear)

Streaming behavior, language support, and transcript formatting.

Text To Speech (Speak)

Voice selection, streaming modes, and PCM output.

Telephony audio

Phone codecs, PCM rates, and resampling boundaries.

Clone

Enroll and use a consented custom voice.

Measuring a first saved artifact

The downloadable Pipecat starter validates the closed WAV before sending an optional metadata-only saved-file receipt. It sends only the original Speak request ID and a fixed starter source with the same key; no audio, text, filename or local path is uploaded for analytics. Set PYAI_REPORT_ARTIFACTS=0 to disable reporting. Reporting failure never retries speech generation. The integration growth report matches receipts to successful Speak usage and separates the first saved artifact, another saved artifact on a later UTC day, and later-day API usage from the signup cohort. These are client-reported saves corroborated by server usage, not proof of audible playback. The complete WebRTC example streams audio and does not save a WAV or emit this receipt. Existing package installs and older starter downloads are not retroactively instrumented.