Skip to main content
Use Live Translation when a customer and a human agent speak different languages. Your calling platform owns the calls; a server-side media bridge connects their separate audio streams to one PyAI interpretation session.
The SDK now includes a two-party room, Twilio media transport, bounded call creation and transfer hooks, and a one-time signed stream admission helper. You still operate the server that stores call and transcript records, validates Twilio upgrades, applies your internal contact allowlist, and owns both calls. Bounded browser-call tests used synthetic speech with Twilio and Plivo for Hindi ↔ English and Punjabi ↔ English. Each provider completed a ten-room run covering playback, echo blocking, saved recordings and transcripts, and agent replacement. These results do not establish human speech quality, PSTN routing, JustCall integration, or capacity and failover for your deployment.
Punjabi captions in the Plivo Console path apply a narrow, display-only spelling correction for the weekday Friday. The raw recognition remains available with its offsets and correction metadata; translation input is not changed. This does not improve recognition accuracy or establish broad Punjabi speech quality. The internal Plivo tester admits up to ten concurrent rooms within its shared test allowance. It is separate from the customer API and does not provision or host your calling integration. Follow the provider-specific guides for setup. Each two-human call needs two independent carrier media streams and one PyAI interpretation session. Ten calls therefore need twenty carrier media sockets and ten PyAI sessions, plus headroom for reconnects. A bridge’s configured room maximum is not evidence of speech-service capacity. Verify the effective limit before a load test.

Keep each call on its owning bridge

The example bridge keeps live room state in one process. Both participants’ media sockets, carrier callbacks and call controls must reach that same owner. An autoscaling maximum of one does not establish this routing guarantee. For a bounded Cloud Run deployment, use service-level manual scaling with one instance (--scaling=1) and request-based billing (--cpu-throttling). This keeps one warm worker and incurs idle charges. Pause new rooms and verify that all existing rooms have ended before replacing it. For multiple workers, add room-owner routing and durable coordination before enabling autoscaling. This deployment setting does not establish failover or concurrent-call capacity.

Onboard a customer

  1. Create or select the customer’s organization and a Live project in Console. Keep each customer’s keys, credit balance and recordings in that customer’s organization.
  2. Create a key with only Translation sessions (interpret:session). Set an appropriate monthly budget and store the secret on the bridge server. Test-environment keys cannot open Live Translation sessions.
  3. Add prepaid credit or use an existing credit grant. A live key with no available credit is rejected with 402; an internal rehearsal that bypasses project billing does not verify customer billing.
  4. Configure the customer’s Twilio or Plivo application, signed webhooks and separate media legs. Use the published @pyai/twilio SDK’s InterpretationRoom and provider transport in your application. The example hosts are bounded rehearsals with explicit deadlines and room limits; they are not permanent customer hosting.
  5. Open one interpreter after both carrier legs are ready, retain its request identity alongside the carrier call IDs, and close it when the conversation ends. Reconcile that identity with Analytics → Usage & cost and Request logs before enabling more traffic.
See PyAI pricing for the current connected call-minute rate. Both directions and generated translated speech are included. Silence during a connected session counts; waiting for the second carrier leg should not open an interpreter. Transfers reuse the same session. Carrier charges are separate. Rejected HTTP upgrades do not incur connected-session usage.

Customer call-history API

Save a call from your own server and retrieve its original/translated transcripts, transfer events and per-party audio with project-scoped API keys. See Translation call history and recordings for archive creation, upload, playback/download and the supported capture limits. This works independently of the internal Console tester.

Provider adapters in your code

The PyAI Translation API has no calling-provider selector. Configure your provider account, signed webhooks and media transport in your own server code. The InterpretationRoom contract, language configuration, transcripts and participant-replacement flow stay the same. Provider differences are at the media and call-control boundary. The Console’s Integration tab shows the shared PyAI room code beside Twilio and Plivo transport excerpts. Selecting a code example does not configure or switch a calling backend. The internal Phone call test uses its configured backend automatically and is separate from customer-owned calling integrations.

Keep the two people on separate media legs

The initial design gives each person an independently controlled provider call and bidirectional WebSocket. The bridge pairs those sockets using a server-owned room and opens one PyAI session for the conversation. Customer microphone audio becomes leg: "customer"; agent microphone audio becomes leg: "agent". Output addressed to target_leg: "agent" plays only to the agent, and vice versa. A normal conference mix can expose original speech alongside its translation and feed playback back into recognition. A conference-based implementation needs separately proven selective routing and muting. For the first bridge, keep both call legs isolated and route their speech through the interpreter.

Open the PyAI session

Keep a live key with interpret:session on your bridge server. Connect with Authorization: Bearer <live-key> to:
Wait for hello. Read supported_languages, the voice entries in languages, and the qualification information in experimental_languages and synthetic_audio_tested_pairs. The managed service currently exposes 17 languages; availability does not certify every language pair. Then send:
A participant’s voice generates speech to that person. Wait for session.started and store the acknowledged revision, epochs and output generations. See the Live Translation API guide for the full wire contract.

Convert and route media continuously

Send raw mono signed little-endian PCM16 at 16 kHz, without a WAV header. A 20 ms frame is 640 bytes; each frame must be nonempty and at most 200 ms.
Use the returned epoch rather than assuming 1. Sequences start at zero and increase by exactly one per leg epoch. Preserve resampler state across provider packets; do not independently resample each packet. Route each audio.chunk by its target_leg, target_epoch, output_generation and increasing audio_sequence. Maintain bounded output queues per listener. input.commit flushes a source context; translation does not need to wait for the end of the entire call.

Confirm playback and prevent echo

For every utterance, retain the generated 16 kHz sample count and its participant identity. A socket send or audio.completed proves neither playback nor audibility. Send playback.report with status: "played" only after a valid provider playback receipt confirms the complete generated utterance. Count original PyAI samples, excluding codec padding and resampler tail. On interruption, record only the confirmed played prefix; keep any unknown position explicit. Respect input.gated and input.resumed. Drop gated microphone audio instead of saving it for later replay. The service’s strict per-listener playback gate and 250 ms tail help prevent translated speech from being translated back; they do not establish acoustic echo cancellation or simultaneous overlap quality. On output.cancelled, stop local pacing, invalidate pending playback receipts, clear the provider queue and discard stale converter state. Send playback.cleared only after the provider’s physical flush is established for the current leg epoch and output generation. An unconfirmed flush must keep input gated or end the session safely. For a valid input.speech_started control, advance the named listener’s tracked output generation once. The resulting output.cancelled events retain the cancelled utterances’ older generations. A playback.cleared control must use the current leg, epoch and output_generation; do not copy the old generation from a cancellation event. Serialize these controls and do not replay an interruption whose acceptance is uncertain.

Preserve transfers and the customer journey

Your application performs the actual carrier transfer and maintains the stable conversation ID. Bind each replacement agent to a new authenticated call socket; fence the old socket before admitting replacement audio. Send leg.replace with the expected session revision, new participant, language and voice. After session.updated, adopt both legs’ new epochs and generations, reset both input sequences, physically clear both playback queues, and acknowledge playback.cleared for both. Changing interpreter state alone does not transfer a telephone call. Persist transcript.committed and translation.committed as append-only ledgers, with their own offsets and sequences joined by source_context_id. Store provider call/stream IDs, PyAI session and participant IDs, transfer history and playback outcomes on the same call record. Transcript storage is your responsibility; no interpretation transcript retrieval endpoint exists.

Limits, billing and the first acceptance call

Start the session within 15 seconds of connecting. Current limits include 60 seconds without client input, 30 minutes per session and four connected sessions in the initial fleet. Transport ping/pong does not replace application input. Inspect the hello context bounds and do not promise transparent replay or gapless continuation after reconnect. PyAI charges $0.06 per connected conversation-minute, both directions included. Silence counts. Both carrier call legs, provider stream fees, phone numbers and your bridge hosting are additional charges. Open one PyAI session for the paired conversation to avoid duplicate interpretation billing. Before enabling customer calls, prove a real conversation in both directions, then cancellation, strict echo injection, participant replacement, late receipts, hangup and cleanup. Join timings from provider input through PyAI processing to carrier playback receipts. Measure those results separately from browser tests and from what a human actually hears. For JustCall, attach these events to the existing call/journey record and run the same transfer and recording checks. These transport guides do not establish a completed JustCall integration.