Skip to main content
Build a media bridge with one bidirectional Twilio stream per person and one PyAI interpretation WebSocket. The bridge decodes each microphone stream, routes it to the correct participant and plays translated speech to the opposite participant.
The SDK provides the two-party translation room and Twilio media transport. Your server remains responsible for owning both calls, validating Twilio’s signed WebSocket upgrade, storing transcripts and call events, and allowlisting internal contacts. Bounded synthetic browser-call tests passed for Hindi ↔ English and Punjabi ↔ English, including both directions, echo blocking, saved transcripts and replacement-agent transfers. Validate your own call setup and human speech quality before production.
Read the shared architecture and PyAI protocol first. Keep your Twilio credentials and live interpret:session key on the bridge server. The repository includes a bounded twilio-live-translation test host for dedicated customer, agent and replacement-agent browser calls. Build the current recipe and SDK from the same repository revision. The host validates webhook signatures, issues single-use participant links, persists transcripts to private storage and applies carrier time limits. Tests used real Twilio calls and the deployed PyAI API; they did not certify PSTN routing, physical speaker audibility or JustCall integration.

Recordings and transcripts in Console

Open Live Translation in Console, select your team’s live project, and start a new conversation or call. Confirm that everyone agrees to recording before starting. For a carrier call, the Console explicitly shows whether the connected host supports recording. Choose End conversation or End call and wait for saving to finish. In Call history, select the call to play its recording, download the audio, or read and download the original and translated transcripts. Other signed-in members of the same project can open the same history. Invitation links and upload tokens do not grant access to stored recordings. For a per-person recording, use Listening view to select Person A or Person B. The left channel contains that person’s original speech; the right contains the incoming translation sent to that person. The agent view includes the replacement agent after a transfer, within the same call record. Browser recordings contain microphone audio and translations rendered by the browser. Carrier recordings contain admitted source audio and translations sent to the call. Cancelled playback can differ from what reached the listener; recordings do not prove that someone heard every sent sample. Older recordings may have a single combined view. Read original speech beside its translation using Combined or By language. Search or filter Call history by language pair to find earlier calls. Both orientations of a pair appear together when history is grouped by pair. Calls made before recording was enabled have no recoverable audio. An incomplete upload is labelled Incomplete, and any successfully saved transcript remains available. Recordings stay private; playback and download require current project access.

Test a conversation in Console

Open Live Translation, select a live project and choose Phone call test. The internal tester uses its configured calling backend automatically; there is no provider selector. Hosted test access requires a verified saaslabs.co account. Choose the customer and agent languages, create a test room, and share their separate invitations. Use separate devices and headphones. Select Transfer to invite a replacement agent, then End call and wait for cleanup before opening Call history for available recordings and transcripts. Your own Twilio application is configured in your code. Open the Console’s Integration tab for the shared PyAI room code and Twilio transport excerpt, then follow the complete webhook and media setup below. The internal tester does not configure your Twilio account or certify your application’s call routing.

Connect the customer call

Return this TwiML from your authenticated answer webhook. The WebSocket URL belongs to your bridge, not the PyAI API:
Create a separate outbound call to the human agent, with a server-controlled answer webhook returning the same structure with leg=agent. Pair both calls in trusted server state. The SDK’s TwilioStreamAdmission issues a short-lived, single-use path and validates X-Twilio-Signature; use that binding together with start.accountSid, start.callSid and start.streamSid before creating the media transport. Twilio supports one bidirectional stream per call and delivers its inbound track. <Connect><Stream> blocks following TwiML while connected. A <Start><Stream> or ordinary conference audio tap cannot supply the required listener-specific bidirectional output. See Media Streams and Stream parameters.

Authenticate and convert the audio

Validate X-Twilio-Signature on webhooks and the WebSocket upgrade using Twilio’s supported validation library and the correct public request URL behind your proxy. Bind the expected account, start.callSid, start.streamSid and room/leg to the authenticated call record. Stream URLs do not support query parameters; use nested Parameter elements for pairing metadata. Twilio media uses base64 headerless μ-law at 8 kHz. Decode and statefully resample to PyAI’s 16 kHz PCM16 input. Convert output back to μ-law at 8 kHz, with bounded, paced queues. The existing @pyai/twilio codec and streaming resampler helpers are used by TwilioInterpretationTransport; pair it with InterpretationRoom and a durable record callback. On the intended listener’s stream, send:

Map marks without inventing playback

After the final media for an utterance, send an identity-bound mark:
Track its stream, participant epoch, output generation, utterance and original PyAI sample count. A returned current mark may confirm complete playback only if that boundary was not invalidated by a clear. For cancellation, invalidate pending marks before stopping local output and sending:
Twilio also returns pending marks for audio discarded by clear. Those marks must never become status: "played". See Twilio’s mark and clear semantics. Twilio has no distinct documented clear acknowledgement. The SDK sends a fresh sentinel mark after clear on the same ordered socket and waits for that barrier before reporting playback.cleared. Carrier transfer tests returned the barrier and resumed the replacement participant. Unit tests separately reject invalidated old marks; the carrier tests did not naturally produce a late stale mark.

Handle transfers and hangup

Perform transfers with Twilio call control and the replacement sequence in the shared guide. Clear both playback queues and fence the previous agent’s call/stream before admitting the new agent. DTMF on bidirectional Media Streams is inbound-only; keep outbound DTMF in your call-control design. When either call ends, apply your room’s waiting/hangup policy, send session.end and close the PyAI socket promptly. Do not replay uncertain speech after reconnect. Validate both translation directions, late marks after clear, echo blocking and a real transfer before enabling customer traffic.