> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pyai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Live Translation with Twilio

> Connect two human Twilio calls to one PyAI interpretation session using bidirectional Media Streams.

Build a media bridge with one bidirectional Twilio stream per person and one
PyAI interpretation WebSocket. The bridge decodes each microphone stream,
routes it to the correct participant and plays translated speech to the
opposite participant.

> [!NOTE]
>
> The SDK provides the two-party translation room and Twilio media transport.
> Your server remains responsible for owning both calls, validating Twilio's
> signed WebSocket upgrade, storing transcripts and call events, and allowlisting
> internal contacts. Bounded synthetic browser-call tests passed for Hindi ↔ English
> and Punjabi ↔ English, including both directions, echo blocking, saved transcripts
> and replacement-agent transfers. Validate your own call setup and human speech
> quality before production.

Read the [shared architecture and PyAI protocol](https://docs.pyai.com/guides/live-translation-telephony.md)
first. Keep your Twilio credentials and live `interpret:session` key on the
bridge server.

The repository includes a bounded
`twilio-live-translation` test host
for dedicated customer, agent and replacement-agent browser calls. Build the
current recipe and SDK from the same repository revision. The host validates
webhook signatures, issues single-use participant links, persists transcripts
to private storage and applies carrier time limits. Tests used real Twilio calls
and the deployed PyAI API; they did not certify PSTN routing, physical speaker
audibility or JustCall integration.

## Recordings and transcripts in Console

Open [Live Translation in Console](https://console.pyai.com/live-translation),
select your team's live project, and start a new conversation or call. Confirm
that everyone agrees to recording before starting. For a carrier call, the
Console explicitly shows whether the connected host supports recording.

Choose **End conversation** or **End call** and wait for saving to finish. In
**Call history**, select the call to play its recording, download the audio, or
read and download the original and translated transcripts. Other signed-in
members of the same project can open the same history. Invitation links and
upload tokens do not grant access to stored recordings.

For a per-person recording, use **Listening view** to select **Person A** or
**Person B**. The left channel contains that person's original speech; the right
contains the incoming translation sent to that person. The agent view includes
the replacement agent after a transfer, within the same call record.

Browser recordings contain microphone audio and translations rendered by the
browser. Carrier recordings contain admitted source audio and translations sent
to the call. Cancelled playback can differ from what reached the listener;
recordings do not prove that someone heard every sent sample. Older recordings
may have a single combined view.

Read original speech beside its translation using **Combined** or **By language**.
Search or filter Call history by language pair to find earlier calls. Both
orientations of a pair appear together when history is grouped by pair.

Calls made before recording was enabled have no recoverable audio. An incomplete upload is labelled
**Incomplete**, and any successfully saved transcript remains available.
Recordings stay private; playback and download require current project access.

## Test a conversation in Console

Open [Live Translation](https://console.pyai.com/live-translation#call-test),
select a live project and choose **Phone call test**. The internal tester uses
its configured calling backend automatically; there is no provider selector.
Hosted test access requires a verified saaslabs.co account.

Choose the customer and agent languages, create a test room, and share their
separate invitations. Use separate devices and headphones. Select **Transfer**
to invite a replacement agent, then **End call** and wait for cleanup before
opening **Call history** for available recordings and transcripts.

Your own Twilio application is configured in your code. Open the Console's
[Integration tab](https://console.pyai.com/live-translation#integration) for the
shared PyAI room code and Twilio transport excerpt, then follow the complete
webhook and media setup below. The internal tester does not configure your
Twilio account or certify your application's call routing.

## Connect the customer call

Return this TwiML from your authenticated answer webhook. The WebSocket URL
belongs to **your bridge**, not the PyAI API:

```xml theme={null}
<Response>
  <Connect>
    <Stream url="wss://bridge.example.com/twilio/media/one-time-server-token" />
  </Connect>
</Response>
```

Create a separate outbound call to the human agent, with a server-controlled
answer webhook returning the same structure with `leg=agent`. Pair both calls
in trusted server state. The SDK's `TwilioStreamAdmission` issues a short-lived,
single-use path and validates `X-Twilio-Signature`; use that binding together
with `start.accountSid`, `start.callSid` and `start.streamSid` before creating
the media transport.

Twilio supports one bidirectional stream per call and delivers its inbound
track. `<Connect><Stream>` blocks following TwiML while connected. A
`<Start><Stream>` or ordinary conference audio tap cannot supply the required
listener-specific bidirectional output. See
[Media Streams](https://www.twilio.com/docs/voice/media-streams) and
[Stream parameters](https://www.twilio.com/docs/voice/twiml/stream).

## Authenticate and convert the audio

Validate `X-Twilio-Signature` on webhooks and the WebSocket upgrade using
Twilio's supported validation library and the correct public request URL behind
your proxy. Bind the expected account, `start.callSid`, `start.streamSid` and
room/leg to the authenticated call record. Stream URLs do not support query
parameters; use nested `Parameter` elements for pairing metadata.

Twilio media uses base64 headerless μ-law at 8 kHz. Decode and statefully
resample to PyAI's 16 kHz PCM16 input. Convert output back to μ-law at 8 kHz,
with bounded, paced queues. The existing `@pyai/twilio` codec and streaming
resampler helpers are used by `TwilioInterpretationTransport`; pair it with
`InterpretationRoom` and a durable `record` callback.

On the intended listener's stream, send:

```json theme={null}
{"event":"media","streamSid":"target-stream-sid","media":{"payload":"<base64 raw mulaw8k>"}}
```

## Map marks without inventing playback

After the final media for an utterance, send an identity-bound mark:

```json theme={null}
{"event":"mark","streamSid":"target-stream-sid","mark":{"name":"unique-playback-boundary"}}
```

Track its stream, participant epoch, output generation, utterance and original
PyAI sample count. A returned current mark may confirm complete playback only
if that boundary was not invalidated by a clear.

For cancellation, invalidate pending marks **before** stopping local output and
sending:

```json theme={null}
{"event":"clear","streamSid":"target-stream-sid"}
```

Twilio also returns pending marks for audio discarded by `clear`. Those marks
must never become `status: "played"`. See
[Twilio's mark and clear semantics](https://www.twilio.com/docs/voice/media-streams/websocket-messages).

Twilio has no distinct documented clear acknowledgement. The SDK sends a fresh
sentinel mark after clear on the same ordered socket and waits for that barrier
before reporting `playback.cleared`. Carrier transfer tests returned the barrier
and resumed the replacement participant. Unit tests separately reject invalidated
old marks; the carrier tests did not naturally produce a late stale mark.

## Handle transfers and hangup

Perform transfers with Twilio call control and the replacement sequence in the
[shared guide](https://docs.pyai.com/guides/live-translation-telephony.md). Clear both playback queues
and fence the previous agent's call/stream before admitting the new agent.
DTMF on bidirectional Media Streams is inbound-only; keep outbound DTMF in your
call-control design.

When either call ends, apply your room's waiting/hangup policy, send
`session.end` and close the PyAI socket promptly. Do not replay uncertain
speech after reconnect. Validate both translation directions, late marks after
clear, echo blocking and a real transfer before enabling customer traffic.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.