Skip to main content
More and more calls are being automated using voice agents. A long pause, a robotic filler, or an agent that talks over them makes callers hang up or ask for a person. This is why builders are under pressure to make automated calls feel natural, and to get the job done during the call. Prism is built exactly for that. It’s a speech-to-speech model that hears the caller’s audio, not just a transcript. It replies at a natural conversation pace. When it needs to look something up, it says something like “Let me check that” while your system works behind the scenes. This way there’s no dead air. Prism speaks in Rime’s natural-sounding voices, which are what callers judge first. Additionally, enterprises can fine-tune Prism to their calls, which can significantly increase the rate of call completion. This ultimately maximizes business outcomes at large call volumes.

A caller reports a lost debit card. Prism finds the card, freezes it, reads the recent charges, and files a dispute.

Build a Prism agent with the SDK

Talk to Prism through your microphone with your own instructions, from Python or Node.js.

​
What Prism handles for you

A voice agent has to listen, know when the caller is done, answer, and stay out of the caller’s way. Prism does all of this over one connection:
  • It hears the caller. Prism works from the caller’s audio, while a cascade’s language model only reads a transcript.
  • It knows when to speak. Prism decides when the caller has finished, and it doesn’t talk over them.
  • It lets the caller cut in. When the caller talks over a reply, Prism stops and answers what they just said.
  • It replaces three services. One WebSocket does the work of speech-to-text, a language model, and text-to-speech, with your instructions and tools.
  • It sends text with every reply. Each spoken reply comes with its text, so you can show captions and keep logs.

​
Before you build

Here’s what to know first:

​
What happens during a call

Your backend sits between the caller and Prism. It streams the caller’s audio to Prism the whole time, plays Prism’s replies, and runs your tools when Prism asks. Here’s how one exchange flows: The caller’s audio can reach your backend in three ways:
  • A browser or mobile app: the app streams the microphone to your backend. The SDK quickstart runs this setup on your computer.
  • A phone call: your telephony provider streams the call to your backend. Convert the phone audio in both directions, as Phone audio describes.
  • A LiveKit room: the LiveKit quickstart adds Prism to a LiveKit agent, and LiveKit telephony connects phone calls to it.
A few things make the difference between a call that feels natural and one that doesn’t:
  1. Keep the API key on your backend.
  2. Keep the audio flowing while Prism speaks. Prism listens to it to catch interruptions.
  3. Send a playback report after an interruption. Prism uses it to decide how much of the interrupted reply it remembers.
  4. Skip the turn detector. You don’t need voice activity detection either, and How a turn ends explains why.

​
Connect Prism to your app

Most teams start with the Rime SDK. If you already use LiveKit Agents, start with the LiveKit quickstart instead. If your backend is in another language, use the WebSocket API directly:

​
Make the agent yours

Prism comes with sensible defaults, so a short set of instructions is enough to start. As your agent grows, these are the controls you’ll reach for: Prism doesn’t offer these settings:
  • Text-only replies
  • Sampling settings, such as temperature
  • A choice of language model
  • A way for your application to end the caller’s turn
If you need any of these, a pipeline with your own language model may fit better. Build a voice agent compares the two.

​
Know the limits

Long calls and busy agents run into a few limits. Knowing them up front saves you a surprise in production:
  • Fixed settings: the turn timing stays as you set it when the call starts. You can still change the instructions, voice, and tools during the call, with the SDK or the WebSocket API. Configure a session lists which settings can change.
  • Connection: one conversation runs on each connection. A dropped connection does not resume on its own. To carry on, open a new session and add the earlier conversation, as Start from an earlier conversation shows.
  • Turn length: a caller’s turn ends after 55 seconds of continuous speech.
  • Idle time: the conversation closes after 5 minutes idle. Silence from your backend does not keep it open.
  • Context: 65,536 tokens in all: instructions, tools, history, and tool results, plus 512 tokens kept for the reply. When a conversation outgrows it, Prism leaves the oldest turns out of each reply’s prompt, so the conversation goes on.
The errors page lists every limit.

​
Moving from OpenAI Realtime

Prism uses the event names of the OpenAI Realtime beta interface. So an OpenAI Realtime client works with Prism after a few changes, and Move from OpenAI Realtime walks through them, including LiveKit agents.