Build a LiveKit voice agent with Prism

In this quickstart, you’ll build a LiveKit voice agent with Prism as its model, and talk to it from your browser. Your agent answers questions about a made-up cafe, and runs a function in your code to look up its hours. Prism plugs into LiveKit Agents as a realtime model, like OpenAI Realtime. It replaces a pipeline agent’s speech-to-text, LLM, and text-to-speech plugins: Prism listens, finds the end of each turn, and replies out loud. LiveKit carries the audio between the room and your agent.

What you’ll need

A Rime API key. If you don’t have one, create one.
A LiveKit Cloud project, with its URL, an API key, and the key’s secret. Your own LiveKit server works too.
uv 0.9.26 or later, which installs Python for the project if needed. The Rime SDK needs Python 3.11 or later.
A browser with a microphone.

Build the agent

Create the project

Create a Python project:

uv init --python 3.12 prism-livekit-agent
cd prism-livekit-agent

Before you install anything, add this to pyproject.toml, so the plugin’s Git workspace doesn’t replace the published LiveKit Agents:

pyproject.toml

[tool.uv]
no-sources-package = ["livekit-agents"]

Now install the packages, pinned to the versions we tested. The Rime plugin is a preview build from Rime’s fork of LiveKit Agents, because no LiveKit release has Prism support yet. Its realtime extra adds the Rime SDK:

uv add "livekit-agents[codecs]==1.8.9" \
  "livekit-plugins-rime[realtime] @ git+https://github.com/rimelabs/agents@82eb9300776981e0863900eee75475a6f92766da#subdirectory=livekit-plugins/livekit-plugins-rime" \
  "rimelabs-sdk==0.1.0a24" \
  "python-dotenv>=1.1"

Set your credentials

Create .env.local in your project folder with your own values. Your LiveKit Cloud project’s settings show its URL and let you create an API key and secret:

.env.local

LIVEKIT_URL=wss://YOUR_PROJECT.livekit.cloud
LIVEKIT_API_KEY=YOUR_LIVEKIT_API_KEY
LIVEKIT_API_SECRET=YOUR_LIVEKIT_API_SECRET
RIME_API_KEY=YOUR_RIME_API_KEY

Save the agent

Save this script as agent.py in your project folder. It registers an agent named prism-agent, which starts a Prism session in the aurora voice and greets the caller in each room it joins.

agent.py

from dotenv import load_dotenv
from livekit import agents
from livekit.agents import Agent, AgentServer, AgentSession, function_tool, room_io
from livekit.plugins import rime

load_dotenv(".env.local")


class CafeAgent(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions=(
                "You answer the phone for Juniper Cafe. Answer in one or two short "
                "sentences. For questions about opening hours, use get_opening_hours."
            ),
        )

    @function_tool
    async def get_opening_hours(self) -> str:
        """Look up Juniper Cafe's opening hours for today."""
        return "Juniper Cafe is open from 8 a.m. to 4 p.m. today."


server = AgentServer()


@server.rtc_session(agent_name="prism-agent")
async def prism_agent(ctx: agents.JobContext) -> None:
    model = rime.realtime.RealtimeModel(voice="aurora")
    ctx.add_shutdown_callback(model.aclose)
    session = AgentSession(llm=model)
    await session.start(
        agent=CafeAgent(),
        room=ctx.room,
        room_options=room_io.RoomOptions(text_input=False),
    )
    await session.generate_reply(
        instructions="Greet the caller and ask how you can help."
    )


if __name__ == "__main__":
    agents.cli.run_app(server)

Here’s what each part does:

rime.realtime.RealtimeModel: Prism, passed to AgentSession as its llm. It connects to wss://prism.api.rime.ai/v1/realtime by default. Prism does all three jobs, so the session has no stt, tts, vad, or turn_detection.
voice: aurora, wawona, brando, or astra.
CafeAgent: Prism’s instructions and tools.
@function_tool: makes get_opening_hours a tool, described to Prism by its docstring. Prism decides when to call it, LiveKit runs it in your agent, and Prism replies with the result.
text_input=False: keeps this example voice only. It’s optional, because the plugin also takes typed input through the history.
ctx.add_shutdown_callback(model.aclose): closes the model and its SDK client when the job ends.
generate_reply: asks Prism to speak first.

Start the agent

Run it in development mode from your project folder:

uv run agent.py dev

The agent connects to your LiveKit project and logs registered worker with "agent_name": "prism-agent". Leave it running. You can ignore the warning that dev is deprecated, or use lk agent dev from the LiveKit CLI instead.

Talk to the agent

In LiveKit Cloud, open your project’s Agent Console.
Set the agent name to prism-agent.
Select Start a session.
When your browser asks, allow microphone access.

Prism greets you as soon as you join, so you know the session works. Now ask, “What time do you close today?”While the tool runs, Prism can say a short filler phrase, or skip it if it already spoke before the call. Then you’ll hear the hours from get_opening_hours: 8 a.m. to 4 p.m. Only the tool result has them, so hearing 4 p.m. means the room, the Prism session, turn detection, and your tool all work.Next, try to cut in: talk while Prism replies, and it stops to answer you.If the call doesn’t work:

The agent never joins: check that the console’s agent name is prism-agent and the terminal shows registered worker.
The agent joins but stays silent: look for a Rime plugin error in the terminal, and look up its code in Errors and limits.

How Prism changes a LiveKit agent

Prism runs the whole conversation, so a few LiveKit features work differently than in a pipeline agent:

Feature	In a Prism agent
Turn detection	Prism finds the end of each turn. Don’t add `vad`, `turn_detection`, or a turn detector plugin.
Turn taking	Prism doesn’t talk over the caller. It keeps listening while it speaks, so the caller can interrupt.
Settings	Change them mid-call with `agent.update_instructions()`, `agent.update_tools()`, or `model.update_session(voice="...")`. Handoffs work, but settings and history update separately, so a handoff isn’t atomic.
Memory	Each session starts fresh, but you can give it earlier text and completed tool calls. A dropped connection can’t resume.
Tool calls	Prism chooses when to call tools. `tool_choice="none"` removes the tools for a while, so no reply calls one, not even a reply that starts on its own. Forced or named choices aren’t supported.
Tool step limit	`max_tool_steps` is 3 by default. On LiveKit Agents 1.8.9, a reply that reaches it gets a final answer without tools.
Speech	All speech comes from Prism. To make it speak, use `generate_reply()` with instructions, as the greeting does. `session.say()` needs a text-to-speech model, and this session has none.
Unsupported features	Video input, manual audio commit, per-reply tools, and turning off server turn detection
Caller transcripts	The plugin passes on interim and final transcripts when Prism sends them. A final transcript can arrive after Prism starts its reply.
Language	English only, in Prism’s own four voices

To choose your own speech-to-text and LLM instead, build a pipeline agent with the LiveKit text-to-speech quickstart, where Rime gives the agent its voice.

Use your own LiveKit server

On your own server, which has no Agent Console, send the agent to a room with the LiveKit CLI:

In .env.local, set LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET to your server’s values, and start the agent the same way.
Send the agent to a room. Pass your server details with --url, --api-key, and --api-secret, or set the same three variables in your shell.
lk dispatch create --room prism-test --agent-name prism-agent
Join the prism-test room from any LiveKit client with a microphone, such as an app built with LiveKit’s frontend SDKs.

Next steps

Now that your agent answers, you can:

Take phone calls with LiveKit telephony.
Write instructions for a phone call, starting with rules for speech.
Choose a voice that suits your callers.
Move an OpenAI Realtime agent on LiveKit to Prism.

Get started

Build with Rime

Integrations

Account & updates

Build a LiveKit voice agent with Prism

What you’ll need

Build the agent

How Prism changes a LiveKit agent

Use your own LiveKit server

Next steps

​What you’ll need

​Build the agent

​How Prism changes a LiveKit agent

​Use your own LiveKit server

​Next steps

What you’ll need

Build the agent

How Prism changes a LiveKit agent

Use your own LiveKit server

Next steps