import asyncio
import json
import websockets
import base64
class RimeClient:
def __init__(self, speaker, api_key):
self.url = f"wss://users-ws.rime.ai/ws3?speaker={speaker}&modelId=mistv3&audioFormat=mp3"
self.auth_headers = {
"Authorization": f"Bearer {api_key}"
}
self.audio_data = b''
async def send_messages(self, websocket, messages):
for message in messages:
await websocket.send(json.dumps(message))
async def handle_audio(self, websocket):
while True:
try:
audio = await websocket.recv()
except websockets.exceptions.ConnectionClosedOK:
break
message = json.loads(audio)
if message['type'] == 'chunk':
self.audio_data += base64.b64decode(message['data'])
if message['type'] == 'timestamps':
print("Rime model pronounced the words...\n")
for w, t in zip(message['word_timestamps']['words'], message['word_timestamps']['start']):
print(f"'{w}' at time {t}")
async def run(self, messages):
async with websockets.connect(self.url, additional_headers=self.auth_headers) as websocket:
await asyncio.gather(
self.send_messages(websocket, messages),
self.handle_audio(websocket),
)
def save_audio(self, file_path):
with open(file_path, 'wb') as f:
f.write(self.audio_data)
print(f"\n Audio saved at {file_path}")
message = [
{"text": "This "},
{"text": "is "},
{"text": "an "},
{"text": "incomplete "},
{"text": "sentence "},
{"operation": "eos"},
]
client = RimeClient("cove", api_key="YOUR_API_KEY")
asyncio.run(client.run(message))
client.save_audio("output.mp3")
Legacy WebSockets
Websockets JSON
Mist v3 JSON WebSocket (/ws3): structured events with base64 audio chunks and word-level timestamps.
import asyncio
import json
import websockets
import base64
class RimeClient:
def __init__(self, speaker, api_key):
self.url = f"wss://users-ws.rime.ai/ws3?speaker={speaker}&modelId=mistv3&audioFormat=mp3"
self.auth_headers = {
"Authorization": f"Bearer {api_key}"
}
self.audio_data = b''
async def send_messages(self, websocket, messages):
for message in messages:
await websocket.send(json.dumps(message))
async def handle_audio(self, websocket):
while True:
try:
audio = await websocket.recv()
except websockets.exceptions.ConnectionClosedOK:
break
message = json.loads(audio)
if message['type'] == 'chunk':
self.audio_data += base64.b64decode(message['data'])
if message['type'] == 'timestamps':
print("Rime model pronounced the words...\n")
for w, t in zip(message['word_timestamps']['words'], message['word_timestamps']['start']):
print(f"'{w}' at time {t}")
async def run(self, messages):
async with websockets.connect(self.url, additional_headers=self.auth_headers) as websocket:
await asyncio.gather(
self.send_messages(websocket, messages),
self.handle_audio(websocket),
)
def save_audio(self, file_path):
with open(file_path, 'wb') as f:
f.write(self.audio_data)
print(f"\n Audio saved at {file_path}")
message = [
{"text": "This "},
{"text": "is "},
{"text": "an "},
{"text": "incomplete "},
{"text": "sentence "},
{"operation": "eos"},
]
client = RimeClient("cove", api_key="YOUR_API_KEY")
asyncio.run(client.run(message))
client.save_audio("output.mp3")
The Rime API authenticates every request with a bearer token in the
examples:
If you set
The three arrays inside
When exactly
Authorization header: Authorization: Bearer YOUR_API_KEY. See API authentication for how to create a key.
This page describes the legacy
/ws3 endpoint, which keeps working for existing integrations. For a new Mist v3 integration, use Mist v3 WebSocket v1.Overview
This WebSocket sends and receives JSON messages. Compared with the plain-text WebSocket, it adds word-level timestamps, adone event after each synthesis, and contextId tagging. You pass every synthesis parameter as a query parameter when you open the connection, and those settings apply for the life of the connection.
The API buffers input until it detects a sentence boundary, typically at ., ?, or !, so synthesis starts once it has enough text for natural prosody. The wait is most noticeable on the first sentence. After that, later text usually arrives while earlier audio is still playing, and the buffering becomes largely invisible. To change when synthesis starts, set segment.
Messages
Send
Text
Sends text for synthesis. This is the message your client sends most often. schema:type TextMessage = {
text: string,
contextId?: string,
}
{
"text": "this is the minimum text message."
}
{
"text": "this is a text message with a context id.",
"contextId": "159495B1-5C81-4C73-A51A-9CE10A08239E"
}
contextId, the server attaches it to the events it sends back for that text. The server tracks one context ID at a time.
Clear
Discards buffered text that hasn’t been synthesized yet. Send it when the user interrupts. It doesn’t cancel a synthesis already in progress, so stop playback in your client and drop any audio that still arrives.{ "operation": "clear" }
Flush
Synthesizes whatever text is buffered, without waiting for a sentence boundary, and sends the audio.{ "operation": "flush" }
EOS
Synthesizes whatever text is buffered, sends the audio, and then the server closes the connection.{ "operation" : "eos" }
Receive
Chunk
Carries a piece of the synthesized audio. This is the event your client receives most often.type Base64String = string
type AudioChunkEvent = {
type: "chunk",
data: Base64String,
contextId: string | null,
}
data holds base64-encoded audio in the audioFormat set at connection time. Decode each chunk and append the bytes in order to rebuild the audio. With pcm, a chunk can end mid-sample, so keep whole samples. If the text that produced this audio had a contextId, the event includes it.
Timestamps
The server sends word-level timestamps alongside the audio chunks, so your client can tell which words have been spoken at any point. The server estimates them from the text rather than measuring the audio. Treat them as approximate. When the user talks over the output, map the playback position back to the last word they actually heard.Timestamps are emitted only when
lang is en/eng or es/spa, or when lang is omitted. Requests in any other language receive chunk and done events with no timestamps event and no error (this includes fr and de). Don’t block playback while waiting for a timestamps event.type TimestampsEvent = {
type: "timestamps",
word_timestamps: {
words: string[],
start: number[],
end: number[],
},
contextId: string | null,
}
word_timestamps are the same length and index-aligned: for a given index i, words[i] is spoken from start[i] to end[i]. Times are in seconds, measured from the beginning of the audio for the current synthesis. If a context id was attached to the text that produced this audio, it is included on the event.
Example payload:
{
"type": "timestamps",
"word_timestamps": {
"words": ["Testing", "mistv3", "timestamps."],
"start": [0, 0.35396, 1.41584],
"end": [0.35396, 1.41584, 3.18564]
},
"contextId": null
}
Done
After the last audio chunk for a synthesis batch has been sent, the server emits adone event. This signals that the current synthesis is fully complete. If the client sends more text and triggers further synthesis, another done will follow.
type DoneEvent = {
type: "done",
contextId: string | null,
}
done fires depends on the segment setting. See Segmentation and behavior settings for full details.
Error
If a message is malformed or unexpected, the server responds immediately with anerror event. The server does not close the connection and keeps accepting well-formed messages, so your client decides whether to close it.
type ErrorEvent = {
type: "error",
message: string,
}
Variable parameters
string
required
Must be a voice from the Rime voice catalog.
string
default:"mistv3"
Set to
mistv3.string
One of
wav, mp3 (or mpeg), ogg (Opus in OGG), webm (Opus in WebM), pcm (or l16), or mulaw (or pcmu). Unrecognized values fall back to wav.string
default:"eng"
If provided, the language must match the language spoken by the selected speaker. Verify the pairing in the Rime voice catalog.
bool
default:"false"
When set to true, adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds.
Example:
Hi. <200> I'd love to have a conversation with you. adds a 200ms pause. Learn more about custom pauses.int
The sampling rate in Hz. Default: 24000. Values above 24000 are upsampled.
string
Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down.
Example: “This is [slow] and [fast]”, use “3, 0.5” to make “slow” slower and “fast” faster.
float
default:"1.0"
Adjusts the speed of speech. Higher than 1.0 is faster and lower than 1.0 is slower.
string
default:"bySentence"
Controls how text is segmented for synthesis. Available options:
- “immediate” - Synthesizes text immediately without waiting for complete sentences
- “never” - Never segments the text, waits for explicit flush or EOS
- “bySentence” (default) - Waits for complete sentences before synthesis
import asyncio
import json
import websockets
import base64
class RimeClient:
def __init__(self, speaker, api_key):
self.url = f"wss://users-ws.rime.ai/ws3?speaker={speaker}&modelId=mistv3&audioFormat=mp3"
self.auth_headers = {
"Authorization": f"Bearer {api_key}"
}
self.audio_data = b''
async def send_messages(self, websocket, messages):
for message in messages:
await websocket.send(json.dumps(message))
async def handle_audio(self, websocket):
while True:
try:
audio = await websocket.recv()
except websockets.exceptions.ConnectionClosedOK:
break
message = json.loads(audio)
if message['type'] == 'chunk':
self.audio_data += base64.b64decode(message['data'])
if message['type'] == 'timestamps':
print("Rime model pronounced the words...\n")
for w, t in zip(message['word_timestamps']['words'], message['word_timestamps']['start']):
print(f"'{w}' at time {t}")
async def run(self, messages):
async with websockets.connect(self.url, additional_headers=self.auth_headers) as websocket:
await asyncio.gather(
self.send_messages(websocket, messages),
self.handle_audio(websocket),
)
def save_audio(self, file_path):
with open(file_path, 'wb') as f:
f.write(self.audio_data)
print(f"\n Audio saved at {file_path}")
message = [
{"text": "This "},
{"text": "is "},
{"text": "an "},
{"text": "incomplete "},
{"text": "sentence "},
{"operation": "eos"},
]
client = RimeClient("cove", api_key="YOUR_API_KEY")
asyncio.run(client.run(message))
client.save_audio("output.mp3")