Streaming Speech-to-Text WebSocket API

Connect to wss://stt.api.rime.ai/ws from your backend, stream audio, and read the transcript as it arrives. If your backend is in Python, Node.js, Go, or Rust, use the Rime SDK instead, because it connects over gRPC and needs less code. The exception is a long recording, which needs this API’s append fragments. For other languages, read Choose how to connect first. That section helps you choose between this endpoint and gRPC, and between the two output contracts.

Item	Value
URL	`wss://stt.api.rime.ai/ws`. Unlike Rime’s text-to-speech WebSockets, this URL takes no query parameters, not even the API key.
API key	Put your API key in a header of the upgrade request: `Authorization: Bearer YOUR_API_KEY`. A browser’s `WebSocket` API cannot set this header.
Session	Each connection is one session, and each session transcribes one utterance.
Audio	Headerless signed 16-bit little-endian PCM at 16 kHz, mono

Session at a glance

Direction	Message	When
Send	`start`	One time, as the first message, within 5 seconds of the upgrade
Receive	`started`	After the server accepts `start`, before any transcript event
Send	`audio`	Many times, while you capture audio
Receive	`hypothesis`	Revised hypotheses contract only. Each one has the whole transcript so far.
Receive	`delta`	Append fragments contract only. Each one has the text of the next window of audio.
Send	`end`	One time, when the utterance is over
Receive	`done`	One time, after `end`, with the final transcript
Send	`cancel`	Instead of `end`, or after it, to stop the utterance
Receive	`cancelled`	After `cancel`
Receive	`error`	Instead of `done`, when the utterance fails

Guarantees

Topic	Guarantee
Order	`started` arrives before any `hypothesis`, `delta`, or `done`.
Revisions	Under revised hypotheses, `revision` starts at 1. It increases by 1 each time the text changes. The server does not send the same text again.
Final text	Under revised hypotheses, `done.text` equals the last `hypothesis.text`, and `done.revision` equals its revision. Under append fragments, `done.text` equals every `delta.text`, joined in arrival order, and `done.revision` is 0.
No speech	An utterance with no recognizable speech can end with no `hypothesis` or `delta`. Its `done.text` is empty, and its revision is 0.
Deltas	The server never revises a `delta`. The server trims the text of each window. Every `delta` after the first starts with `\n`.
Language	`started.accepted.language.tag` and `done.language.tag` contain the language, `en` or `es`. `source` is always `LANGUAGE_SOURCE_SELECTED`.
Request ID	Every `error` event has a `requestId`.
Terminal event	A session ends with at most one terminal event: `done`, `error`, or `cancelled`. The server tries one time, with a time limit, to send this event and then a Close frame. If the connection drops or your client stops reading, the session can end with neither. Treat a session without a terminal event as incomplete.

Connection

Response encoding

Set the response encoding in the Sec-WebSocket-Protocol header of the upgrade request:

Subprotocol	Response encoding
`rime.v1.json`	JSON text frames
`rime.v1.binary`	Protobuf binary frames
Both	The first one that you offer. The server echoes it back.
No subprotocol offered	JSON text frames

The response encoding stays the same for the whole connection. The subprotocol sets only the response encoding, because the server decodes each client message by its frame type. For example, you can send JSON text frames and receive protobuf. Both encodings carry the SpeechWebSocketRequest and SpeechWebSocketResponse envelopes, which the speech-to-text schema defines. JSON uses the canonical proto3 mapping:

Protobuf value	JSON representation
Field name	camelCase
`bytes`	Base64 string
Enum	Symbolic name
`uint64`	Decimal string
Field at its default value	Omitted

Upgrade responses

The server answers the upgrade request with one of these HTTP statuses. A refusal is a plain-text HTTP response, and it arrives before any event:

Status	Meaning
`101 Switching Protocols`	The upgrade succeeded. Its `x-request-id` header contains the request ID.
`400`	The request is not a WebSocket upgrade.
`401`	The `Authorization` header is missing, or Rime does not accept the key. The response has a `WWW-Authenticate: Bearer` header and an empty body.
`404`	The server has no route for the path, for example `/stt`. Use `/ws`.
`503`	The service is not ready, is shutting down, or has no WebSocket capacity.

Messages

Send

Start

start opens the utterance. outputContract and language are required. mode and contextTerms are optional, and they work the same way under both output contracts. Start parameters describes each field.

Rule	When you break it
Send `start` one time only, as the first message.	A second `start` fails with `invalid_input`.
Send `start` within 5 seconds of the upgrade.	The server sends an `error` with kind `timeout`, and closes the connection.

type StartMessage = {
  start: {
    outputContract:
      | "STREAMING_OUTPUT_CONTRACT_REVISED_HYPOTHESES"
      | "STREAMING_OUTPUT_CONTRACT_APPEND_FRAGMENTS",
    language: string,           // BCP 47 tag of the spoken language: "en" or "es"
    mode?: "TRANSCRIPTION_MODE_WRITTEN" | "TRANSCRIPTION_MODE_VERBATIM",
    contextTerms?: string[],
  }
}

This start message asks for English with revised hypotheses. It adds recognition hints for an airline, a fare name, and a caller’s surname:

{
  "start": {
    "outputContract": "STREAMING_OUTPUT_CONTRACT_REVISED_HYPOTHESES",
    "language": "en",
    "mode": "TRANSCRIPTION_MODE_WRITTEN",
    "contextTerms": ["Northwind Air", "FlexPlus", "Okonkwo"]
  }
}

This minimal start message selects append fragments for Spanish. It uses the default written mode, and the server confirms es in started:

{
  "start": {
    "outputContract": "STREAMING_OUTPUT_CONTRACT_APPEND_FRAGMENTS",
    "language": "es"
  }
}

Audio

Each audio message carries one chunk of audio, and it must meet these requirements:

Requirement	Value
Encoding	Base64 in the JSON `audio` field. In binary frames, put the audio in the `bytes` field of the protobuf envelope. Never send bare PCM frames.
Decoded chunk	Not empty, with an even byte count. It fits in 64 KiB.
Encoded message	Fits in 96 KiB
Order	After `start`. `audio` before `start` fails with `invalid_input`, and no `started` follows.

All of the decoded audio for the utterance must fit in 16 MiB. Within that limit, your client sets the chunk size by how many bytes it puts in each audio message. It sets the pace by when it sends each message. 100 ms of audio is 3,200 bytes, and you can send a recording faster than real time. Limits lists the error kind for each limit.

type AudioMessage = {
  audio: string, // base64
}

This JSON shows the shape of the message. The example shortens the base64 value:

{ "audio": "AAAAAAEA//8CAP7/AwD9/wQA/P8FAPv/BgD6/w..." }

End

end ends the audio and commits the utterance. After you send it, keep reading:

After `end`	What happens
The utterance succeeds	The server sends the remaining transcript events, then `done` and a Close frame.
The utterance fails, and the connection is still writable	The server sends `error` instead of `done`.
You send audio, or a second `end`	The message fails with `invalid_input`.

{ "end": {} }

Cancel

cancel stops the utterance and the work that Rime does on it. You can send it before or after end:

Case	What you receive
The connection is still writable	`cancelled`, then a Close frame
The server already produced `delta` events	Those `delta` events can arrive before `cancelled`.
You send `cancel` before `start`	`cancelled`, without `started`
You close the connection instead of sending `cancel`	No acknowledgement. The work also stops.

{ "cancel": {} }

Receive

Started

started confirms that the server accepted your start.

type StartedEvent = {
  started: {
    requestId: string,
    accepted: {
      outputContract:
        | "STREAMING_OUTPUT_CONTRACT_REVISED_HYPOTHESES"
        | "STREAMING_OUTPUT_CONTRACT_APPEND_FRAGMENTS",
      language: { tag: string, source: "LANGUAGE_SOURCE_SELECTED" },
    },
  }
}

Field	Meaning
`requestId`	The request ID. Keep it, because you need it when you report a failure.
`accepted.outputContract`	Echoes your contract
`accepted.language`	Confirms the language

The session for the flight recording used English and revised hypotheses. It started with this event:

{
  "started": {
    "requestId": "fbd8066c-c67d-4529-80a3-75c4a288132d",
    "accepted": {
      "outputContract": "STREAMING_OUTPUT_CONTRACT_REVISED_HYPOTHESES",
      "language": { "tag": "en", "source": "LANGUAGE_SOURCE_SELECTED" }
    }
  }
}

Hypothesis

Under revised hypotheses, each hypothesis is a partial transcript.

type HypothesisEvent = {
  hypothesis: {
    text?: string,    // omitted when the transcript is empty
    revision: string, // uint64 as a decimal string
  }
}

Field	Meaning
`text`	The complete transcript so far. Replace all of the previous text with it.
`revision`	The revision number of this text. JSON encodes it as a decimal string.

This is the second of the four hypotheses in that session:

{"hypothesis":{"text":"Hi! I'm calling about flight 247 to Denver.","revision":"2"}}

Delta

Under append fragments, the server transcribes each five-second window of audio one time, on its own. It sends the text of the window as a delta, and a window with no recognized text produces no delta.

type DeltaEvent = {
  delta: {
    text?: string, // omitted when empty
  }
}

Field	Meaning
`text`	The text of one window. Append it exactly as you receive it. The first delta has no separator at the start. Every later delta starts with a newline, `\n`, that separates it from the window before.

Under append fragments, the same recording produced two deltas. The second delta starts with the newline separator:

{"delta":{"text":"Hi! I'm calling about flight 247 to Denver."}}
{"delta":{"text":"\nCan I move it to Friday?"}}

The done.text of that session was the two deltas joined in order: "Hi! I'm calling about flight 247 to Denver.\nCan I move it to Friday?". Append fragments also suit a recording that is too long for a revised-hypotheses session, because the model’s context does not grow during the session. Transcribe a long recording has the steps and the limits that still apply.

Done

A successful utterance ends with one done event, then a Close frame. Under both contracts, language confirms the language.

Contract	`text`	`revision`
Revised hypotheses	Equals the last hypothesis	Equals the revision of the last hypothesis
Append fragments	Every delta, joined in arrival order	0. JSON omits it.
No recognized speech, under either contract	Empty. JSON omits it.	0. JSON omits it.

type DoneEvent = {
  done: {
    text?: string,     // omitted when empty
    revision?: string, // omitted when 0
    language: {
      tag: string,
      source: "LANGUAGE_SOURCE_SELECTED",
    },
  }
}

This is the done event for the flight recording, under revised hypotheses:

{
  "done": {
    "text": "Hi! I'm calling about flight 247 to Denver. Can I, move it to Friday?",
    "language": { "tag": "en", "source": "LANGUAGE_SOURCE_SELECTED" },
    "revision": "4"
  }
}

Error

error ends a failed utterance. If the connection is still writable, a Close frame follows.

type ErrorEvent = {
  error: {
    kind: "invalid_input" | "resource_exhausted" | "timeout" | "unavailable" | "cancelled" | "internal",
    message: string,
    requestId: string,
  }
}

Field	Meaning
`kind`	The error category. Branch on it, and retry only the kinds that Errors and limits marks as worth a retry.
`message`	A description of the error. Do not branch on it, because the text can change.
`requestId`	The request ID of the session

A start message without outputContract fails with invalid_input:

{
  "error": {
    "kind": "invalid_input",
    "message": "streaming output contract must be explicitly selected",
    "requestId": "26c49cfc-ef9a-495a-9984-3db3fa159516"
  }
}

Cancelled

cancelled confirms your cancel. If the connection is still writable, a Close frame follows.

{ "cancelled": {} }

Start parameters

string

required

Set one of these two values. There is no default, and an omitted, unspecified, or unknown value fails with invalid_input. Compare the two contracts to choose between them:

Value	Updates	Use it when
`STREAMING_OUTPUT_CONTRACT_REVISED_HYPOTHESES`	The whole transcript so far, in each `hypothesis`	Your consumer can replace text.
`STREAMING_OUTPUT_CONTRACT_APPEND_FRAGMENTS`	One `delta` for each window of audio	Your consumer can only append.

string

required

The spoken language: en or es. The service does not detect the language. An omitted tag, und, an unsupported language, or a malformed tag fails with invalid_input.

string

default:"TRANSCRIPTION_MODE_WRITTEN"

Set one of these values. The mode guides the model, but it does not guarantee exact output.

Value	Asks for	Example
`TRANSCRIPTION_MODE_WRITTEN`	Numbers as digits, and compact codes	Spoken “flight two four seven” becomes `flight 247`.
`TRANSCRIPTION_MODE_VERBATIM`	Word-for-word mode: every filler, repetition, and false start, and numbers as spoken words	”uh flight two four seven” stays as spoken.

string[]

Recognition hints, such as ["Northwind Air", "FlexPlus", "Okonkwo"]. The terms make the model more likely to recognize them, but they do not force a match. The context-term rules apply, and a value that breaks one fails with invalid_input:

Rule	Requirement
Each term	Not empty, and not only whitespace
The value	An array, not a single string
The terms, joined with `,`	Fit in 1,024 UTF-8 bytes

Limits

Limit	Value	Error kind
Time to send `start` after connecting	5 seconds	`timeout`
Time without audio	15 seconds. Silence samples count as audio. WebSocket pings do not.	`timeout`
Total time per stream, from connecting to the final transcript	120 seconds	`timeout`
Audio per utterance, decoded	16 MiB	`resource_exhausted`
Audio per message, decoded	64 KiB, non-empty, with an even byte count	`resource_exhausted` over 64 KiB. `invalid_input` if empty or odd
WebSocket message, encoded	96 KiB	`resource_exhausted`
Context terms, joined with `,`	1,024 UTF-8 bytes	`invalid_input`
Transcript per utterance	1 MiB	`resource_exhausted`
Model context	Depends on the audio, context terms, and transcript so far	`resource_exhausted`

The idle timer and model context follow some more rules.

Transcribe a long recording

Some recordings are too long for one revised-hypotheses stream. Send them through the direct API with append fragments. Rime recognizes each five-second window on its own, so the model’s context does not fill up as the recording continues. For now, only the direct API can do this, because the SDK receives only revised hypotheses.

Open a WebSocket stream, or a gRPC TranscribeStreaming call, with STREAMING_OUTPUT_CONTRACT_APPEND_FRAGMENTS.
Send the audio as fast as the connection accepts it, not at the speed of the recording. The 120-second limit is wall-clock time from connecting to done, not the length of the audio.
Keep each stream within the 16 MiB audio limit, which is about 8.7 minutes of 16 kHz mono PCM16. Send a longer file in parts, with one stream for each part. Join the done.text values of the parts in order, with a newline between them, because a newline also separates deltas.

Complete Python client

This client streams a 16 kHz mono PCM16 WAV file over the WebSocket API. It opens the file with Python’s wave module, and it sends the samples without the header, as fast as it reads them. As the hypotheses arrive, the client replaces the console line with each one. At the end, it prints the final done result. The client uses revised hypotheses, which suit a short file. For a long recording, use append fragments. Before you run the client:

Download the sample recording.
Set RIME_API_KEY in the environment of your backend.
Install websockets 14 or later with pip install "websockets>=14". The client uses the additional_headers argument of that version.

To use your own recording, convert it to a WAV file in the same format:

Install FFmpeg.
Convert the recording. This command reads input.m4a, or any other format that FFmpeg can decode. It writes sample.wav as 16 kHz mono PCM16, and it keeps the WAV header.
ffmpeg -i input.m4a -ac 1 -ar 16000 -c:a pcm_s16le sample.wav
The conversion in the quickstart writes headerless PCM instead, which wave.open() rejects.

transcribe_ws.py

import asyncio
import base64
import json
import os
import sys
import wave

import websockets

URL = "wss://stt.api.rime.ai/ws"
API_KEY = os.environ["RIME_API_KEY"]
CHUNK_BYTES = 3_200  # 100 ms of PCM16 mono at 16 kHz


async def transcribe(path):
    with wave.open(path, "rb") as source:
        if (source.getnchannels(), source.getsampwidth(), source.getframerate()) != (1, 2, 16000):
            raise ValueError("expected 16 kHz mono 16-bit PCM")
        pcm = source.readframes(source.getnframes())

    headers = {"Authorization": f"Bearer {API_KEY}"}
    async with websockets.connect(URL, additional_headers=headers) as socket:
        await socket.send(json.dumps({
            "start": {
                "outputContract": "STREAMING_OUTPUT_CONTRACT_REVISED_HYPOTHESES",
                "language": "en",
            }
        }))
        started = json.loads(await socket.recv())
        if "started" not in started:
            raise RuntimeError(started)
        print("Request:", started["started"]["requestId"])

        async def send_audio():
            for offset in range(0, len(pcm), CHUNK_BYTES):
                chunk = pcm[offset:offset + CHUNK_BYTES]
                await socket.send(json.dumps({"audio": base64.b64encode(chunk).decode()}))
            await socket.send(json.dumps({"end": {}}))

        async def receive():
            async for frame in socket:
                event = json.loads(frame)
                if "hypothesis" in event:
                    print("\r\x1b[K" + event["hypothesis"].get("text", ""), end="", flush=True)
                elif "done" in event:
                    print()
                    return event["done"]
                elif "error" in event:
                    raise RuntimeError(f'{event["error"]["kind"]}: {event["error"]["message"]}')
                elif "cancelled" in event:
                    raise RuntimeError("cancelled")
            raise RuntimeError("connection closed before done")

        _, done = await asyncio.gather(send_audio(), receive())
        return done


print(asyncio.run(transcribe(sys.argv[1])))

Run the client with the file name, for example python transcribe_ws.py flight.wav. A successful run prints the request ID, then the latest hypothesis on one line, and then the done result. A recording with no speech can return done with no text.

Overview

Coda API reference

Mist v3 API reference

Speech-to-text API reference

API Metadata

Other APIs

Streaming WebSocket

Session at a glance

Guarantees

Connection

Response encoding

Upgrade responses

Messages

Send

Start

Audio

End

Cancel

Receive

Started

Hypothesis

Delta

Done

Error

Cancelled

Start parameters

Limits

Transcribe a long recording

Complete Python client

​Session at a glance

​Guarantees

​Connection

​Response encoding

​Upgrade responses

​Messages

​Send

​Start

​Audio

​End

​Cancel

​Receive

​Started

​Hypothesis

​Delta

​Done

​Error

​Cancelled

​Start parameters

​Limits

​Transcribe a long recording

​Complete Python client

Session at a glance

Guarantees

Connection

Response encoding

Upgrade responses

Messages

Send

Start

Audio

End

Cancel

Receive

Started

Hypothesis

Delta

Done

Error

Cancelled

Start parameters

Limits

Transcribe a long recording

Complete Python client