Stream live audio

​

How a stream works

• A stream fails after 15 seconds without audio. Send audio through pauses, and include silence.
• A stream must finish within 120 seconds of connecting.

​

Where audio flows

Browser or phone ──audio──▶ Your backend ──audio──▶ Rime STT
                 ◀──text───              ◀──text───

​

Send audio as you capture it

from rimelabs_sdk import PCMFormat

END_OF_UTTERANCE = None


async def from_queue(queue):
    """Yield captured audio until your turn detector puts END_OF_UTTERANCE."""
    while (chunk := await queue.get()) is not END_OF_UTTERANCE:
        yield chunk


async def transcribe_utterance(client, queue, show_partial):
    """Transcribe one utterance. Return its final text and request ID."""
    final_text = None
    async with client.stt.stream(
        from_queue(queue),
        language="en",
        input_format=PCMFormat(sample_rate=48000, channels=1),  # match your capture
        timeout=120,
    ) as transcript:
        async for update in transcript:
            if update.kind == "partial":
                show_partial(update.text)  # replace the text on screen
            else:
                final_text = update.text
    if final_text is None:
        raise RuntimeError(f"No final transcript (request {transcript.request_id})")
    return final_text, transcript.request_id

import asyncio

from rimelabs_sdk import Rime
from stream_utterance import END_OF_UTTERANCE, transcribe_utterance


async def run_utterance(client: Rime):
    loop = asyncio.get_running_loop()
    queue = asyncio.Queue()

    def on_audio_frame(frame: bytes):  # call from your capture code
        loop.call_soon_threadsafe(queue.put_nowait, frame)

    def on_speech_end():  # call from your turn detector
        loop.call_soon_threadsafe(queue.put_nowait, END_OF_UTTERANCE)

    # Register on_audio_frame and on_speech_end with your capture and turn detection here.
    return await transcribe_utterance(client, queue, show_partial=print)

// Transcribe one utterance. Return its final text and request ID.
export async function transcribeUtterance(client, audio, showPartial) {
  const transcript = client.stt.stream(audio, {
    language: "en",
    inputFormat: { sampleRate: 48000, channels: 1 }, // match your capture
    timeout: 120,
  });
  let finalText = null;
  for await (const update of transcript) {
    if (update.kind === "partial") showPartial(update.text); // replace the text on screen
    else finalText = update.text;
  }
  if (finalText === null) {
    throw new Error(`No final transcript (request ${transcript.requestId})`);
  }
  return { text: finalText, requestId: transcript.requestId };
}

import { PassThrough } from "node:stream";
import { Rime } from "@rimelabs/sdk";
import { transcribeUtterance } from "./stream-utterance.mjs";

const client = new Rime();
const audio = new PassThrough(); // one per utterance

// In your capture code, for each PCM16 frame: audio.write(frame)
// When your turn detector decides the speaker finished: audio.end()

const { text, requestId } = await transcribeUtterance(client, audio, console.log);

• Input format: declare the format that your capture produces. The example uses 48 kHz mono, a common rate for browser and WebRTC audio. For telephone audio, use 8000. The SDK converts 8, 16, 24, or 48 kHz audio, mono or stereo, to the 16 kHz mono that the service reads. Because the SDK converts from the rate that you declare, a wrong rate distorts the audio, and the SDK does not raise an error.
• Timeout: a timeout of 120 stops the call on the client side at the service limit. Python and Node.js both take the value in seconds. The timeout clock includes the time that the SDK waits for your audio source.
• Update loop: do not do slow work in the update loop, such as waiting for your language model. A slow loop also slows the stream, through backpressure.

​

Open each stream when speech starts

• It holds service capacity from the time Rime accepts it until it ends, even when nobody speaks.
• Its 120-second clock starts when it connects.
• Silence before speech counts toward the stream’s limits. For each partial transcript, the model recognizes all of the stream’s audio again. Leading silence uses model context that the speech needs.

• Create one client per process. Pass it to every utterance, as the examples do. A new client for each utterance must open its own connection first.
• Keep a short pre-roll. A voice activity detector reports speech a short time after the speech starts. By then, the first syllable is past. Keep the most recent audio in a short rolling buffer. Put its contents in the new queue before any other frame.

​

Show partials and act on the final

​

End the utterance

​

Split long turns at pauses

1. Find the first pause after about a minute of speech. Split the turn at that pause. Do this even when the pause is too short to end the turn for your turn detector.
2. To split, start the next stream on a new queue. Switch your capture code to the new queue, and then end the old utterance. This order keeps every frame of audio in one of the two streams.
3. While a later part streams, show the earlier final transcripts, followed by the current partial. When the turn ends, join the final transcripts in order, with a space between them. Send the joined text to your language model.

​

Cancel when the caller hangs up

​

Common mistakes

​

Call the API directly

​

Next