> ## Documentation Index > Fetch the complete documentation index at: https://docs.rime.ai/llms.txt > Use this file to discover all available pages before exploring further. # Stream live audio > Stream live browser or phone audio to Rime speech-to-text, and send each final transcript to your language model. The guide also covers long turns, hang-ups, and common mistakes. Rime does not detect when someone stops speaking. Your voice activity detector, turn detector, or push-to-talk control ends each utterance. The final transcript arrives after that. The code on this page connects your audio to the same stream call that the [quickstart](/docs/stt/quickstart) used on a file. The code is not a complete program. Your application supplies the audio capture, any decoding to PCM16, and the turn detection that ends each utterance. > **[Hear it working first](https://github.com/rimelabs/rime-sdk/blob/main/docs/terminal-voice.md)** > > The SDK's terminal voice loop records from your microphone and prints partial and final transcripts. It speaks each final transcript back through Rime text-to-speech. It needs a Rime API key and SoX. ## How a stream works ```mermaid theme={null} sequenceDiagram autonumber participant Client as Browser or phone participant Backend as Your backend participant Rime as Rime speech-to-text Note over Backend: Your voice activity detector reports speech. Backend->>Rime: Open a stream Note over Rime: The 120-second clock starts. Client->>Backend: Audio frames Backend->>Rime: Audio frames, also silence Note over Backend,Rime: Audio streams the whole time, also during pauses. Rime->>Backend: Partial transcripts Backend->>Client: Show each partial Note over Backend: Your turn detector ends the utterance. Backend->>Rime: End of the audio Rime->>Backend: Final transcript Note over Backend: Send the final transcript to your LLM. Note over Backend,Rime: The stream closes. The next utterance gets a new stream. ``` Your code must meet two deadlines on each stream: * A stream fails after 15 seconds without audio. Send audio through pauses, and include silence. * A stream must finish within 120 seconds of connecting. The SDK meets a third deadline for you, because it sends the stream settings as soon as it connects. A direct WebSocket or gRPC client must send them within 5 seconds. The [limits](/docs/stt/errors#limits) table lists every limit. ## Where audio flows ```text theme={null} Browser or phone ──audio──▶ Your backend ──audio──▶ Rime STT ◀──text─── ◀──text─── ``` Browsers and phone systems send their audio to your backend. Your backend holds the Rime API key and opens every stream. A browser cannot call Rime directly, because the browser `WebSocket` API cannot set the `Authorization` header that carries the key. Decode the audio to headerless PCM16 before you pass it to the SDK. PCM16 is signed 16-bit little-endian samples, with stereo channels interleaved. The SDK does not decode other formats. | Source | Usual format | What to do | | - | - | - | | Browser recorder | Compressed audio | Decode it to PCM16. | | Phone call, such as a Twilio media stream | 8 kHz mu-law | Decode the mu-law samples to PCM16, and keep the 8 kHz rate. Declare that rate as the [input format](#send-audio-as-you-capture-it). The SDK resamples the audio to 16 kHz. If callers read out digits, use [word-for-word mode](/docs/stt/recognition#word-for-word-mode). | ## Send audio as you capture it Give each utterance its own queue. Your capture code puts audio frames in the queue as they arrive, and the SDK takes the frames out at the other end. Capture never waits for the network, because audio that you capture while the stream connects stays in the queue until Rime accepts the stream. If your capture path drops silent frames, put frames of zeros in the queue in their place. This function transcribes one utterance from a queue. It returns the final transcript and the request ID. In each tab, the comments in the second code block show where to connect your capture and turn detection. **Python** ```python stream_utterance.py theme={null} from rimelabs_sdk import PCMFormat END_OF_UTTERANCE = None async def from_queue(queue): """Yield captured audio until your turn detector puts END_OF_UTTERANCE.""" while (chunk := await queue.get()) is not END_OF_UTTERANCE: yield chunk async def transcribe_utterance(client, queue, show_partial): """Transcribe one utterance. Return its final text and request ID.""" final_text = None async with client.stt.stream( from_queue(queue), language="en", input_format=PCMFormat(sample_rate=48000, channels=1), # match your capture timeout=120, ) as transcript: async for update in transcript: if update.kind == "partial": show_partial(update.text) # replace the text on screen else: final_text = update.text if final_text is None: raise RuntimeError(f"No final transcript (request {transcript.request_id})") return final_text, transcript.request_id ``` Give each utterance a new queue, and connect the queue to your capture code and turn detector. Audio callbacks often run on another thread. The `call_soon_threadsafe` method safely passes their frames to the event loop from any thread: ```python theme={null} import asyncio from rimelabs_sdk import Rime from stream_utterance import END_OF_UTTERANCE, transcribe_utterance async def run_utterance(client: Rime): loop = asyncio.get_running_loop() queue = asyncio.Queue() def on_audio_frame(frame: bytes): # call from your capture code loop.call_soon_threadsafe(queue.put_nowait, frame) def on_speech_end(): # call from your turn detector loop.call_soon_threadsafe(queue.put_nowait, END_OF_UTTERANCE) # Register on_audio_frame and on_speech_end with your capture and turn detection here. return await transcribe_utterance(client, queue, show_partial=print) ``` **Node.js** ```javascript stream-utterance.mjs theme={null} // Transcribe one utterance. Return its final text and request ID. export async function transcribeUtterance(client, audio, showPartial) { const transcript = client.stt.stream(audio, { language: "en", inputFormat: { sampleRate: 48000, channels: 1 }, // match your capture timeout: 120, }); let finalText = null; for await (const update of transcript) { if (update.kind === "partial") showPartial(update.text); // replace the text on screen else finalText = update.text; } if (finalText === null) { throw new Error(`No final transcript (request ${transcript.requestId})`); } return { text: finalText, requestId: transcript.requestId }; } ``` A Node.js `PassThrough` stream can be the queue. Your capture code writes frames into it. The SDK reads the frames out: ```javascript theme={null} import { PassThrough } from "node:stream"; import { Rime } from "@rimelabs/sdk"; import { transcribeUtterance } from "./stream-utterance.mjs"; const client = new Rime(); const audio = new PassThrough(); // one per utterance // In your capture code, for each PCM16 frame: audio.write(frame) // When your turn detector decides the speaker finished: audio.end() const { text, requestId } = await transcribeUtterance(client, audio, console.log); ``` Keep these points in mind when you adapt the example: * **Input format:** declare the format that your capture produces. The example uses 48 kHz mono, a common rate for browser and WebRTC audio. For telephone audio, use 8000. The SDK converts 8, 16, 24, or 48 kHz audio, mono or stereo, to the 16 kHz mono that the service reads. Because the SDK converts from the rate that you declare, a wrong rate distorts the audio, and the SDK does not raise an error. * **Timeout:** a `timeout` of 120 stops the call on the client side at the service limit. Python and Node.js both take the value in seconds. The timeout clock includes the time that the SDK waits for your audio source. * **Update loop:** do not do slow work in the update loop, such as waiting for your language model. A slow loop also slows the stream, through backpressure. ## Open each stream when speech starts Start the stream for each utterance when your voice activity detector reports speech. Do not start it earlier. A stream that opens early costs more than it saves: * It holds service capacity from the time Rime accepts it until it ends, even when nobody speaks. * Its 120-second clock starts when it connects. * Silence before speech counts toward the stream's limits. For each partial transcript, the model recognizes all of the stream's audio again. Leading silence uses [model context](/docs/stt/errors#limits) that the speech needs. You do not lose the first words when you wait for speech. Each SDK client keeps one gRPC connection and uses it again for every stream. A new stream on that connection takes one round trip, for Rime to accept the configuration. During that round trip, the queue for the utterance holds the audio that you capture. These two steps keep the first words: * **Create one client per process.** Pass it to every utterance, as the examples do. A new client for each utterance must open its own connection first. * **Keep a short pre-roll.** A voice activity detector reports speech a short time after the speech starts. By then, the first syllable is past. Keep the most recent audio in a short rolling buffer. Put its contents in the new queue before any other frame. ## Show partials and act on the final Each partial transcript is the full transcript so far, so replace the text on screen with each partial. If you append partials, words repeat. The service also does not send an update for each audio chunk. For example, the [quickstart's run](/docs/stt/quickstart) got three partials in a 9-second call, then the final transcript. Send only the final transcript to your language model. Partials can still change, and only the final transcript covers all of the audio. Even an empty final transcript is a successful result. It means that the audio had no recognizable speech, and your agent decides whether to respond. ## End the utterance End the audio source when your turn detector decides that the speaker has finished: | SDK | End the audio source | | - | - | | Python | Put `END_OF_UTTERANCE` on the queue. | | Node.js | Call `audio.end()`. | Ending the source does not cancel the stream. The SDK sends the remaining audio, and Rime recognizes that audio and sends the final transcript. Read updates until the final transcript arrives, and then start a new stream for the next utterance. ## Split long turns at pauses For each partial transcript, the model reads all of the stream's audio again. So a long turn runs out of [model context](/docs/stt/errors#limits) before the stream's 120 seconds end. The turn then fails with a resource-limit error and no final transcript. When one speaker talks for a long time, split the turn into several utterances: 1. Find the first pause after about a minute of speech. Split the turn at that pause. Do this even when the pause is too short to end the turn for your turn detector. 2. To split, start the next stream on a new queue. Switch your capture code to the new queue, and then end the old utterance. This order keeps every frame of audio in one of the two streams. 3. While a later part streams, show the earlier final transcripts, followed by the current partial. When the turn ends, join the final transcripts in order, with a space between them. Send the joined text to your language model. A split at a pause keeps each word whole. A split after about a minute leaves time to wait for a pause before the context runs out. ## Cancel when the caller hangs up Cancel the stream when the caller hangs up during a sentence, or when your agent no longer needs the utterance. Also stop capturing audio for it. A cancelled stream never produces a final transcript. > **How to cancel in each SDK** > > | SDK | Cancel one stream | > | - | - | > | Python | Leave the `async with` block early, call `await transcript.cancel()`, or cancel the task that reads it | > | Node.js | Leave the `for await` loop, call `await transcript.cancel()`, or abort the `AbortSignal` that you passed as the `signal` option | > | Go | Cancel the stream's `context`, or call `transcript.Close()` | > | Rust | Drop the stream, call `transcript.cancel()`, or call `transcript.close().await` to wait for cleanup | > > Closing the client cancels every stream that it owns. Cancelling one stream does not stop the others. Each SDK documents its rules for releasing your audio source: [Python](https://github.com/rimelabs/rime-sdk/blob/main/python/docs/stt.md#lifetime-and-limits), [Node.js](https://github.com/rimelabs/rime-sdk/blob/main/typescript/docs/stt.md#lifetime-and-limits), [Go](https://github.com/rimelabs/rime-sdk/blob/main/go/README.md#stt), and [Rust](https://github.com/rimelabs/rime-sdk/blob/main/rust/README.md#stt). ## Common mistakes Most of these mistakes do not raise an error. Each row links to the section that covers it: | Mistake | What to do | | - | - | | [Appending partials](#show-partials-and-act-on-the-final) | Replace the previous partial. | | [Sending a file header or compressed audio](#where-audio-flows) | Send headerless PCM16. | | [Declaring the wrong input format](#send-audio-as-you-capture-it) | Match it to what your capture produces. | | [Selecting the wrong language](/docs/stt/recognition#language) | Select the speaker's language. | | [Treating a closed or failed stream as final](/docs/stt/errors) | Act only on a final update. | | [Reusing a stream for the next utterance](#end-the-utterance) | Open a new stream for each utterance. | ## Call the API directly To call the WebSocket or gRPC API without the SDK, start at [Choose how to connect](/docs/speech-to-text#choose-how-to-connect), which compares the transports and the output contracts. The [WebSocket](/api-reference/stt/websockets) and [gRPC](/api-reference/stt/grpc) references give each message and audio rule, and the WebSocket reference shows how to [transcribe a long recording](/api-reference/stt/websockets#transcribe-a-long-recording). ## Next > **[Recognition settings](/docs/stt/recognition)** > > Switch to word-for-word text, or add the names that your callers say. > **[Speak the reply](/docs/voice-agents)** > > Choose a starter that streams your agent's reply through Rime text-to-speech. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.