Run Rime on your own NVIDIA host as a paired API service and TTS model service. Each model deployment needs one of each service; multiple pairs can share a machine when capacity allows.

On-prem deployment requires registry access, a Rime license, and API credentials. Contact help@rime.ai before starting. The running services also need outbound HTTPS access to Rime’s usage and license endpoints.

Architecture

The API service handles HTTP and WebSocket requests and verifies the license. The TTS service performs model inference. On-prem keeps all audio and text within your own network; nothing is sent to Rime’s cloud. This supports data-residency requirements and compliance with data privacy and protection regulations.

Published latency figures

Coda: Available on-premises. Setup details and performance numbers coming in a follow-up release.
Mist v2: Our tests have shown median latency of 175ms with randomly generated sentences between 40 and 50 characters on A10Gs and similar GPUs.
Arcana: See performance tuning.

Prerequisites

Hardware requirements

GPU
- For Mist
  - NVIDIA T4, L4, A10, or higher
- For Arcana
  - NVIDIA A100, H100 MIG 3g.40gb, or higher
Storage
- 50 GB storage
CPU
- 8 vCPUs
Memory requirements
- 32 GiB

Software requirements

Supported Linux Distributions
- Debian 12 (bookworm), x86_64
- Ubuntu Server 24.04 (noble), x86_64
NVIDIA drivers
- Minimum: 525.60.13
- Recommended: 570.133.20 or higher
Docker
NVIDIA Container Toolkit

Installations

NVIDIA drivers

Follow https://www.nvidia.com/en-us/drivers to install the latest NVIDIA drivers, or use the following instructions on Debian-based systems:

NVIDIA Driver Installation (Debian-based)

# Update packages
sudo apt-get update

# Install basic toolchain and kernel headers
sudo apt-get install -y gcc make wget linux-headers-$(uname -r)

# Download and install the NVIDIA driver.
NVIDIA_DRIVER_VERSION=580.95.05
NVIDIA_DRIVER_PATH=/opt/NVIDIA-Linux-x86_64-${NVIDIA_DRIVER_VERSION}.run
sudo rm -f "${NVIDIA_DRIVER_PATH}"
sudo wget "https://us.download.nvidia.com/tesla/${NVIDIA_DRIVER_VERSION}/NVIDIA-Linux-x86_64-${NVIDIA_DRIVER_VERSION}.run" -O "${NVIDIA_DRIVER_PATH}"
sudo chmod +x "${NVIDIA_DRIVER_PATH}"
sudo "${NVIDIA_DRIVER_PATH}" --silent --no-questions

Docker

Follow https://docs.docker.com/engine/install to install Docker on your system. Optionally, add the current user to the docker group for convenience: https://docs.docker.com/engine/install/linux-postinstall. The code snippets below assume that you can run docker as the current login.

NVIDIA Container Toolkit

Follow https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html to install the NVIDIA Container Toolkit. Note that you should follow both the Installation and the Configuration sections.

Verification

To verify that you have all the prerequisites installed, run the following command:

Verify Prerequisites

docker run --rm --gpus all nvidia/cuda:12.8.1-base-ubi9 nvidia-smi

You should see your GPU listed in the output, alongside the driver version and CUDA version.

Firewall requirements

The Rime API service listens on port 8000 for HTTP, port 8002 for binary WebSockets, and port 8003 for JSON WebSockets. You will also need to allow the following outbound traffic in your firewall rules:

https://optimize.rime.ai/usage: registers on-prem usage with our servers.
https://optimize.rime.ai/license: verifies that your on-prem license is active.
us-docker.pkg.dev on port 443: container image registry.

API and registry credentials

Generate an API key

Create a Rime API key in the dashboard. Rime supplies the Artifact Registry key and license access separately.

Deployment

The following sections select the model and API images, configure the service pair, and start it with Docker Compose. Use the Artifact Registry key provided by Rime:

cat KEY-FILE | docker login -u _json_key --password-stdin https://us-docker.pkg.dev

Container images

TTS service

Arcana

The Arcana images can be found at us-docker.pkg.dev/rime-labs/arcana/v2/<language>:<tag>.

The support languages are: en, es, fr, de, si.
The latest version is 20260420.

Arcana v3 (multilingual)

The Arcana v3 images can be found at us-docker.pkg.dev/rime-labs/arcana/v3/ennea:<tag>.

The support languages are: en, es, fr, pt, de, ja, ta, si, he.
The latest version is 20260420.

For Arcana only, you can also load the engine and data packages from different containers:

us-docker.pkg.dev/rime-labs/engine/arcana:<tag>
us-docker.pkg.dev/rime-labs/package/arcana/<language>:<tag>

Coda (multilingual)

The Coda v1 images can be found at us-docker.pkg.dev/rime-labs/coda/v1/coda:<tag>.

The support languages are: en, es, fr, pt, de, ja.
The latest version is 20260517.

Mist v3 (multilingual)

The Mist v3 images can be found at us-docker.pkg.dev/rime-labs/mist/v3/omni:<tag>

The support languages are: de, en, es, fr.
The latest version is 20260420.

API service

The latest image version is:

us-docker.pkg.dev/rime-labs/api/service:20260424

Docker Compose configuration

A simple way of deploying on a machine is to use Docker Compose. Create a compose.yml file with your editor of choice to define the services and their configurations:

compose.yml

version: '3.8'
services:
  api:
    image: us-docker.pkg.dev/rime-labs/api/service:<tag>
    depends_on:
      - model
    ports:
      - "8000:8000"
      - "8002:8002" # binary websockets api
      - "8003:8003" # json websockets api
    restart: unless-stopped
    environment:
      - MODEL_URL=http://model:8080/invocations

  model:
    image: us-docker.pkg.dev/rime-labs/<model>/<version>/<language>:<tag>
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
              count: all
    restart: unless-stopped

When running on Kubernetes, ensure that MODEL_URL points to http://0.0.0.0:8080/invocations instead of the Docker Compose service name.

Multi-model backend

If you want to serve multiple Arcana languages via a single API instance, you can create a compose.yml like the following:

compose.yml

services:
  en-api:
    image: us-docker.pkg.dev/rime-labs/api/service:<tag>
    depends_on:
      - en-model
      - es-model
    ports:
      - "8000:8000"
      - "8001:8001"
      - "8002:8002"
      - "8003:8003"
    restart: unless-stopped
    environment:
      - MODEL_URL=http://en-model:8080/invocations
      - ARCANA_ENG_MODEL_URL=http://en-model:8080/invocations
      - ARCANA_SPA_MODEL_URL=http://es-model:8080/invocations
  en-model:
    image: us-docker.pkg.dev/rime-labs/arcana/v2/en:<tag>
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
              count: all
    restart: unless-stopped

  es-model:
    image: us-docker.pkg.dev/rime-labs/arcana/v2/es:<tag>
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
              count: all
    restart: unless-stopped

Note that the ARCANA_{LANG}_MODEL_URL environment variable must point to the container running the Arcana image for that language, but you should still point MODEL_URL to a default model container. The model environment variables currently supported are:

ARCANA_ENG_MODEL_URL
ARCANA_SPA_MODEL_URL
ARCANA_FRA_MODEL_URL
ARCANA_GER_MODEL_URL

The API will route to these model backends based on the request parameter lang.

Authentication configuration

By default, callers must pass their Rime API key in every request via the Authorization: Bearer <key> header. Two additional environment variables let you configure authentication at the deployment level instead.

Pre-configuring the API key (`RIME_API_KEY`)

If you set RIME_API_KEY, the API service will use it to authenticate with the Rime license server automatically, and callers do not need to include an API key in their requests. You can supply it as an environment variable:

compose.yml

environment:
  - MODEL_URL=http://model:8080/invocations
  - RIME_API_KEY=<your-rime-api-key>

Or mount it as a secret file at /secrets/rime_api_key inside the container:

compose.yml

services:
  api:
    image: us-docker.pkg.dev/rime-labs/api/service:<tag>
    environment:
      - MODEL_URL=http://model:8080/invocations
    volumes:
      - /run/secrets/rime_api_key:/secrets/rime_api_key:ro

When neither is provided, the per-request Authorization header pathway remains active as normal.

Alternate API key header (`API_KEY_HEADER`)

On platforms that intercept the Authorization header, set API_KEY_HEADER to the name of an alternate header that callers will use to pass their Rime API key:

compose.yml

environment:
  - MODEL_URL=http://model:8080/invocations
  - API_KEY_HEADER=x-my-platform-rime-api-key

Callers then authenticate with:

curl -H "x-my-platform-rime-api-key: <your-rime-api-key>" ...

Platform API key (`PLATFORM_API_KEY`)

On platforms that require authenticated inter-container requests, set PLATFORM_API_KEY so the API service can reach the model backend. It can also be mounted as a secret at /secrets/platform_api_key:

compose.yml

environment:
  - MODEL_URL=http://model:8080/invocations
  - PLATFORM_API_KEY=<your-platform-api-key>

Start Docker Compose

docker compose up -d

Allow approximately five minutes for model warm-up before sending the first synthesis request.

Verify health

Health check

curl http://localhost:8000/health

A ready deployment returns apiStatus: "ok", licenseStatus: "valid", and modelReachable: true:

{
  "apiStatus": "ok",
  "timestamp": "2026-07-24T18:00:00.000Z",
  "licenseStatus": "valid",
  "modelReachable": true
}

AWS video walkthrough

Requests and response formats

HTTP requests

Request:

Request example

curl -X POST "http://localhost:8000" \
  -H "Authorization: Bearer <API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "text": "I would love to have a conversation with you. The new model is out.",
  "speaker": "joy",
  "modelId": "mist"
}' -o result_mist.txt

Response:

Response format

{"audioContent":{"model_output":"<base64>"}}

Sample response file: result.txt

Receiving a response in MP3 format

Request:

Request example

curl -X POST "http://localhost:8000" -H "Authorization: Bearer <API KEY>" -H "Content-Type: application/json" -H "Accept: audio/mpeg" -d '{
  "text": "I would love to have a conversation with you.",
  "speaker": "joy",
  "modelId": "mist"
}' -o result.mp3

Response: Sample response file: result.mp3

Receiving a response in PCM (raw) format

Request:

Request example

curl -X POST "http://localhost:8000" -H "Authorization: Bearer <API KEY>" -H "Content-Type: application/json" -H "Accept: audio/L16" -d '{
  "text": "I would love to have a conversation with you.",
  "speaker": "joy",
  "modelId": "mist"
}' -o result.pcm

Response: Sample response file: result.pcm

WebSocket endpoints

JSON websockets

The JSON WebSocket endpoint compatible with coda, arcana, and mist models will be served at port 8003. For example, ws://localhost:8003, which is equivalent to our cloud JSON WebSocket API. See the Coda JSON WebSocket docs, Arcana JSON WebSocket docs, and Mist JSON WebSocket docs depending on which model backend you have configured.

Non-JSON websockets

The non-JSON WebSocket endpoint will be served at port 8002. For example, ws://localhost:8002, which is equivalent to our cloud WebSocket API.

Deprecated

A deprecated WebSocket endpoint is served on port 8001. It is only compatible with the Mist model family; use the current WebSocket endpoints for new integrations.

​Architecture

​Published latency figures

​Prerequisites

​Hardware requirements

​Software requirements

​Installations

​NVIDIA drivers

​Docker

​NVIDIA Container Toolkit

​Verification

​Firewall requirements

​API and registry credentials

​Generate an API key

​Deployment

​Artifact Registry login

​Container images

​TTS service

​Arcana

​Arcana v3 (multilingual)

​Coda (multilingual)

​Mist v3 (multilingual)

​API service

​Docker Compose configuration

​Multi-model backend

​Authentication configuration

​Pre-configuring the API key (RIME_API_KEY)

​Alternate API key header (API_KEY_HEADER)

​Platform API key (PLATFORM_API_KEY)

​Start Docker Compose

​Verify health

​AWS video walkthrough

​Requests and response formats

​HTTP requests

​Receiving a response in MP3 format

​Receiving a response in PCM (raw) format

​WebSocket endpoints

​JSON websockets

​Non-JSON websockets

​Deprecated

Architecture

Published latency figures

Prerequisites

Hardware requirements

Software requirements

Installations

NVIDIA drivers

Docker

NVIDIA Container Toolkit

Verification

Firewall requirements

API and registry credentials

Generate an API key

Deployment

Artifact Registry login

Container images

TTS service

Arcana

Arcana v3 (multilingual)

Coda (multilingual)

Mist v3 (multilingual)

API service

Docker Compose configuration

Multi-model backend

Authentication configuration

Pre-configuring the API key (`RIME_API_KEY`)

Alternate API key header (`API_KEY_HEADER`)

Platform API key (`PLATFORM_API_KEY`)

Start Docker Compose

Verify health

AWS video walkthrough

Requests and response formats

HTTP requests

Receiving a response in MP3 format

Receiving a response in PCM (raw) format

WebSocket endpoints

JSON websockets

Non-JSON websockets

Deprecated