> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rime.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Load balancing

> Load-balance Rime on-prem deployments using the ORCA cost-header signal.

Real-time streaming often requires limiting how many inference requests a model
processes concurrently.

Rime's Coda and Mist model images return an [Open Request Cost
Aggregation (ORCA) header](https://github.com/envoyproxy/envoy/issues/6614) on
every engine HTTP response, reporting concurrent request load to your load
balancer. Set `ENDPOINT_LOAD_METRICS_ENABLE=false` on the model container to
suppress it.

## HTTP ORCA header

The ORCA header in HTTP responses looks like:

```
endpoint-load-metrics: TEXT application_utilization=0.5, cpu_utilization=0.3128, mem_utilization=0.2453, rps_fractional=0.0000, eps=0.0000
```

## Max concurrency

The `application_utilization` metric is calculated by dividing the number of
concurrent inference requests by a preconfigured max concurrency.

You can override the max concurrency after [parameter
tuning](/docs/on-prem/performance) by setting the
`INFERENCE_CONCURRENCY_CAPACITY` to the desired max concurrency.

<Note>On Mist images, `INFERENCE_CONCURRENCY_CAPACITY` only scales
the utilization figure reported to the load balancer; it does not reject or
queue requests. On Coda images it additionally sets the generator's admission
limit and sizes its KV cache, so lowering it queues excess requests and
raising it costs GPU memory.</Note>