Adaptive Load Balancing

Note

For provider routing in general - how adaptive load balancing combines with governance routing, provider and key selection, and Model Catalog integration - see the Provider Routing Guide.

This page focuses on what you turn on and what you watch in the dashboard.

What it does

Adaptive Load Balancing automatically shifts traffic toward the providers and API keys that are performing best right now, and away from those that are erroring or slow - without you adjusting weights by hand. It reacts to live error rates and latency, removes failing keys from rotation, and brings them back once they recover, so your traffic keeps flowing through the healthiest routes.

Feature	What you get
Dynamic weight adjustment	Traffic re-balances automatically as provider/key performance changes - no manual tuning
Automatic failover & recovery	Poorly performing keys are pulled from rotation and restored once they recover
Fast recovery	A route that starts succeeding again quickly regains its share of traffic
Live dashboard	Per-key state, active and total requests, tokens, and errors, refreshed every 10 seconds

Tip

Weight recomputation happens in the background and the request path reads a precomputed snapshot. Snapshot lookup, routing, and instrumentation still have bounded, measurable overhead, and a request can briefly observe the most recent published snapshot rather than an update still being computed. Benchmark route selection at your rule count and concurrency.

---

What drives the routing

You don’t tune any of these - they are observed automatically from your live traffic. Knowing what the system reacts to helps you read the dashboard:

Signal	Effect on your traffic
Error rate	The biggest factor - routes returning more errors receive proportionally less traffic.
Latency	Routes responding abnormally slowly are de-prioritized (large requests aren’t unfairly penalized).
Utilization	Prevents any single high-performing route from being overloaded.
Recovery momentum	A route that has started succeeding again is rewarded so it regains traffic quickly.

To get more traffic onto a specific route, fix its underlying error rate or latency - the system will shift traffic back automatically once it sees the route succeeding again.

---

Monitoring routing in the dashboard

The dashboard is where you observe and verify adaptive load balancing in action. Use it to:

• See each virtual key’s providers, key count, and strategy.
• Watch each key’s active requests, total requests, tokens, and errors to spot a degrading provider before it causes failures.
• Track each key’s state (Healthy, Recovering, Circuit Open) and the time of its last error.
• Open Analytics → Overview → Load Balancer to compare requests, cost, and latency by engine and requests per key.

A key that flips to Circuit Open (for example, after a provider rate limit) is pulled from rotation automatically and returns to Healthy once it sustains low error rates - no action required from you.

---

Next Steps

• Enable Adaptive Load Balancing - Available on the Team plan and above. Open Access & Credentials → Load Balancer in the workspace sidebar, select a virtual key that has provider configurations to open Configure Load Balancing, choose a Strategy for each provider and save, and watch your keys under Key Health.
• Monitor your keys - Use the dashboard to watch key health, error counts, and state changes.
• Provider Routing Guide - See how adaptive load balancing combines with governance routing and the Model Catalog.