A request becomes an ordered list of candidates: each model in models (or the single model), and for each model the endpoints able to serve it, ordered by your provider preferences, their measured health and their price. The first candidate that answers serves the request. Only the first three models of a chain are considered, and at most three attempts are made in all, across every endpoint of those models.
Declare a fallback chain
Pass models instead of model: if the first cannot be served, the next is tried. Even a single model fails over, between the providers that serve it.
{
"models": ["anthropic/claude-sonnet-5", "openai/gpt-5.5", "google/gemini-3.1-pro"],
"messages": [{"role": "user", "content": "Hello"}]
}From the OpenAI SDK, which always sends model, add models alongside it (in Python through extra_body); when models is present it takes the place of model.
client.chat.completions.create(
model="anthropic/claude-sonnet-5",
messages=[{"role": "user", "content": "Hello"}],
extra_body={"models": ["anthropic/claude-sonnet-5", "openai/gpt-5.5"]},
)Fallbacks for every request in the account
PUT /v1/routing (or Workspace → Routing) sets a default_model, used when a request names none, and up to eight fallback_models, added after the models a request names. The chain they make is cut to three models like any other.
How candidates are ordered
Within each model, endpoints are ranked by measured health and price, as a weighted random draw, so traffic spreads across the healthy ones rather than piling onto one. A provider.sort of price, latency or throughput replaces the draw with a fixed order, and provider preferences can require, prefer or exclude providers — see Routing.
What moves on to the next candidate
- A provider error, 5xx, or a 429 rate limit from the provider.
- A provider that does not answer in time. Each phase has its own limit: 30 seconds for a streamed response’s first byte, 60 seconds of silence between streamed chunks, and 180 seconds for a whole response that is not streamed. A timeout is reported as
524. - An endpoint that cannot carry the request, such as a content type that provider does not accept.
What does not
- A request error every endpoint would refuse, such as malformed messages or an invalid parameter. It returns 400 at once, and no other endpoint is tried.
- A failure after the first byte of a response has been sent. Text already delivered cannot be taken back, so the request is not restarted on another provider.
- A refusal by your own limits or guardrails, which is final.
- Your client disconnecting. The request stops, and nothing is retried on your behalf.
Spend limits and guardrails
Key spend ceilings, guardrail budgets, workspace budgets and agent budgets are all checked once, before the first attempt, and guardrails decide which providers may be candidates at all. Failover never gets around either: a request over a limit is refused with 403 budget_exceeded before any provider is contacted, and a provider a guardrail excludes is never tried as a fallback.
Every attempt runs on your own provider key. Autonomous Relay charges nothing for an attempt, served or failed; whether a provider bills one that failed on its side is between you and that provider.
Health
Every attempt feeds the endpoint’s measured health, and health ranks the candidates, so an endpoint that is failing now is tried later. Health is measured over a rolling 30 minutes and published once an endpoint has served 100 requests; see Status. A disconnect by your client is not counted against the provider.
Preview the order before you send
Takes the same body as a chat completion and runs the same gates, filters and ranking, without contacting a provider or holding anything: it answers which candidates would be tried, in what order, and whether the request would be refused. ordering says whether the order is fixed or sampled — a sampled order is drawn again for a real request.
Seeing what happened
One row per attempt made for a request, in order, with each provider’s outcome. A request that failed over twice shows three rows.