Skip to content

openrouter.md

Latest commit

 

History

History
188 lines (150 loc) · 8.27 KB

openrouter.md

File metadata and controls

188 lines (150 loc) · 8.27 KB
Raw

Routing through OpenRouter

OpenRouter is one of the providers HybridInference can route to. Configure its credentials and routes in Admin. This page covers the OpenRouter adapter specifically. For how routing works in general, see Architecture; for the model-registry syntax, see Adding a New Model.

Configure OpenRouter

Add an OpenRouter credential under Admin → Providers, then create a model and verify its OpenRouter route under Admin → Routing. A fresh gateway has no configured provider or model.

For catalog metadata and aliases, config/examples/models.openrouter.json provides optional JSON import input for the MODEL_CATALOG setting. Add the referenced keys as secret settings before importing it and follow the Restart required marker.

curl -s --noproxy '*' http://127.0.0.1:8080/routing

Verify the model identifiers and prices for your account before importing the example. The gateway uses catalog prices for its own usage accounting.

Route syntax

An OpenRouter route is an entry in a model's route: list. There are two forms of kind:.

kind: openrouter

OpenRouter picks the upstream provider itself, under its own default policy.

[
  {
    "kind": "openrouter",
    "weight": 1.0,
    "base_url": "https://openrouter.ai/api/v1",
    "api_keys": [
      "${OPENROUTER_API_KEY}"
    ],
    "provider_model_id": "meta-llama/llama-3.3-70b-instruct"
  }
]

provider_model_id is OpenRouter's own slug for the model. The id: your clients ask for is the gateway's; the two are independent on purpose.

kind: openrouter[<slug>]

Pins every request on that leg to one OpenRouter upstream, by sending provider: {order: [<slug>], allow_fallbacks: false}.

[
  {
    "kind": "openrouter[deepinfra]",
    "weight": 1.0,
    "base_url": "https://openrouter.ai/api/v1",
    "api_keys": [
      "${OPENROUTER_API_KEY}"
    ],
    "provider_model_id": "meta-llama/llama-3.3-70b-instruct"
  }
]

The slug must match [A-Za-z0-9_.-]+, optionally with /-separated segments (deepinfra, deepinfra/turbo). A malformed bracket form — empty pin, whitespace, nested or unmatched brackets — raises at registration rather than being silently ignored. OpenRouter's list of provider slugs is at https://openrouter.ai/docs/features/provider-routing.

Two legs that share a base_url but pin different upstreams still get distinct endpoint_ids, because the bracketed kind survives into the identifier. Their circuit-breaker and availability state therefore stay isolated: one flaky upstream does not take the other out. In api_logs, though, both forms record provider = "openrouter", so analytics sees a single OpenRouter cohort.

Sort policy

An OpenRouter route created through the admin provider-routes API may carry an openrouter_sort policy of price, throughput, or latency, which is sent as provider: {sort: <policy>}. It applies only to unpinned routes — a route with a bracket-form pin sends order instead, and the pin wins. The API rejects the field with 422 on a non-OpenRouter route or an unrecognised value.

What the adapter sends

OpenRouterAdapter (apps/backend/serving/adapters/openrouter.py) is a thin subclass of the generic OpenAI-compatible adapter. On top of the normal request it adds:

  • Attribution headers. OpenRouter credits traffic — for its leaderboard and its free-tier limits — to the site these two headers name. X-Title carries your site's name and HTTP-Referer its public address, taken from SITE_NAME and SITE_PUBLIC_BASE_URL or from SITE_CONFIGURATION. Without a public address no HTTP-Referer is sent, and without a site name X-Title is HybridInference. Set both if you want your own site credited.
  • usage: {include: true} on every request, so OpenRouter returns its per-request cost field.
  • stream_options: {include_usage: true} on streaming requests, so the final SSE chunk carries the usage block.
  • provider: {...} for the pinned and sort cases described above.

Cost accounting

Two numbers, deliberately kept apart:

Column in api_logs Meaning
cost_usd What the caller is charged: tokens × the pricing declared for the model in your registry. Unaffected by OpenRouter
upstream_cost_usd What OpenRouter reported it charged you for that request

upstream_cost_usd is NULL for non-OpenRouter routes, and NULL for an OpenRouter route when the response carried no cost figure.

API keys

A route's api_keys: is a list, and the adapter treats it as a pool even when it holds a single key. With several keys the pool rotates: a key that hits a key-specific or transient failure (429, 401/402/403, 408/425, 5xx, or a timeout) hands the request to the next key. The failing key is muted only once the request has nowhere left to rotate to, and then for 20 seconds. Request-scoped failures such as 400 and 422 fail identically on every key, so they propagate immediately instead of burning the pool.

An api_keys: entry that expands from an unset environment variable is treated as absent. A route marked optional: true is then skipped with a warning naming the model and kind; any other route raises MissingEnvBackedKeyError and startup fails. Either way a route is never registered pointing at an endpoint it cannot authenticate to.

Error handling

OpenRouter errors travel the same path as any other OpenAI-compatible provider's. Two mechanisms handle them, and they are independent:

Fallback. When a request to an OpenRouter leg fails for any reason, the router records the failure and tries the model's remaining legs in route order, skipping any that are admin-disabled, modality-incompatible, or circuit-open. The exception is a request the caller explicitly pinned with X-Route-Pin, which never falls back. If every leg fails, the first error is what the client sees. Failure status codes do not select between "fall back" and "propagate" — that decision is only about whether another leg is available.

The circuit breaker. Failures accumulate per endpoint_id; after CIRCUIT_FAILURE_THRESHOLD consecutive failures (default 3) the endpoint stops receiving traffic for CIRCUIT_COOLDOWN_SECONDS (default 30) before a half-open probe. Client errors are exempt: a 4xx other than 408, 429, 401, and 407 does not open the circuit, because letting one caller's bad request take the endpoint away from everyone else is the cascade the exemption exists to prevent. 408 and 429 mean OpenRouter is overloaded and do count. 401 and 407 mean the gateway's configured credential was rejected — 401 the OPENROUTER_API_KEY, 407 an egress proxy demanding its own — which no caller can work around, so they count and additionally page.

402 and 403 do not count against the circuit by themselves. On a route whose keys are listed under api_keys:, though, the gateway treats them as a problem with the key, as described under API keys: it moves on to the next key and, once none is left, sets the key aside for a while. An account with no credit, which answers 402 to everything, therefore soon has no usable key, and that does open the circuit. A route with a single api_key: has no pool, so there a 402 never opens it. Unlike 401 and 407, 403 does not page, because providers also use it to refuse single requests — content policy, region blocks — while the service itself is healthy.

Troubleshooting

Every OpenRouter request returns 401. The gateway's key was rejected, not the caller's. Check the credential in Admin Providers or the referenced secret setting. A key that is unset entirely never gets that far — the route is skipped or startup fails, as above — so a 401 means a key was present and OpenRouter refused it.

The model is missing from /v1/models. Check its required role, route verification and applied status in Admin Routing. For a catalog model, check MODEL_CATALOG, its referenced secret settings and its Restart required marker; see Configuration.

Requests reach the wrong upstream. Use GET /routing to see the effective weight distribution per model. It answers anyone, but shows each route's upstream base URL only with an administrator's JWT or ADMIN_TOKEN; see Architecture.