Model Routing & Fallback
One endpoint, several models
Applications that call a provider SDK directly end up with the same five concerns copy-pasted into every service: which key to use, how to retry, what the timeout is, how spend is attributed, and what to do when the provider returns a 529. An AI gateway is the tier that owns those concerns once.
Concretely it holds provider credentials so application code never sees them, applies per-tenant rate limits and spend caps, records prompt and completion metadata for later evaluation, and chooses which model receives the request. That last job is routing, and it is what turns a model bill from a fixed cost into a decision.
The gateway also gives you a place to stand when a provider changes. Model names get deprecated, prices change, a new tier appears that is better on your workload. If every service names a model directly, that is a fleet-wide migration; if the gateway names it, it is a config change with a canary.
┌── auth, keys, quota, spend caps
│
client ──▶ AI gateway ──┬──▶ fast tier (most requests)
│ │
│ └──▶ deep tier (the ones that need it)
│
└── logging, evals, cost attributionThe gateway is on the request path of every model call you make, which makes it both the best place to put policy and a single point of failure worth designing carefully.