The OmniRoute backstory

I've rebuilt the way my agents talk to language models three times in about three months. Each rewrite solved the previous one's biggest pain and then exposed the next one. The version I run today routes everything through a multi-provider gateway called OmniRoute — but that choice only makes sense if you know the two experiments that came before it, because the gateway isn't the interesting part. The shape is. Let me walk you through the lineage, because you're probably about to build some slice of this yourself, and I can save you a rewrite or two.

One thing up front so I don't mislead you: OmniRoute is not something I wrote. It's an off-the-shelf gateway I adopted at the end of this story. What I built — and rebuilt — was everything around it. That distinction turns out to be the whole lesson.

Stage one: the relay that could reach anything and decide nothing

The first version had one goal: make many providers look like one. I wanted a single endpoint my code could call and stop caring which vendor was behind it. The specific itch was unlocking a coding-assistant subscription I already paid for as a backend provider, which meant a device-flow OAuth handshake and a token sidecar to keep it alive.

So I built a relay. A thin service with a provider-adapter contract — a uniform interface, so a new provider was just another adapter you plugged in. It worked. I could hit a dozen providers through one door.

And then I stared at it and realized it was only half of a system. The relay knew how to reach every provider. It had no opinion whatsoever about which one to use, or when, or why. Every routing decision still lived in my application code, hard-coded and duplicated. A relay is transport. Transport doesn't make choices. That's not a bug you fix in the relay — it's a missing layer.

Stage two: the router that made the choices

So I built the missing layer: a reasoning router that sat above the relay and decided things. Which local model handles this? When do I escalate from a small local model to a bigger one? When is it worth reaching for an external paid model, and which one? Those are policy questions — cost, latency, quality — and the router answered them on every request, instrumented so I could see what it chose and what it cost.

The critical design call, the one that survived everything after it: the router did not replace the relay. It sat on top. I now had two clean layers with a clear seam between them — a policy layer that answered "which model and why," and a transport layer that answered "how to reach it." Keep policy out of transport. Write that on a sticky note. It's the load-bearing idea in this entire post, and the reason the next rewrite didn't hurt.

Think of it like a dispatcher and a road network. The dispatcher decides which truck goes where based on cost and deadline; the roads just get you there. You don't teach the roads to make dispatch decisions, and you don't make the dispatcher repave anything.

Stage three: stop hand-rolling the roads

Here's where I made the pragmatic call. My hand-rolled transport layer was fine at a dozen providers. It was not going to be fine at hundreds — plus semantic caching, quota tracking across every vendor, prompt compression, and server-side fallback. I could spend the next year rebuilding a mature gateway, or I could adopt one and put my energy where it actually differentiated me: the policy above it.

I adopted OmniRoute — a gateway that already speaks to hundreds of providers with a stack of routing strategies — and moved my rotation logic down into its combos. A combo is a weighted group of models the gateway rotates through, with fallback built in. The mental shift that mattered: the combo, not the individual model, became my execution unit. My orchestrator stopped iterating over models one at a time. It hands work to a combo and trusts the gateway to try every member. "Each node tries every model" became a guarantee I got for free instead of code I maintained.

That unlocked the resilience patterns I'd been faking:

• Round-robin dispatch to spread load, so I don't concentrate every request on one provider and either burn its paid credits or earn a rate-ban.
• Two kinds of cooldown — a short one for a rate-limited model (back off for a minute) and a long one for a model that's blown its daily quota (park it for the day). A 429 shouldn't crash a run; it should rotate.
• Free-before-paid interleaving — try the free attempts first, and let paid slots fire only after the free ones miss. Cost discipline as a routing rule, not a hope.

The availability floor, and why I stopped trusting my own docs

The most important rung in any combo is the last one: a free, self-hosted local model that's always reachable. I call it the availability floor. When every fancy provider is rate-limited, quota-exhausted, or down, the floor catches the request so the loop keeps moving. No floor means a bad afternoon of "circuit breaker open" errors and dead runs.

And this is where I earned a scar I want you to skip. I had the floor documented. The config file clearly showed a local model as the terminal rung. I'd written "done" and moved on. Weeks later, when runs kept dying on circuit-breaker-open, I finally checked the live gateway — and the running combos had no local member at all, and were set to plain round-robin instead of ordered priority. Round-robin doesn't guarantee the floor gets tried; only an ordered combo does. The doc was right. The system was wrong. They'd drifted, and I'd trusted the doc.

The lesson, which I've now relearned enough times to tattoo on: a documented fix is not a fix. Verify the change is in the file and applied to the live system before you write "done." Especially for routing, where the failure is silent until everything's on fire at once.

Treat the gateway as a dependency you don't control

Adopting someone else's gateway bought me a year of engineering and a new problem: it's a black box I don't govern. It ships breaking changes on its schedule. Its reported pricing isn't always trustworthy — a model that claims to cost nothing might just have a missing price. And it doesn't take my pull requests, so I can't fix any of that upstream.

My answer was to wrap it behind a single anti-corruption seam. Everything my agents see is a trusted projection of the gateway: a curated model catalog I control, and a pricing overlay that refuses to believe a "$0" unless I've explicitly whitelisted it — unknown-price models get excluded, not defaulted to cheap. Upstream churn hits that one seam instead of rippling into every node. And because the seam exists, I could rip out OmniRoute and drop in my own router later without touching a single agent. That's the same instinct as the provider-adapter contract from stage one, just aimed at a bigger dependency: never let an outside thing's shape become your shape.

The throughline

Three rewrites, one pattern: a reasoning router that owns policy, sitting above an aggregation gateway that owns transport, with a hard seam between them. The relay taught me transport isn't enough. The router taught me policy belongs in its own layer. The gateway taught me to buy transport instead of building it — and to fence it off like the untrusted dependency it is. The standalone projects are all retired now. The pattern is the only thing I kept, and it's the only thing that mattered.

If you're wiring up model routing for your own agents, start with the seam, not the gateway. Decide where "which model and why" lives before you pick "how to reach it," and you'll survive swapping the second one out. If you've drawn the line somewhere different — or found a gateway you trust more — I'd love to hear how it's holding up. As always, I'm here to help!