60 days isn't a lot of runway
Your business has built a workflow around Claude or GPT. Maybe it's a customer service bot, an internal knowledge assistant, or an automated case-handling process. Then one day, the notice arrives: the model you built on is being retired.
Anthropic guarantees at least 60 days' notice before a publicly released model is taken out of service. That's a real commitment, not just a formality — and, compared to what most people assume, it's stricter than the alternative. OpenAI gives at least six months' notice for its standard models, but for preview releases (anything with "preview" in the name), the notice can be as short as two weeks.
Two weeks isn't enough time to test, validate, and roll out a replacement for something running in production. Even 60 days is tight if you haven't prepared anything in advance.
This isn't an argument for abandoning Claude or GPT. It's an argument against building everything on one fragile thread.
The problem isn't the vendor — it's the absence of a plan B
The major providers are where they are for good reasons: quality, safety, support, and an ecosystem that works. There's nothing wrong with building the core of your business on Claude or GPT.
The problem starts when your entire workflow is hardcoded against one model, one API, one provider — with no way out if something changes. Models get retired. APIs change. Prices shift. Services go down. None of these events should be able to stop your business entirely.
The solution has three layers, built in increasing order of severity — most businesses don't need to build all three on day one.
Layer 1: A gateway between your application and the provider
The simplest layer is a thin gateway between your application code and the AI provider itself, so switching models doesn't require rewriting code.
Two tools dominate this space: LiteLLM and Portkey. Both are open source and let you define a primary model with a list of fallbacks — if the primary model fails or gets rate-limited, the system automatically routes to the next in line, transparently to the application.
Be honest with yourself about what this actually takes. Self-hosted LiteLLM as a proxy needs its own database and typically Redis as well, and realistic setup for a team doing it for the first time takes 2–4 weeks — plus ongoing maintenance afterward. Portkey has more production-hardened failure handling with built-in circuit breakers, but the full feature set lives in a paid, hosted service.
Worth noting: Portkey is currently in the process of being acquired by Palo Alto Networks. It's a useful reminder that even the tools you pick to avoid vendor dependency aren't immune to consolidation. Build with your eyes open, not with a false sense that any single choice is permanently safe.
For most SMBs, the conclusion is simple: a self-hosted gateway layer is something you build once you have dedicated capacity to run it — not something you set up on a Friday afternoon.
Layer 2: The built-in option, if you're already on Azure
If your business already runs Microsoft 365 and Azure — as most of our clients do — there's an easier place to start. Model Router in Azure AI Foundry automatically routes across more than 25 models from different vendors, with failover switched on by default for the standard deployment.
Two things worth knowing before you set it up: for custom model subsets, you need to select at least two models for automatic failover to work — pick just one, and the redundancy disappears. And if you want Claude included, it needs to be set up as a separate deployment in Foundry first — it isn't included automatically.
For most businesses in our target group, this is the most realistic first step. It doesn't require new infrastructure, just correct configuration of something you're already paying for.
Layer 3: Open source as a last resort
The third layer is an open-weight model you can run yourself — not as your daily driver, but as an emergency fallback for two specific scenarios: your provider does something you won't accept (a terms change, a price spike, a policy shift that no longer fits your business), or data must legally stay in the EU and can't be sent to a US cloud.
The quality gap hasn't closed — it's moved
It used to be a common claim that open models had caught up to closed ones on pure knowledge. That's true, but slightly misleading: the MMLU benchmark commonly used to measure this is now widely considered "saturated" — most leading models, open and closed alike, score 88–94 percent. The test no longer discriminates well between them. On harder knowledge benchmarks like GPQA Diamond, a roughly seven-point gap still favors closed models.
Where the gap is real and still significant is on agentic tasks and production coding — exactly what most SMBs actually want to use AI for in practice. On SWE-bench, one of the most widely used benchmarks for AI solving real coding tasks, the best closed models sit roughly 12 percentage points ahead of the best open ones. And according to Stanford's AI Index report, this gap is actually widening again, after a period where it appeared to be closing. The Opus and GPT-5 class of models still leads clearly here.
The takeaway: use an open model as a fallback or for lower-stakes tasks, by all means. Don't expect it to match Claude or GPT on complex, multi-step tasks that require genuine reasoning.
Cost: not automatically cheaper
The most common misconception is that self-hosting is cheaper because you "own" the model. That's true only at high, steady volume — and the break-even point varies far more than people assume, depending on which API you're comparing against and which GPU provider you use. As a rough rule of thumb, you'll typically need somewhere between 2 and 30 million tokens a day in steady traffic before self-hosting starts to beat an expensive API — and against a cheap open API, that number can be much higher.
On top of that comes the operational cost that never shows up on an invoice: a working self-hosted stack needs ongoing monitoring, upgrades, and troubleshooting. Our estimate, based on what we see with clients who've tried this themselves, is somewhere between 20 and 30 percent of an engineer's time — but treat this as an indication, not a hard figure. There's no good shortcut here.
For the vast majority of SMBs with moderate, variable volume, the conclusion is simple: self-hosting doesn't win on pure cost. It's done for control, not to save money.
If you're going to run it yourself anyway
Most teams pick vLLM or TensorRT-LLM as the serving engine — both are mature, production-ready inference engines. SGLang has also emerged as a genuine third option, particularly for workloads with a lot of repeated context, like agentic tasks and RAG.
At scale, this is typically orchestrated in Kubernetes, but for a single GPU node, a plain Docker container running vLLM is often enough. Don't build more infrastructure than the task actually needs.
If you don't want to run GPUs yourself, managed inference providers like Together AI and Fireworks AI exist, and Azure and AWS both offer hosting of open models directly. Pricing there typically lands well below Claude or GPT API pricing, but above raw self-hosted compute at high volume — a sensible middle step for most businesses.
The best open models right now
The landscape moves fast, but as of today: Kimi K2.7 Code from Moonshot AI (modified MIT license) is strong on coding and agentic tasks. DeepSeek V4 (MIT license) remains a solid choice for reasoning and math. Qwen 3.5 from Alibaba (Apache 2.0) is generally regarded as the best all-round open model — note that newer Qwen releases have partly moved away from a fully open license. GLM-5.2 from Zhipu offers a one-million-token context window and is strong on agentic tasks. Llama 4 Scout from Meta still offers 10 million tokens of context and broad platform support, even though Meta hasn't shipped a successor yet. Mistral remains the natural choice if you need an EU-based provider. Gemma 4 from Google is best suited for local or edge deployment with limited resources.
The orchestrator pattern: let the expensive model think, let the cheap models work
A pattern getting increasing attention is having an expensive, powerful model — Opus or GPT-5 class — plan and quality-check the work, while cheaper or open models like Qwen or DeepSeek do the actual heavy lifting.
Figures of 5-10x cost reduction circulate around this pattern. That number reflects the raw price gap between a frontier model and a small one, not what you actually end up with once the system is built and running in production. The most solid, measured figures we found — from the RouteLLM research project and from Anyscale — show a real cost reduction of 70-85 percent, roughly 3-7x, while retaining around 95 percent of the quality the expensive model alone would deliver. Still significant, but not quite as dramatic as 5-10x suggests.
The pattern also has real weaknesses a balanced assessment should mention: errors from a cheap model can propagate downstream, work misrouted to a model that isn't good enough can cause quality loss that's hard to detect, and the complexity of debugging across multiple models isn't free. Done properly, this requires an evaluation process that catches quality drops before they reach the customer — not just a cost spreadsheet.
What we actually recommend
Don't build all three layers at once. Most Norwegian SMBs should start here:
- Check whether you already have Azure Model Router available. If you're on Microsoft 365 and Azure, this is the cheapest and fastest redundancy layer you can set up — configuration, not new infrastructure.
- Build a gateway layer only once you have the volume and the capacity to run it — not because it sounds smart.
- Keep an open model in reserve as an emergency fallback, not as your daily driver — and know that it doesn't yet match Claude or GPT on complex tasks.
At its core, this isn't really a technology choice. It's structure — knowing what dependencies you have, what happens if one of them fails, and who decides what you do about it. That's exactly the kind of mapping we do in an AI Ready assessment, alongside the rest of the governance work around an AI implementation.
AI doesn't start with technology. It starts with structure.