Most teams pick an LLM path the wrong way: they compare leaderboard scores and per-token price lists and pick whatever looks strongest and cheapest on paper. That is how you end up self-hosting a model your evaluation suite cannot actually vouch for, or staying on an API that quietly bills more than an owned GPU would at your volume. The better approach is to decide on a short list of factors that genuinely change the answer, score your real workload against them, and only then look at models. This guide gives you that rubric. The one thing to internalize first: this is as much an evaluation problem as a plumbing one, and no leaderboard settles it for the tasks you are actually moving.
Get these facts down before you shortlist a model
Before you shortlist anything, write down two numbers, one bar, and one list: your realistic monthly token volume (annualized), the utilization profile of that volume across a day and a week, the minimum quality bar each workload must clear, and the list of governance constraints you are actually bound by rather than merely aware of. Utilization is the number teams most often get wrong, because a self-hosted GPU bills around the clock whether it is saturated or idle, while an API charges nothing on a quiet week. Get those honest first; the model shortlist falls out of them.
What separates self-hosting from an API
- Workload volume and GPU utilization. Self-hosting only saves money at steady, high, predictable utilization, where a fixed GPU cost amortizes across enough tokens to beat the API rate. Bursty or low-volume traffic sits below that crossover, and an API stays cheaper.
- Your quality bar and eval maturity. If you cannot yet measure quality on your own tasks, you cannot safely switch anything. A mature evaluation suite is the prerequisite, not a nice-to-have, because the quality gap between two models is task-specific.
- Data residency and governance. A hard requirement to keep data in-house or in a region pushes you toward self-hosting or a residency-compliant managed offering, and can override the cost math entirely.
- Latency needs. Tight, predictable latency SLOs favor infrastructure you control and can size for concurrency; looser tolerances leave the managed API firmly in play.
- GPU access and MLOps skills. Serving models at production quality takes reservable GPU capacity and people who can run inference, quantization, and a safety layer. Without both, self-hosting is cheaper only in the spreadsheet.
- Open weights versus managed. Open weights buy control and portability; a managed frontier API buys the long tail of hard tasks and offloads the safety and abuse tooling you would otherwise own.
Which path fits which workload
Score your workload against the criteria above, then use these as starting points rather than conclusions. Steady, high-volume, well-bounded tasks with a mature eval suite and in-house GPU and MLOps skills are the ones where self-hosting an open-weight model, Llama, Mistral, DeepSeek, or Qwen, on your own inference tends to pay off. Bursty, low-volume, or highly varied work usually stays cheaper and simpler on a managed API such as OpenAI GPT, Anthropic Claude, Google Gemini, or Cohere. A shop already committed to a hyperscaler often finds Azure OpenAI the path of least resistance for the managed side. Crucially, do not assume a specific open model matches a specific closed model’s quality on your tasks; that is a claim your evals have to earn per workload, not one a benchmark can grant. A hybrid end state, self-hosting the high-volume bounded tasks and keeping an API for the long tail, is normal and often correct.
How teams talk themselves into the wrong path
Three errors show up repeatedly. The first is deciding on a public benchmark, which tells you little about your own prompts, JSON validity, instruction-following, or refusal behavior. The second is sizing self-hosting on peak or on single-request latency rather than on real utilization and concurrency, which flatters the GPU economics until the idle overnight hours show up on the bill. The third is forgetting that the managed API bundled content moderation, abuse monitoring, and prompt-injection defenses that all become your job the moment you self-host. A quieter fourth: treating prompts and function schemas as portable, when they often need re-tuning between models.
Prove it on your own tasks
Turn your top candidate into a controlled trial with written acceptance criteria: run the open model against a golden set built from your real use cases, judged on the failure modes you care about, and compare quality, latency, and cost-per-token against the incumbent. Shift traffic gradually behind an OpenAI-compatible gateway, workload by workload, keeping the incumbent API as instant fallback so a rollback is a routing change rather than a re-migration. Model the token-based total cost with your own volume and utilization, and treat the figures as illustrative until your evals confirm the quality holds. Choose the path that clears your quality bar at an acceptable cost and operational load, not the one that topped a leaderboard.
Use the TCO calculator to model a token-based comparison, then let your own evals decide per workload.