Proprietary LLM APIs bill per token and send your data to a third party. At high, predictable volume, self-hosting open-weight models, Llama, Mistral, DeepSeek, Qwen, on your own inference (vLLM/TGI) can cut cost and keep data in-house. The trick is doing it behind an OpenAI-compatible gateway so application code barely changes, and proving quality with evals.
When it makes sense
Self-hosting wins on steady, high-volume workloads and data-residency requirements. For spiky or low-volume usage, a hosted API is often cheaper and simpler. Know which one you have before committing GPUs.
The reason the decision is subtle is that the two pricing models have opposite shapes. A proprietary API has no fixed cost: you pay per token, and a quiet week is nearly free. A self-hosted GPU has a fixed cost: a reserved card bills around the clock whether it is saturated or idle. So the comparison is not which per-token rate is lower, it is whether your token volume, spread across an always-on GPU, drives the effective per-token cost below the API rate. That produces a crossover point: below it the API is cheaper because you are not paying for idle silicon; above it self-hosting wins because the fixed cost is amortized across enough tokens. Bursty and low-volume workloads sit below the line; only steady, high, predictable throughput sits above it.
Stand up inference
Serve the model with vLLM or TGI (GPU nodes or managed inference), exposing an OpenAI-compatible endpoint. Front it with a gateway so you set OPENAI_BASE_URL and minimal else changes in apps. Bring your prompt library, tool/function schemas, RAG/grounding, and, critically, your evaluation suite.
The evaluation problem
Moving providers is as much an evaluation problem as a plumbing one. The same prompt sent to two models produces different output, and the gap is task-specific, so a public benchmark tells you little about the workloads you are actually moving. Build an eval suite from your real use cases: a golden set of representative prompts with expected outputs or rubric scores, judged on the failure modes you care about such as instruction-following, JSON validity, and refusal behavior. Prompts and function-calling do not always port cleanly between models, so re-test them rather than assuming parity.
A/B before you commit
Run the open model against your eval set and compare quality, latency, and cost-per-1k-tokens versus the incumbent. Then shift traffic gradually, workload by workload, behind the gateway, keeping the incumbent API as instant fallback. Tune quantization for the latency/quality/cost balance you need. A hybrid end state is normal: self-host the high-volume, well-bounded tasks and keep the API for the long tail where a frontier model still leads. Do not assume a specific open model matches a specific closed model’s quality, let your evals decide per workload.
What has to pass before you flip traffic
Quality/regression evals vs incumbent, latency/throughput and cost tests, and guardrail/safety + jailbreak checks. Rollback is routing traffic back to the incumbent at the gateway, instant, since app code is unchanged.
Where the GPU math bites
Model on monthly token volume (annualized) and required latency SLOs; self-hosting trades per-token API spend for GPU/compute and MLOps effort. Utilization is the number teams most often get wrong: a card busy during the day and idle overnight is not the same economics as one saturated around the clock, so size for concurrency and watch the KV cache rather than single-request latency. Remember too that the incumbent API bundled things you now own, content moderation, abuse monitoring, and prompt-injection defenses move to your gateway, so budget engineering time for the safety layer alongside the hardware.
Use the TCO calculator to model a token-based comparison, then treat the figures as illustrative until your own evals confirm quality at your volume.