The reason teams look at this migration is rarely the model itself. It is the bill. Azure OpenAI charges per input and output token, and at sustained high volume that meter never stops. Self-hosting an open-weight Llama model swaps a per-token charge for a fixed GPU cost you control, plus the data residency and customization that come from running the weights yourself. The catch is that you now own the GPU ops, the eval, and the safety layer that Azure handled for you.
Scale-to-zero billing vs a GPU that never stops
Because this migration is a bill decision, it is worth being precise about the two cost shapes before touching any infrastructure. Azure OpenAI’s per-token billing has no fixed cost: it effectively scales to zero, so a quiet week is nearly free and a bursty workload never pays for capacity it did not use. A self-hosted GPU is the opposite, a fixed cost that bills around the clock whether the card is saturated or idle.
That means the honest comparison is not “which per-token rate is lower.” It is “does my token volume, spread across an always-on GPU, drive the effective per-token cost below Azure’s rate?” There is a crossover point: below it, Azure’s scale-to-zero pricing wins because you are not paying for idle silicon; above it, self-hosting wins because the fixed GPU cost is amortized across enough tokens. Bursty, seasonal, or low-volume workloads sit below the line and should usually stay on Azure. Only steady, high, predictable throughput sits above it. Model your real monthly token volume against the GPU hourly rate and expected utilization in the calculator above, and treat a single busy day as unrepresentative.
Keep the API surface stable
The single best decision you can make is to not rewrite your application’s call sites. Serve Llama behind an OpenAI-compatible API so your existing SDK calls barely change.
vLLM does this out of the box:
vllm serve meta-llama/Llama-3.3-70B-Instruct \
--tensor-parallel-size 2 \
--max-model-len 16384 \
--gpu-memory-utilization 0.92That exposes /v1/chat/completions. In your client, the change is the base URL and the model name:
client = OpenAI(base_url="http://llama-gw:8000/v1", api_key="local")
resp = client.chat.completions.create(
model="Llama-3.3-70B-Instruct",
messages=msgs,
)Azure OpenAI addresses models by deployment name, not model id. Build a small mapping table so each old deployment name resolves to a Llama endpoint, and keep it in config so you can re-point traffic without a code change.
Sizing the GPUs
This is where the planning lives. Rough VRAM at 16-bit weights is about 2 GB per billion parameters, plus headroom for the KV cache:
- Llama 8B: fits on a single 24 GB card (one A10 or L4) comfortably.
- Llama 70B at FP16: needs roughly 140 GB, so two 80 GB H100s with tensor parallelism, or four 40 GB cards.
- Quantize to 4-bit (AWQ or GPTQ) and a 70B model drops to about 40 GB, fitting a single 48 GB card with a quality trade-off you must measure, not assume.
Throughput comes from continuous batching, which vLLM does automatically. A box that serves one request slowly will serve fifty concurrent requests at high aggregate tokens per second, so size for concurrency and watch the KV cache, not single-request latency.
Evaluate on YOUR prompts
Do not trust a leaderboard. A GPT-4-class model and a Llama model differ on real workloads, and the gap is task-specific. Before cutover, run your actual production prompts through both and score them:
- Build a golden set of 200 to 500 real prompts with expected outputs or rubric scores.
- Run Azure OpenAI and Llama side by side, then A/B a slice of live traffic with logging.
- Watch for the failure modes that matter to you: instruction-following, JSON validity, refusal behavior, long-context recall.
The point is that this migration is as much an evaluation problem as a plumbing one. Two models given the identical prompt respond differently, and the gap is task-specific, so a leaderboard average tells you nothing about the workloads you are actually moving. Make the golden-set eval a gate that every traffic shift has to pass, and dual-run Azure and Llama in parallel before cutover so you have real evidence rather than a hunch. If a workload regresses, that is a signal to keep it on Azure, not a bug to force past. Keep logging the live A/B after cutover so quality drift shows up on a dashboard instead of in a support ticket.
Safety is now yours
Azure ships content filters and abuse monitoring. When you self-host, those are gone. You own the guardrails: add an input/output moderation pass (Llama Guard is a natural fit), rate limiting, and prompt-injection defenses at the gateway. Budget engineering time for this; it is not optional in production.
Where Azure OpenAI still wins
Be honest before you buy GPUs:
- Frontier quality. For the hardest reasoning and coding tasks, the top Azure-hosted models still lead open weights. If quality is the product, do not trade it for a lower bill.
- Zero GPU ops. No capacity planning, no driver hell, no on-call for a wedged inference server.
- Low-volume economics. Azure effectively scales to zero: at bursty or low volume you pay only for tokens used. A reserved GPU bills 24/7 whether or not it serves traffic.
- Compliance and filters. Built-in certifications and content moderation that you would otherwise build and maintain.
It comes down to a volume bet
This migration is fundamentally a volume bet. Put Llama behind an OpenAI-compatible vLLM endpoint, map deployment names to model endpoints, size GPUs against your concurrency, quantize where the eval allows, and dual-run before you trust it. Self-hosting pays off only at sustained high token throughput; below that line, Azure’s scale-to-zero pricing and zero ops usually win. Plug your monthly token volume, GPU hourly rate, and utilization into the calculator above to find exactly where your break-even sits before committing to either path.