How Much VRAM a Language Model Needs: Sizing Guide

A 7-billion-parameter model takes 14 GB in fp16 or 4 GB with 4-bit quantization — a 3.5x difference for the same set of weights. Buying a GPU server without calculating VRAM first is a common mistake: the card either sits half-idle, or the model does not fit and crashes with CUDA out of memory. Let's work out how to size it precisely instead of guessing.

Why calculate VRAM in advance

Video memory is the one GPU resource you cannot extend on the fly, unlike system RAM or disk. If a model weighs more than the available VRAM, inference either fails to start, or the driver starts offloading layers to system RAM through --gpu-memory-utilization or a similar flag, and speed drops several times over. Sizing it before buying a server saves both money and the time spent rebuilding the setup.

The rule is simple: you need to count not just the model weights but also the context, the batch, and CUDA's own buffers. Sometimes the context alone eats half the card on long conversations.

The formula for sizing VRAM for inference

A basic estimate of memory needed for the weights:

VRAM_GB = (params_billion * bytes_per_param) * 1.2

The 1.2 factor is a margin for activations, the KV cache, and memory fragmentation. bytes_per_param depends on precision: 2 bytes for fp16/bf16, 1 byte for int8, 0.5 bytes for int4. For a 13-billion-parameter model in fp16 you get 13 * 2 * 1.2 ≈ 31.2 GB — that already will not fit on a single 24 GB card.

Add the KV cache separately: it grows linearly with context length and the number of concurrent requests. For a 7B model with an 8192-token context and one conversation, the cache takes roughly 1-2 GB, but with ten parallel sessions it climbs to 10-20 GB.

How quantization changes memory requirements

Quantization lowers weight precision but sharply cuts VRAM usage and barely hurts answer quality at 8 and 4 bits. Below is a reference point for a 7-billion-parameter model.

For running on a CPU with no GPU at all, check out GGUF quantization in llama.cpp — the same formula applies there, but to system RAM instead of VRAM.

VRAM for fine-tuning differs from inference

Fine-tuning needs 3-4 times more memory than plain inference of the same model, because you also have to store gradients and optimizer state. Full fine-tuning of a 7B model in fp16 takes about 60-70 GB of VRAM — already several GPUs. The LoRA method removes this problem: only small adapter matrices get trained, while the base weights stay frozen at 4-bit quantization.

With LoRA and QLoRA quantization, fine-tuning that same 7B model fits into 10-12 GB of VRAM — doable on a single mid-range card instead of a cluster.

How much VRAM popular models need

These figures include a margin for a 4096-token context with a single user; as load grows, budget an extra 10-20% for each parallel request stream.

Which GPU to pick on a budget

For a 7B model at 4-bit quantization, a card with 8 GB of VRAM is enough — a fitting task for Ollama on a VDS with a single consumer GPU. For 13B in fp16, or a chat service with several concurrent users, you need a card with 24 GB of VRAM. For 70B without quantization there is no reasonable alternative to a multi-GPU cluster — take the int4 version and a single 48 GB card, or split layers across two cards using vLLM with tensor parallelism.

• Up to 8 GB VRAM — 7B models in int4, personal and small projects.
• 16-24 GB VRAM — 7-13B in fp16 or 34B in int4.
• 48 GB and above — 70B in int4 or fine-tuning mid-sized models.
• Multiple GPUs — 70B in fp16 or training from scratch.

Summary: how to avoid sizing mistakes

Before ordering a server, calculate memory with the formula, add a KV cache for the real context length and the number of parallel requests, then add a 15-20% margin for fragmentation. It is cheaper to buy a card with extra VRAM once than to migrate a model later and lose time moving the weights.

• Determine the model's parameter count and target precision (fp16, int8, int4).
• Calculate the base amount with the formula and add a KV cache for the context.
• For fine-tuning, budget 3-4 times more memory than for inference, or use LoRA.
• Pick a GPU with a 15-20% margin above the calculation — for fragmentation and load growth.

Frequently asked questions

What happens if there is not enough VRAM — does the model simply refuse to start?

The behavior depends on the runtime: some frameworks fail immediately with an out-of-memory error, while others automatically offload part of the model's layers to system RAM and keep working, just noticeably slower. That mode suits testing and occasional requests, but a production service with live traffic should be sized with a proper VRAM margin from the start.

Does quantization noticeably hurt answer quality in practice?

Quantization is barely noticeable on everyday conversations and simple tasks, but on complex reasoning, precise calculations, and long logical chains the quality gap becomes noticeable at the most aggressive compression levels. For most hosting scenarios moderate quantization is a sound trade-off, while heavy compression is worth testing against your own sample tasks before relying on it in production.

Should the number of simultaneous chat users factor into choosing a GPU?

Yes, the number of concurrent conversations directly affects how much memory is needed, because every active request keeps its own context cache on top of the base model weights. If a service must serve several parallel users, the card should be sized with that in mind rather than just for a single test request typed into a console.

Can a very large model run if it does not fit into one card's VRAM?

Yes, a model can be split across several GPUs when it does not fit on a single card, using parallelism modes that divide its layers or tensors between devices. That setup is more complex to configure and usually trails a single powerful card in speed, but it remains a workable way to run a model that would not otherwise fit.