# Dedicated GPU Server: How to Choose a Card for ML

> A dedicated GPU server requires calculating VRAM, power, and PCIe lanes: choosing a card for training, inference, rendering, and passthrough virtualization.

## Which GPU do you need: training, inference, or rendering

Choosing a GPU for a dedicated server depends on the task, not on "which card is more powerful." Training large models requires a lot of VRAM and high memory bandwidth. Inference (running an already trained model) can run on a card with less memory, but with a focus on low latency. Rendering and video encoding depend more on the number of CUDA/Tensor cores and support for the codecs you need.

A common beginner mistake is buying the most powerful card "just in case" without checking whether the model or scene even fits in its memory. If it does not fit, thousands of extra cores do not help much.

## VRAM matters more than core count

The amount of video memory determines which model you can load at all and what batch size you can train with. Rough task classes:

| Task class | Minimum VRAM | Comment |
| --- | --- | --- |
| Inference of small models, scene rendering | 16 GB | Enough for most applied tasks |
| Fine-tuning medium models | 24-48 GB | Requires a workstation-class card |
| Training large models from scratch | 80 GB and up | Usually several cards with NVLink |

If a model does not fit in one card's memory, you either have to cut the batch size or split the model across several GPUs — and that means a different server architecture and different requirements for the [processor](https://www.zevshost.net/en/kb/vybor-protsessora-servera/), which has to keep feeding data to the cards.

## Power and cooling for a GPU server

A top-tier ML card draws 300-700 W under load. A server with 4-8 such cards easily reaches 4-6 kW of total consumption — that is no longer a standard 16 A rack circuit, but a separate calculation for [power redundancy](https://www.zevshost.net/en/kb/elektropitanie-servera-rezerv/) with two independent feeds.

Cooling should not be underestimated either: a GPU server packed with cards needs directed front-to-back airflow and may not fit into a compact chassis without hurting the thermal envelope.

## CPU, PCIe, and form factor for multiple cards

Each data-center-class GPU occupies a PCIe x16 slot. With 4 or more cards the processor must provide 64+ PCIe lanes without sharing them with network cards and NVMe drives — one of the reasons GPU servers are almost always built on platforms with a high PCIe lane count rather than consumer chipsets.

The form factor matters too: full-height cards with active cooling physically do not fit into just any chassis, and it is worth checking the article on [1U and 2U form factors](https://www.zevshost.net/en/kb/server-formfaktor-1u-2u/) — most multi-card GPU servers are built in 3U-4U chassis exactly because of cooling and card-length requirements.

## NVLink, passthrough, and GPU virtualization

If the cards need to exchange data directly, without going through the host's system memory, you need NVLink or a similar bus — it speeds up distributed training across several GPUs several times over compared to exchanging data through PCIe.

Virtualization uses either PCIe passthrough (the whole card is given to one virtual machine) or vGPU (the card is split among several VMs in software, if the vendor supports it). Passthrough is simpler to configure and delivers full performance, but rules out sharing the card.

```
nvidia-smi
nvidia-smi -q -d POWER
```

## GPU server selection checklist

- Calculate how much VRAM your model actually needs instead of buying extra just in case.
- Check the total power draw of all cards and the power supply headroom.
- Confirm the number of PCIe lanes on the processor when using 4 or more cards.
- Decide in advance: passthrough or vGPU, if the server will be virtualized.

## Frequently asked questions

### What should be done if a model does not fit into the memory of a single GPU?

If a model does not fit in one card's memory, you either cut the batch size, which slows training, or split the model across several GPUs. That second path needs a different server design and a processor able to feed every card with data quickly, so pick the card after calculating the memory the task needs, not with extra margin.

### Is NVLink required whenever a server has several GPUs?

No, NVLink is not always needed — only when cards must exchange data directly with each other, bypassing the host's system memory, which matters for distributed training of large models across several GPUs at once. If the task is inference or rendering on a single card, that kind of direct link between cards is not needed at all.

### What is the difference between passthrough and vGPU when choosing how to virtualize a GPU?

Passthrough gives the whole card to one virtual machine, is simpler to set up, and delivers full performance, but rules out sharing the card with other machines. vGPU splits a single card among several virtual machines in software, if the vendor supports it, trading some performance for flexibility in splitting resources among projects.

### Why does chassis form factor matter for a multi-GPU server, not just the number of cards?

The form factor matters because full-height cards with active cooling do not physically fit into just any compact chassis, and packing several cards tightly needs directed front-to-back airflow to keep the thermal envelope under control. That is why multi-card GPU servers are almost always built in tall chassis designed exactly for that kind of cooling and card length.

---

Source: https://www.zevshost.net/en/kb/server-s-gpu-vybor/