Sizing a GPU server for LLM inference

Most LLM sizing questions reduce to one: do the model and the memory it needs to serve your traffic fit in GPU memory? This guide shows the arithmetic, applies it to FlexDedi's two GPU servers, GPU L40S with one NVIDIA L40S 48 GB and GPU RTX PRO with two NVIDIA RTX PRO 6000 Blackwell 96 GB, and gives vLLM commands for both.

The VRAM equation

VRAM needed ≈ weights + KV cache + runtime overhead

Runtime overhead covers the CUDA context, activations, CUDA graphs and allocator fragmentation; budget a few GB per GPU. vLLM reserves the fraction of GPU memory set by --gpu-memory-utilization (0.90 in the examples below), loads the weights, profiles activation memory, and hands everything left over to the KV cache.

Plan with the capacity nvidia-smi reports, which can be a few GB below the nominal figure because ECC on GDDR memory reserves part of it. As working budgets at 0.90 utilization, this guide uses about 40 GB for one L40S and about 85 GB per RTX PRO 6000, roughly 170 GB across both.

Weights: parameters × bytes per parameter

weights (GB) ≈ parameters (billions) × bytes per parameter

INT4 costs a little more than half a byte per parameter because quantization scales, and often the embeddings and output layer, stay in higher precision. Both GPUs run FP8 in hardware. Blackwell adds FP4 tensor cores, which help only if your serving stack has FP4 kernels for that GPU and model. Lower precision can cost accuracy, so validate a quantized model on your own evaluation set before switching.

For mixture-of-experts (MoE) models, memory follows the total parameter count, because every expert must be resident, while per-token compute follows the much smaller active count.

KV cache: the part that grows with traffic

Each token of each active sequence keeps a key and a value vector in every layer:

KV bytes per token = 2 × layers × KV heads × head dimension × bytes per element
KV cache ≈ KV bytes per token × tokens per sequence × concurrent sequences

Read the inputs from the model's config.json: num_hidden_layers, num_key_value_heads, and head_dim (or hidden_size divided by num_attention_heads). Grouped-query attention, with far fewer KV heads than attention heads, is what keeps these numbers manageable.

--kv-cache-dtype fp8 halves the cache, usually at a small accuracy cost that is worth checking on your evaluation set. vLLM allocates the cache in blocks as tokens arrive, so short requests do not reserve the full context. Plan for the worst case you are willing to serve anyway: when the cache is full, new requests wait.

Example 1: a 14B model on one L40S

Serving Qwen/Qwen3-14B-FP8 on GPU L40S, all figures approximate:

• Budget: about 40 GB
• Weights: about 16 GB (14.8B parameters at 1 byte, plus layers kept in higher precision)
• Overhead: about 3 GB
• KV cache: 40 − 16 − 3 ≈ 21 GB
• Capacity at 80 KiB per token: 21 GB ÷ 80 KiB ≈ 250,000 tokens

That is roughly 7 concurrent sequences at the model's full 32K context, or around 60 at 4K tokens each.

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv --python 3.12 --seed && source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels   # optional: fetch prebuilt kernels instead of compiling at startup

vllm serve Qwen/Qwen3-14B-FP8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 64 \
  --host 10.0.0.10 --port 8000 \
  --api-key "$VLLM_API_KEY"

Install a current NVIDIA data-center driver first and confirm that nvidia-smi lists the GPU; Blackwell cards such as the RTX PRO 6000 require NVIDIA's open kernel modules. At startup, vLLM logs the KV cache size and the maximum concurrency for your --max-model-len, so compare them with your estimate. Binding --host to a private address and setting --api-key keeps the endpoint off the public internet and requires a bearer token on every request.

Example 2: a 70B model across two RTX PRO 6000

A 70B model in BF16 needs about 141 GB for weights (70.6B parameters × 2 bytes), more than one GPU holds. With tensor parallelism, each GPU holds half the weights and half the KV heads:

• Budget per GPU: about 85 GB
• Weights per GPU: about 70.5 GB
• Overhead per GPU: about 3 GB
• KV cache per GPU: about 11.5 GB
• Per token per GPU at FP8: 160 KiB ÷ 2 = 80 KiB
• Capacity: 11.5 GB ÷ 80 KiB ≈ 140,000 tokens, about 4 sequences at 32K or 34 at 4K

With FP8 weights instead (about 70 GB, or 35 GB per GPU), the KV budget grows to about 47 GB per GPU: roughly 570,000 tokens, or 17 sequences at 32K. That difference is why FP8 checkpoints are worth evaluating for 70B-class models on this server.

export HF_TOKEN=hf_your_token   # Llama weights are gated: accept the license on Hugging Face first
vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype fp8 \
  --host 10.0.0.10 --port 8000 \
  --api-key "$VLLM_API_KEY"

Tensor parallelism over PCIe

Neither GPU has NVLink, so traffic between the two RTX PRO 6000 cards runs over PCIe 5.0; nvidia-smi topo -m shows the path. Tensor parallelism synchronizes the GPUs inside every layer, so it pays off when a model does not fit on one GPU, or when you need lower latency per request on a large model.

For a model that fits on one GPU, run one replica per GPU instead. There is no traffic between the cards, and vLLM spreads requests across both:

vllm serve Qwen/Qwen3-14B-FP8 --data-parallel-size 2 --max-model-len 32768 --kv-cache-dtype fp8

--pipeline-parallel-size 2 is a third option for large models. It splits layers rather than tensors and exchanges less data, at the cost of pipeline bubbles, so benchmark it against tensor parallelism on your own traffic.

What fits where

Approximate fit at 0.90 utilization with an FP8 KV cache:

CPU, RAM and NVMe still matter

System RAM. Weights pass through host memory when they load, and the Linux page cache keeps recently loaded model files so a restart does not reread them from disk. A common rule of thumb is at least as much system RAM as total VRAM: GPU L40S has 256 GB for 48 GB of VRAM, and GPU RTX PRO has 576 GB for 192 GB. The rest serves the API server, tokenization, retrieval components such as vector databases and embedding models, and KV-cache offloading to CPU memory with tools such as LMCache.

CPU. The EPYC 9355P (32 cores) in GPU L40S and the EPYC 9455P (48 cores) in GPU RTX PRO handle scheduling, tokenization, and pre- and post-processing. A saturated host CPU shows up as GPU idle time between steps, which nvidia-smi dmon makes visible.

NVMe. Load time is roughly model size ÷ sustained read speed. As an estimate, 141 GB at an assumed 5 GB/s takes about 30 seconds. The Gen5 drives, 2 × 3.84 TB on GPU L40S and 2 × 7.68 TB on GPU RTX PRO, also hold several model versions side by side; point the Hugging Face cache at them with export HF_HOME=/srv/models.

Throughput: what to expect and how to measure it

Inference has two phases. Prefill processes the prompt in parallel and is compute-bound; long prompts raise the time to first token (TTFT). Decode produces one token per step for each sequence and, at small batch sizes, is limited by memory bandwidth, because every step reads the weights. That gives a ceiling for a single stream, not a measurement:

decode tokens/s per sequence ≤ memory bandwidth ÷ bytes read per token (≈ weight bytes)

For example, a 14B FP8 model (about 16 GB) on an L40S, with 864 GB/s of memory bandwidth, cannot exceed roughly 54 tokens/s for one stream, and real numbers are lower. NVIDIA lists about 1.6 TB/s for the RTX PRO 6000 Blackwell Server Edition.

Batching changes the picture. Each decode step reads the weights once for every sequence in the batch, so aggregate throughput rises with concurrency until compute or KV cache runs out, while each user's tokens per second fall. vLLM's continuous batching handles this automatically; your job is to choose the trade-off between total throughput and time per output token (TPOT) that your product can accept. Measure it with your own prompt lengths:

export OPENAI_API_KEY="$VLLM_API_KEY"   # sent as the bearer token
vllm bench serve --backend vllm --model Qwen/Qwen3-14B-FP8 \
  --base-url http://10.0.0.10:8000 \
  --dataset-name random --random-input-len 1024 --random-output-len 256 \
  --num-prompts 200 --max-concurrency 16 \
  --percentile-metrics ttft,tpot --metric-percentiles 95,99

Repeat at several --max-concurrency values and pick the highest one that still meets your p95 and p99 latency targets.

GPU L40S or GPU RTX PRO

Choose GPU L40S ($979/mo) when:

• your models are up to about 14B parameters in BF16 or FP8, or around 30B in INT4;
• you serve embeddings, rerankers, speech-to-text or image generation next to a mid-size chat model;
• you prefer to scale out across several smaller servers, or need Dallas as well as Ashburn, Amsterdam or Frankfurt.

Choose GPU RTX PRO ($2,199/mo) when:

• you serve 70B-class dense models, large MoE models, or long contexts for many concurrent users;
• you want two models side by side, one per GPU;
• you fine-tune with LoRA or QLoRA as well as serve, which benefits from 192 GB of VRAM;
• you want FP4 hardware available as your serving stack adopts it.

Per GB of VRAM, the RTX PRO server costs less: about $11.45 per GB per month, against about $20.40 for the L40S server.

Run it on FlexDedi

Start with GPU L40S ($979/mo) for models up to about 14B parameters, and GPU RTX PRO ($2,199/mo) for 70B-class models, large MoE models or long-context traffic. Both have 25 Gbps ports with unmetered traffic and Gen5 NVMe, and both are assembled to order and usually delivered the same day. Put application servers next to them on free private networking so the inference API never faces the internet, and apply the Linux security baseline to every host. More on the AI inference page.