Serve AI models on dedicated NVIDIA GPUs

Start with memory: a model that doesn't fit in VRAM won't run at useful speed. Weights take roughly parameters × bytes per parameter: 2 bytes in BF16, 1 byte in FP8 and about 0.5 byte at 4-bit. These estimates cover the weights alone:

Then budget the KV cache, which grows with context length and concurrency. For a Llama-style 70B model with grouped-query attention, each token of context costs about 0.33 MB in FP16, so 32 concurrent sessions of 8,000 tokens need roughly 85 GB on top of the weights. Long-context, high-concurrency services often run out of KV cache before they run out of compute, so keep 20 to 40% of VRAM free after loading the weights.

The GPU RTX PRO has two RTX PRO 6000 Blackwell cards on PCIe 5.0, without NVLink between them. That shapes how you deploy:

• The model fits on one card: run two independent replicas, one per GPU, behind a load balancer. You get roughly twice the throughput and no inter-GPU traffic.
• The model needs both cards: use tensor parallelism of 2 in vLLM or SGLang. It is a common setup for inference over PCIe, but each GPU does less useful work than it would in a separate replica.
• Mixed workloads: pin the LLM to one card and use the other for embeddings, reranking, speech or image models.

The GPU L40S is a single 48 GB card, so scale it out: several servers behind a load balancer, with internal traffic on a free private VLAN. Both GPU servers are assembled to order and usually delivered the same day, in Ashburn, Amsterdam and Frankfurt, and the GPU L40S in Dallas too.

• Drivers and CUDA: install the NVIDIA driver and the NVIDIA Container Toolkit after deployment, and pin their versions. Blackwell GPUs need CUDA 12.8 or newer, so rebuild images based on older CUDA or PyTorch releases.
• Weights on local NVMe: keep models on the NVMe Gen5 drives and pre-pull them, so a restart reloads from local disk instead of a remote registry.
• Batching: continuous batching in vLLM, SGLang or TensorRT-LLM is usually the biggest throughput lever. Tune the maximum number of concurrent sequences against your latency target.
• Quantization: FP8 halves weight memory compared with BF16, often with little quality loss. Validate on your own evaluation set before switching.
• Exposure: keep inference ports on the private VLAN or behind an authenticated gateway. Always-on DDoS protection filters floods at the network edge, but it doesn't replace authentication.

Servers are unmanaged: you get root access, IPMI and the KVM console, and you control the whole software stack, from the driver branch to the serving framework.