Training vs inference: what changes on the server
NVIDIA describes training as adjusting a network's weights from data and feedback, and inference as a trained model making predictions on new data. For the server, the difference is mostly memory. During inference the GPU holds the model's weights and a working cache. During training it also holds a gradient for every weight, the optimiser's state for every weight, and the activations from each forward pass, which is why the same model needs several times more memory to train than to serve.
The two jobs also run differently. Training is a batch job: it runs flat out for hours or days, then stops. Inference is a service that runs all day, and how quickly it answers each user depends heavily on how fast the card reads its own memory. NVIDIA notes that LLM inference is often limited by memory bandwidth rather than raw compute.
| Inference (serving a model) | Training or fine-tuning | |
|---|---|---|
| Held in GPU memory | Weights, KV cache, runtime overhead | Weights, gradients, optimiser states, activations |
| Memory grows with | Context length and concurrent requests | Model size, batch size and sequence length |
| Usually limited by | Memory capacity, then memory bandwidth | Memory capacity, then compute |
| Runs as | A service, all day | A job with an end |
How much GPU memory do you need to run an LLM?
Add up three things: the weights, the KV cache and some overhead.
Weights are parameters multiplied by bytes per parameter. At 16-bit (fp16 or bf16) each parameter takes 2 bytes, so a 7B model needs about 14 GB for its weights alone, by NVIDIA's arithmetic. Hugging Face gives the same picture for an 8B model: about 16 GB in bf16, about 8 GB at 8-bit and about 4–5 GB at 4-bit. Quantising to fewer bits saves memory but costs some accuracy, so test a quantised model on your own prompts before you size hardware around it.
The KV cache stores attention keys and values for every token being processed. Its size is batch size × sequence length × 2 × number of layers × hidden size × bytes per value, so it grows linearly with both the number of requests in flight and the context length. Doubling either roughly doubles the cache. It is easy to leave out of a sizing estimate, and it is the part that grows once real users arrive.
Overhead. Hugging Face's memory estimator covers only what it takes to load a model, and its docs say to expect up to about 20% more for inference.
So the working rule is: weights, plus a KV cache sized for your longest context and busiest moment, plus about 20%. If that total does not fit in one card's memory, you need a bigger card, a smaller or more heavily quantised model, or several GPUs.
How much GPU memory does training need?
Training costs far more per parameter. Hugging Face's breakdown for mixed-precision training with the Adam optimiser comes to about 18 bytes per parameter before activations:
- Weights: 6 bytes, an fp16 copy plus an fp32 copy.
- Optimiser states: 8 bytes for Adam.
- Gradients: 4 bytes.
- Activations on top, growing with batch size and sequence length.
At that rate a 7B model needs about 126 GB before a single activation is stored, which is more than any one card in this guide holds. Hugging Face's own example is a 4B model trained in mixed precision at batch size 16, which needs roughly 85 GB of GPU memory. Full training of anything mid-sized is therefore a multi-GPU job on 80 GB cards.
Most teams do not need full training. Parameter-efficient fine-tuning (PEFT) methods such as LoRA train only a small number of extra parameters, which Hugging Face says cuts compute and storage costs significantly. Because the base weights are frozen, gradients and optimiser states are kept only for that small set, which is why a LoRA fine-tune fits on far less memory than a full one.
LLM VRAM requirements by model size
These are the rough working figures we size GPU builds against, including some overhead. Add headroom for the KV cache on top, especially for long contexts or many simultaneous users.
| Model size | Inference, 8-bit | Inference, 16-bit | LoRA fine-tune | Full fine-tune |
|---|---|---|---|---|
| 7B | ~8 GB | ~16 GB | ~24 GB | ~80 GB |
| 13B | ~14 GB | ~28 GB | ~40 GB | Multi-GPU |
| 70B | ~70 GB | Multi-GPU | Multi-GPU | Multi-GPU |
Treat the full fine-tune column as a floor. A plain mixed-precision Adam run, at 18 bytes per parameter, needs more than ~80 GB for a 7B model, so plan for memory-saving settings or a second card.
Which GPU for AI: 24 GB, 48 GB or 80 GB?
Pick the memory class first, then the card. These are the cards that come up most often for AI servers, with memory figures from NVIDIA's own pages:
| Card | Memory | Worth knowing | Typical AI job |
|---|---|---|---|
| RTX 4090 | 24 GB | GeForce card | Serving a 7B–13B model, Stable Diffusion |
| L4 | 24 GB | Data-centre card, 72 W maximum TDP | Serving a 7B–13B model, transcoding |
| L40S | 48 GB GDDR6 with ECC | Data-centre card | LoRA and full fine-tunes of smaller models, larger batches |
| RTX A6000 | 48 GB GDDR6 with ECC | Two cards can be joined by an NVLink bridge for 96 GB combined | LoRA and full fine-tunes of smaller models, larger batches |
| A100 | 80 GB HBM2e | 1,935 GB/s (PCIe) or 2,039 GB/s (SXM) memory bandwidth | 70B models, multi-GPU training |
| H100 SXM | 80 GB | 3.35 TB/s memory bandwidth | 70B models, multi-GPU training |
For one 7B or 13B model serving a modest number of users, a 24 GB card is usually the sensible starting point. Move to 48 GB when you fine-tune, need larger batches or want long contexts on a 13B model. Pay for 80 GB when the model itself needs it, as a 70B model does, or when training throughput matters more than the monthly figure. Because inference is often bandwidth-bound, the HBM memory on the A100 and H100 also helps serve large models to many users at once.
When one GPU is not enough
A 70B model at 16-bit needs about 140 GB for its weights alone (70 billion × 2 bytes), so it cannot be served from any single 80 GB card. You have two choices: quantise it to 8-bit, which brings it to about 70 GB but leaves little room for the KV cache on an 80 GB card, or split it across GPUs. Serving engines such as vLLM can split a model across several GPUs with tensor parallelism when it does not fit on one card.
When cards share work that closely, the link between them matters. NVLink is NVIDIA's direct GPU-to-GPU interconnect inside a server. Bandwidth depends on the generation: NVIDIA lists 900 GB/s per GPU for fourth-generation NVLink and 3 TB/s for the sixth, while two RTX A6000 cards joined by an NVLink bridge get up to 112 GB/s. For multi-GPU training or tensor-parallel inference, ask whether the cards in a build are NVLinked, not just how many there are.
On the software side, the NVIDIA Container Toolkit lets you build and run GPU-accelerated containers, so a framework and its CUDA libraries can ship as one image rather than being installed by hand on the server.
Can you run AI inference on a CPU server?
For small models, often yes. llama.cpp runs LLM inference on a wide range of hardware, uses AVX, AVX2, AVX512 and AMX on x86 CPUs, and supports quantisation from 1.5-bit to 8-bit. It can also split a model between CPU and GPU when the model is larger than the card's memory. A small quantised model on a CPU server is practical for internal tools, document processing and low-traffic assistants. It will not match a GPU for many simultaneous users, and it is not a way to train.
If you go this way, check that the server exposes the CPU's instruction set. On our VDS plans the server sees the host's own CPU model with its full instruction set (such as AVX2), on dedicated cores that are never shared; VDS 8 has 8 dedicated cores and 32 GB of RAM at ₹6,908 (US$78.50)/mo for each 30-day period, excluding GST, and is set up after an enquiry for now. A KVM VPS uses fair-share vCPUs on a generic virtual CPU model, which is fine for experiments but less predictable for sustained inference.
Monthly GPU server or hourly cloud GPU?
Hourly cloud GPUs from the large providers are the right tool for occasional experiments: start an instance, run the job, delete it. They are a poor fit for a model that serves users all day, because the hours add up when the card never stops.
The crossover is usually around 200–300 hours a month. Past that, a dedicated card rented by the month normally costs less than the same hours bought one at a time, and it stays yours when a cloud region runs out of capacity. A month has about 730 hours, so an inference service that runs around the clock is well past the crossover, while a training run of a few days a month may not be.
A dedicated GPU server also gives you the whole card: no time-slicing and no shared VRAM, which matters when you have sized memory tightly.
Why host an AI server in India
- Data residency. If your training data or users' prompts must stay in the country, an overseas GPU cloud may be ruled out. Services in our Mumbai (BOM-1) and Delhi NCR (DEL-1) data centres keep customer data inside India, which simplifies compliance under the DPDP Act, 2023.
- Latency for inference. A server in Singapore is typically 55–70 ms from Indian cities and one on the US East coast 220–260 ms. For a chat interface that streams tokens to Indian users, that delay is added to every request.
- Billing in rupees. No forex markup on a card, and no bill that moves with the exchange rate.
Both our facilities are Tier III, with N+1 power and precision cooling. See data centres and latency for details.
Renting an AI server from Bigdomainhost
Our GPU servers are bare-metal NVIDIA servers in Mumbai and Delhi, in three classes: inference (RTX 4090 / L4 class, 24 GB), fine-tuning (L40S / A6000 class, 48 GB) and training (A100 / H100 class, 80 GB). Multi-GPU builds, NVLink and specific CUDA versions are quoted per build.
There is no price list, because card supply and pricing move week to week. Each build is quoted within one working day, in rupees with no forex markup and excluding 18% GST. Monthly terms are possible; longer terms price considerably better because the card is bought for you. A build we have the parts for is provisioned in one to three working days, and if a card has to be sourced we tell you before you pay. Before handover we install Ubuntu LTS or the distribution you name, the NVIDIA driver, the CUDA toolkit version you ask for and, if you want it, Docker with the NVIDIA container toolkit, then check the card with nvidia-smi and a short burn-in. There is no refund once a build is provisioned, so ask every sizing question first.
To get an accurate quote, send:
- The model and its size, and the precision or quantisation you plan to run.
- Whether it is training, fine-tuning (LoRA or full) or serving.
- The batch size, the longest context and, for serving, how many requests at once.
- The CUDA version or framework your code expects.
- The term you have in mind, and whether data must stay in India.
If the honest answer is that a CPU server will do, we will say so.
Memory figures are estimates for planning; real use depends on the framework, settings and workload. Check a model's own documentation before you buy hardware for it.
Sources (checked )
- NVIDIA: What's the Difference Between Deep Learning Training and Inference?
- NVIDIA: Mastering LLM Techniques: Inference Optimization
- Hugging Face: GPU memory usage (Transformers docs)
- Hugging Face: Overview - Quantization (Transformers docs)
- Hugging Face: Model memory estimator (Accelerate docs)
- Hugging Face: PEFT
- NVIDIA: GeForce RTX 4090
- NVIDIA: L4 Tensor Core GPU
- NVIDIA: L40S GPU
- NVIDIA: NVIDIA RTX A6000
- NVIDIA: NVIDIA A100
- NVIDIA: H100 GPU
- NVIDIA: NVLink and NVLink Switch
- vLLM: Conserving Memory (vLLM docs)
- GitHub: NVIDIA/nvidia-container-toolkit
- GitHub: ggml-org/llama.cpp: LLM inference in C/C++