NVGPU

Compute, considered.

How to choose and test a GPU for AI training

A reproducible decision process for memory, hardware variants, distributed training and total job cost.

By NVGPU · Substantively reviewed September 26, 2026 · Published documentation and calculations; no NVGPU hardware benchmark.

Which GPU is best for training?

There is no universal best GPU. First eliminate configurations that cannot fit your model and training state. Then run the same workload on the remaining configurations and compare time and cost to the same quality target. A published TFLOPS number or a low hourly rate cannot answer that alone.

Budget memory before comparing prices

Training memory includes model weights, gradients, optimizer state, activations and temporary buffers. Weight storage alone is approximately parameter count × bytes per parameter. A hypothetical one-billion-parameter model at two bytes per parameter needs about 2 GB for weights alone; that is not its complete training requirement.

Batch size, sequence length, optimizer, precision and activation checkpointing change the budget. Profile a representative step and leave headroom. The PyTorch memory-management documentation explains allocated versus reserved memory.

Use separate A100 40GB and 80GB rows. H100 NVL 94GB and H100 80GB are distinct variants. Rental partitions must be compared using their reported VRAM rather than the name of the full physical GPU.

Check the complete training system

Multiple GPUs do not automatically expose one shared pool of VRAM. Distributed training software must divide or replicate the workload, and communication can limit scaling. Confirm GPU topology, NVLink or PCIe paths, host RAM, CPU resources, storage throughput and network fabric on the exact rented instance.

NVIDIA lists different reference configurations for H100 SXM and NVL. That does not verify any provider's cloud topology. Use our sourced reference table, then confirm the instance details.

A comparison you can reproduce

  1. Fix the model revision, dataset split, target quality, framework, driver, CUDA version, precision and optimizer.
  2. Record GPU model and variant, count, memory, interconnect, region, CPU, RAM, disk and pricing type.
  3. Run a warm-up and representative training interval. Record wall time, examples or tokens processed, peak memory and quality metrics. Repeat enough times to describe variation; include failures.
  4. Measure billed setup, download, idle, checkpoint and restart time separately from steady-state steps.
  5. Multiply the full-instance rate by billed runtime and add actual storage and transfer charges. Publish the conditions alongside the result.

This is a proposed protocol, not a report of a test NVGPU has performed.

When does spot make sense?

Use interruptible capacity only if recovery is implemented and tested. Savings depend on the rate, lost work, restart delay and storage costs. A long job without durable checkpoints can cost more after interruptions. Check the provider's own termination behavior, not an assumed percentage discount. AWS documents Spot interruption behavior.

Build your shortlist

Compare H100, A100, L40S and RTX 4090 only after checking memory and software requirements. Their prices are workload inputs, not benchmark rankings. For a specific budget calculation, use the cost-planning guide.