GPU (AI Accelerators)

What is a GPU in the AI context?

A GPU executes thousands of operations in parallel — originally for graphics, now the workhorse of AI because neural networks are mostly giant matrix multiplications. NVIDIA dominates (H100/H200/Blackwell class, with the CUDA software moat), while alternatives include Google TPUs, AWS Trainium/Inferentia, and AMD Instinct. “Accelerator” is the vendor-neutral term.

The two numbers that matter

For LLM work, a GPU is characterized by memory capacity (can the model + KV cache fit? an 80 GB card holds a 70B model only at 8-bit) and memory bandwidth (how fast weights stream — this, not raw FLOPS, bounds tokens/sec for inference). Training cares about interconnect too (NVLink, InfiniBand) since models span many GPUs.

How teams actually buy compute

• API models — someone else’s GPUs; zero GPU knowledge required. The right default.
• Cloud GPU rental — $1–10+/hr per card from hyperscalers or neo-clouds (CoreWeave, Lambda); spot/preemptible cuts 50–70% for interruptible work.
• Serverless GPU (Modal, RunPod, Cloud Run GPU) — per-second billing with cold starts; good for spiky inference.
• Buying hardware — justified only by sustained multi-year utilization; capacity planning and depreciation become your problem.

Cost reality

GPU spend is dominated by utilization, not price: a rented H100 idling at 15% utilization costs ~7× per useful token what a batched, saturated one does. Before shopping for cheaper cards, measure utilization — batching, quantization, and right-sizing models routinely beat hardware negotiations.

What people get wrong

• Sizing by FLOPS. LLM inference is memory-bound; a card with more bandwidth beats one with more compute for serving.
• Ignoring the software stack. Non-NVIDIA hardware can be cheaper per hour but cost more in engineering time if your stack assumes CUDA.
• Reserving before profiling. Committing to a year of capacity based on projected (not measured) load is how AI infra budgets die.

Why GPUs Specifically, Not CPUs

GPUs are suited to AI workloads because neural network training and inference are dominated by matrix multiplication — an operation that can be broken into many small, independent calculations performed simultaneously. A CPU has a small number of powerful cores optimized for sequential, general-purpose logic, while a GPU has thousands of simpler cores optimized specifically for exactly this kind of massively parallel arithmetic, making it dramatically more efficient for the matrix operations that make up the bulk of a transformer’s computation. Specialized tensor cores, first introduced by NVIDIA and now common across AI-focused GPU generations, further accelerate the specific lower-precision matrix multiplications (FP16, BF16, FP8) that dominate modern AI workloads, on top of the GPU’s already parallel general architecture.