Flexible parallel compute
Streaming multiprocessors, Tensor Cores, caches and hardware scheduling make GPUs adaptable to changing graphs, custom kernels and mixed workloads. The software ecosystem is part of the product.
CUDA / ROCm
AI infrastructure · updated September 2026
AI hardware is a workload decision, not a benchmark winner.
GPUs, TPUs and specialized accelerators trade flexibility, memory, compiler assumptions, network topology and economics differently. The useful question is not “which chip is fastest?” but “which system delivers the required quality and SLO at the lowest fully loaded cost?”
Decision model
1. Workload physics
prefilldecodetrainingMoEagent loops
Measure the bottleneck
Memory & interconnect
HBM, SRAM, scale-up, scale-out
Software & portability
CUDA, ROCm, JAX/XLA, compiler maturity
Economics
Cost per successful task, not chip-hour alone
Output
Workload-matched accelerator choice
Fast routing guide
Start with the constraint you cannot ignore.
These are starting hypotheses, not procurement conclusions. Benchmark the exact model, context length, batch/concurrency mix, serving engine and quality target before committing.
Mixed enterprise AI: NVIDIA remains the lowest-friction starting point when PyTorch/CUDA compatibility, third-party kernels and broad deployment tooling dominate. Also qualify AMD when memory capacity and open-stack economics matter, and TPU when the workload fits Google Cloud/JAX/XLA well.
Current landscape
Representative datacenter accelerators in 2026.
The table deliberately mixes current and announced platforms only to show architectural direction. Peak FLOPS across vendors are not directly comparable unless precision, sparsity and system configuration are normalized.
| Platform | Status | Memory | Bandwidth | Architecture / software | Best starting fit |
|---|---|---|---|---|---|
| NVIDIA H200 | Available | 141 GB HBM3e | 4.8 TB/s | Hopper · CUDA | Broad enterprise inference/fine-tuning |
| NVIDIA B200 | Available | 180 GB HBM3e | Up to 8 TB/s | Blackwell · CUDA | High-throughput training & inference |
| NVIDIA B300 / GB300 | Available | Up to 288 GB HBM3e | Up to 8 TB/s per GPU | Blackwell Ultra · NVLink rack scale | Reasoning, long context, high concurrency |
| NVIDIA Rubin | 2026 ramp | 288 GB HBM4 | Up to 22 TB/s | Rubin · NVLink 6 | Next-gen agentic/reasoning AI |
| Google Trillium (v6e) | Available | 32 GiB HBM | 1.638 TB/s | TPU MXU · JAX/XLA, PyTorch/XLA | Cost-oriented training, fine-tuning & serving |
| Google Ironwood (TPU7x) | Available | 192 GiB HBM | 7.38 TB/s | 2 TensorCores + 4 SparseCores · 3D torus | Large-scale training & memory-heavy TPU workloads |
| Google TPU 8t | Coming soon | 216 GB HBM + 128 MB SRAM | 6.528 TB/s | 3D torus · SparseCore · Virgo scale-out | Massive pre-training & embedding-heavy systems |
| Google TPU 8i | Coming soon | 288 GB HBM + 384 MB SRAM | 8.601 TB/s | Boardfly · Collectives Acceleration Engine | Reasoning, post-training & serving |
| AMD MI355X | Available | 288 GB HBM3E | 8 TB/s | CDNA4 · ROCm | Large-memory open-stack AI |
| AMD MI455X | 2026 deployments | 432 GB HBM4 | 23.3 TB/s | CDNA5 · Helios / ROCm | Frontier rack-scale training & inference |
| Cerebras WSE-3 | Available | 44 GB on-chip SRAM | 21 PB/s on-chip | Wafer-scale spatial compute | Very large models with simplified model parallelism |
| Groq LPU | Available service | SRAM-first | Architecture-specific | Statically scheduled deterministic execution | Latency-sensitive inference |
Do not rank this table by one column. A chip with more memory or peak FLOPS can still lose on latency, utilization, framework support, networking, availability or cost per successful request.
Architecture
Three different ways to fight data movement.
Compiler-shaped matrix systems
TPUs organize matrix engines, vector units, HBM and inter-chip fabrics around compiler-managed tensor workloads. The system-level advantage appears when the model maps cleanly to the TPU stack and scale topology.
JAX / XLA / PyTorch
Bring more data onto the compute fabric
Architectures such as Cerebras and Groq attack memory movement differently: larger on-chip memory, spatial execution or static scheduling. They can be compelling for specific latency or scale regimes, but require workload-specific qualification.
Specialized compilers
LLM workload physics
Prefill and decode stress different resources.
The same accelerator may behave differently across long-context prefill, batch-1 decode, high-concurrency serving, fine-tuning and training.
Often compute-intensive
Prompt tokens can be processed in parallel, creating matrix-matrix work that can drive high arithmetic intensity. Peak tensor compute and scale-up efficiency matter, but long prompts can still pressure memory and attention bandwidth.
Often memory- and latency-sensitive
Autoregressive generation exposes weight/KV-cache movement, synchronization and tail-latency costs. Larger HBM, faster bandwidth, on-chip SRAM and serving software can matter more than peak FLOPS.
Modern serving increasingly disaggregates prefill and decode. Evaluate whether separate resource pools, different accelerators or different parallelism strategies improve end-to-end SLO and economics.
Concept demo
Why systolic arrays reuse data.
This is a conceptual animation—not a cycle-accurate TPU simulator. It simply illustrates data reuse as values move through neighboring processing elements.
Cycle: 0
What the animation represents
1
Load tiles
Weights/activations are arranged to maximize local reuse.
2
Neighbor-to-neighbor flow
Values move through the array rather than repeatedly returning to far memory.
3
Accumulate results
Multiply-accumulate work proceeds in a regular dataflow that a compiler can schedule efficiently.
Benchmark protocol
Measure the system, not the spec sheet.
Quality
Hold output quality fixed
Same checkpoint, quantization tolerance and task-quality threshold.
Performance
Measure TTFT, TPOT & throughput
Include p50/p95/p99 and long-context/concurrency behavior.
Utilization
Measure real goodput
Include compilation, padding, network stalls, retries and idle capacity.
Economics
Use fully loaded cost
Compute, CPU, network, storage, engineering, migration and failure recovery.
DataKnobs connection: treat accelerator family, precision, batch size, context length, serving engine, parallelism, quantization and routing as explicit Knobs. Compare controlled variants on the same workload rather than making a permanent vendor choice from one benchmark.
Primary sources
Current first-party references used for the rebuild.
FAQ
Common architecture questions
Is a TPU always more efficient than a GPU?
No. Efficiency depends on the workload, compiler mapping, batch/concurrency profile, interconnect, software maturity and utilization. TPU advantages are strongest when the workload fits its system design and software stack.
Why is memory bandwidth so important for LLM inference?
Autoregressive decode repeatedly accesses model state and KV cache while doing relatively little work per byte moved. That can make decode memory- or communication-bound even when peak tensor compute is very high.
Should I compare accelerators by FLOPS per dollar?
Only as a coarse screening metric. Production decisions should compare cost per successful task at the required quality, latency and reliability target.