For organizations running AI at scale, inference – not training – is becoming the dominant
workload. Deloitte projects inference will account for roughly two-thirds of all AI compute by 2026,
with the inference-optimized chip market exceeding $50 billion[1]. Yet most infrastructure
guidance still treats inference as a simplified extension of training. It is not.
Inference workloads are persistent, latency-sensitive, and directly tied to business SLAs. At the
hardware layer, memory bandwidth – not raw compute – is the primary performance bottleneck.
GPU selection must be evaluated in terms of High Bandwidth Memory (HBM) capacity, bandwidth
per watt, interconnect topology, and consolidation efficiency. Node design decisions around PCIe
versus NVLink, scale-up versus scale-out, and cooling strategy directly affect tail latency,
utilization rates, and long-term capital efficiency.
This white paper provides IT leaders with a hardware-first framework for designing inference
clusters. It examines the technical constraints that differentiate inference from training, analyzes
the cost trade-offs of on-premises versus cloud deployment, evaluates GPU alternatives beyond
NVIDIA, and addresses procurement strategies in an era of GPU scarcity and 36-to-52-week
lead times.
Every enterprise deploying large language models (LLMs) faces the same inflection point: the model works in the lab,
but deploying it at production scale – with predictable latency, at acceptable cost, under regulatory constraints – requires
a fundamentally different class of infrastructure.
Training is episodic. A team provisions a cluster, runs a job for days or weeks, and releases the resources. Inference is
continuous. It serves real users, processes real transactions, and runs against real inference latency SLAs – 24 hours a day,
7 days a week. For organizations operating under data sovereignty requirements, ultra-low-latency mandates, or regulatory
constraints like HIPAA, ITAR, or GDPR, inference cannot be treated as an afterthought delegated to whichever cloud
provider offers spare GPU capacity.
The hardware decisions an IT leader makes when designing an inference cluster – which GPU, how many, what
interconnect, what cooling, and from whom – will determine latency, throughput, operating cost, and strategic flexibility for
years to come. These decisions are also constrained by a supply chain that remains strained: data center GPU lead times
run 36 to 52 weeks[2], HBM is sold out across all three major suppliers through 2026[3], and enterprise buyers sit at the
bottom of NVIDIA’s allocation hierarchy[4].
This paper argues that inference clusters must be designed as mission-critical on-premises systems – purpose-built for
memory bandwidth, not borrowed from training environments. It provides the technical and economic framework IT leaders
need to make those design decisions with confidence.
The most consequential insight in inference hardware design is that LLM inference is memory-bandwidth-bound, not compute-bound[5]. Understanding why changes every downstream decision – from GPU selection to cooling strategy
Training processes massive batches of data through the model simultaneously, saturating the GPU’s arithmetic units.
Inference generates tokens one at a time. Each token requires loading the model’s full weight matrix from HBM – a
sequential, memory-dependent operation. On a Llama-2-7B model, the arithmetic intensity drops by roughly 1,000x between
the prefill phase (processing the input prompt in parallel) and the decode phase (generating output tokens sequentially)[6].
During decode, the GPU spends most of its time waiting for data to arrive from memory, not performing calculations.
A ~1,000x gap in arithmetic intensity between prefill and decode explains why inference hardware selection must prioritize
memory bandwidth over raw compute throughput. The widening gap between compute and bandwidth scaling means each
GPU generation pushes inference further into the memory-bound regime.
Table 1: Arithmetic intensity comparison – prefill vs. decode phases in LLM inference. Source:[5],[6]
Table 2: Compute and memory bandwidth scaling rates. Source:[7]
The vendor will prioritize Buyer B. The friction introduced by Buyer A effectively deprioritizes their order.
For SMBs, maintaining complex procurement rituals results in inflated lead times and, frequently, “no-bid”
responses from top-tier suppliers.
The result is that the traditional RFP process, intended to de-risk procurement, now introduces risk:
the risk of delay, the risk of technological misalignment, and the risk of capital being tied up in suboptimal,
inflexible infrastructure.
A widely used estimate: approximately 2 GB of GPU memory per billion parameters in FP16 precision[8]. A 70-billion-parameter model requires roughly 140 GB – more than a single NVIDIA H100 (80 GB) can hold. A 405-billion-parameter model requires approximately 810 GB, distributed across multiple GPUs.
The real pressure comes from the KV-cache – the growing memory structure that stores attention context for each active request. KV-cache memory scales linearly with both context length and the number of concurrent users. For Llama-2 70B, KV-cache consumes approximately 2.6 MB per token of context[8]. At 128K-token context, KV-cache alone can consume 39 GB or more – often exceeding the space allocated for model weights. As context windows grow toward one million tokens and beyond, HBM capacity becomes the binding constraint on what an inference node can serve.
The practical consequence: evaluate inference GPUs by memory bandwidth and capacity first, FLOPS second. The table below compares current-generation accelerators on the metrics that matter most for inference.
Table 3: Current-generation accelerator specifications for inference – HBM capacity, memory bandwidth, and thermal design power. Source:[8], [9]
The NVIDIA B200 delivers up to 4x higher per-GPU inference throughput versus the H200 at matched interactivity levels[9] – a gain driven primarily by doubling the memory bandwidth from 4.8 to 8 TB/s.
Note: B200 delivers up to 4x inference throughput versus H200 (vendor-reported)[9] – driven primarily by the bandwidth increase from 4.8 to 8 TB/s. Inference throughput scales with memory bandwidth for memory-bound decode workloads[5].
Inference clusters and training clusters share GPUs as a common building block, but the similarities end there. Designing
inference infrastructure based on training conventions leads to over-provisioned interconnects, underutilized compute,
and misaligned power budgets.
Table 4: Architectural differences between training and inference workloads. Source:[10], [11], [12]
Training requires every GPU to synchronize gradients with every other GPU at each step – demanding InfiniBand at 400
Gb/s with sub-microsecond latency in a fat-tree topology. Inference requests are largely independent: each can be served
by a single GPU or a small group, with no inter-request synchronization. Commodity Ethernet at 100 Gb/s is sufficient for
most deployments[11].
The exception: Models exceeding 400 billion parameters require tensor parallelism across multiple GPUs, which does demand high-bandwidth NVLink within the serving node. But even then, inference needs fast interconnect within a node, not between nodes – a critical distinction that allows scale-out over Ethernet for throughput[12].
Node-level architecture choices cascade into physical infrastructure requirements. On multi-chiplet GPUs like the AMD MI300X, Non-Uniform Memory Access (NUMA)-aware scheduling achieves up to 50% higher performance by keeping data local to the chiplet that processes it[13]. On multi-socket CPU servers hosting inference endpoints, cross-socket DMA traffic triggers latency spikes at the 99th percentile.
Power density is the headline constraint. Training racks draw 100–200+ kW; inference racks typically run 30–150 kW[14]. Below 40 kW per rack, air cooling remains feasible. Above 50 kW, liquid cooling is effectively mandatory[15]. The NVIDIA GB200 NVL72 rack – 72 GPUs connected by NVLink acting as one compute pool – draws 132 kW and requires direct-to-chip liquid cooling. IT leaders deploying dense inference nodes must plan cooling infrastructure alongside GPU procurement.
Table 5: Rack power density ranges and cooling requirements for AI deployments. Source:[14], [15]
Thresholds: Below 40 kW per rack, air cooling remains feasible[15]. Above 50 kW, liquid cooling is effectively mandatory[15]. IT leaders deploying dense inference nodes must plan cooling infrastructure alongside GPU procurement.
For persistent inference workloads, on-premises deployment delivers a compelling total cost of ownership (TCO)
advantage – but only when GPU utilization remains high enough to justify the capital investment.
A Lenovo TCO analysis finds that on-premises infrastructure breaks even against hyperscale cloud in as few as four months
for workloads sustaining above 20% GPU utilization[16]. Over a five-year lifecycle, an 8xH100 on-premises server costs
approximately $872,000 total (including power, cooling, and facilities), versus $4.3 million for equivalent continuous AWS
usage[16]. An independent academic study corroborates the direction: on-premises breakeven ranges from 0.3 to 3 months
for small models (24–32B parameters) and 3.8 to 34 months for medium models (70–120B)[17].
Self-hosted inference can deliver up to 18x lower cost per million tokens compared to commercial API services over a five-year lifecycle[18]. The crossover threshold is straightforward: if a system runs more than six hours per day in the cloud, on-premises becomes cheaper[16].
The utilization challenge
The complication: inference GPUs are chronically underutilized. Most AI accelerators run under 50% capacity for inference workloads[10]. Request volumes are bursty – high during business hours, low at night – and latency SLAs prevent aggressive batching that would improve utilization. A day/night strategy – inference by day, batch training or fine-tuning by night – can push effective utilization to 60–85%[19].
Combined software optimizations further improve economics. Continuous batching, quantization (which alone reduces
operational costs 60–70%), and speculative decoding can deliver up to 16x effective cost reduction when applied
together[18].
On-premises breakeven occurs in as few as four months at sustained GPU utilization above 20%[16]. Self-hosted inference delivers up to 18x lower cost per million tokens versus commercial API services[18]. TCO figures include power, cooling, and facilities for on-premises; continuous reserved pricing for cloud (vendor-reported)[16].
NVIDIA controls an estimated 80–92% of the AI accelerator market[20], but its dominance is no longer uncontested.
Enterprises evaluating GPUs for inference have viable alternatives – each with distinct trade-offs in HBM capacity, inference
performance, and software ecosystem maturity
The NVIDIA H200 (141 GB HBM3e, 4.8 TB/s) is the current workhorse for production inference. The B200 (192 GB, 8 TB/s)
doubles bandwidth and delivers 4x throughput improvement[9], but at 1,000–1,200 W TDP, it requires liquid cooling.
NVIDIA’s software ecosystem – CUDA®, TensorRT™, and now the Dynamo framework for disaggregated prefill/decode
serving[21] – remains the strongest competitive advantage.
AMD’s MI300X (192 GB HBM3, 5.3 TB/s) and MI325X (256 GB HBM3e, 6 TB/s) have moved beyond proof-of-concept. Microsoft Azure runs GPT-3.5 and GPT-4 inference on MI300X. Meta uses MI300X exclusively for Llama 405B inference. Oracle deploys 16,384-GPU MI300X superclusters[22]. ROCm 7.x has reached production maturity, and MI325X delivers 1.2–1.4x inference performance versus the H200 on key models[23]. AMD generated over $5 billion in accelerator revenue in 2024[24].
For latency-critical workloads, purpose-built inference ASICs offer compelling performance. Groq reports its LPU achieves
1,660 tokens per second on Llama 3 70B[25]. Cerebras claims 21x faster inference at one-third the cost of NVIDIA’s DGX
B200[26]. On-premises options exist (Groq’s GroqRack for regulated environments), though at multi-million-dollar price points.
For models under 13 billion parameters with latency tolerance in seconds rather than milliseconds, server CPUs – AMD EPYC or Intel Xeon with AMX extensions – offer cost-effective inference without GPU procurement bottlenecks[27].
Table 6: Inference accelerator comparison – memory bandwidth, reported performance, and ecosystem maturity. Source:[9], [22], [23], [25], [26], [28]
Note: GPU and ASIC architectures use fundamentally different memory hierarchies, so direct bandwidth comparisons across types are not meaningful. GPU performance claims are vendor-reported[9], [23]. ASIC claims are vendor-reported[25], [26]. Framework-level portability via PyTorch, JAX, and Triton is reducing CUDA lock-in for GPU-based accelerators[28].
CUDA remains the primary barrier to diversification. But AMD ROCm™ maturation, combined with framework-level abstractions in PyTorch, JAX, and Triton, has reduced switching costs enough that Microsoft, Meta, and Oracle now run production inference on non-NVIDIA hardware[28]. For IT leaders evaluating GPU procurement, maintaining software portability across vendors is no longer aspirational – it is a procurement risk management strategy.
Selecting the right GPU is only half the challenge. Securing it is the other.
Data center GPU lead times range from 36 to 52 weeks[2]. NVIDIA Blackwell and Rubin orders exceed $500 billion through
2026[29]. HBM is sold out across all three major producers – SK Hynix, Samsung, and Micron – through at least late 2026,
with the shortage projected to last through 2027[3]. U.S. tariffs enacted in 2025 added 20–40% to accelerator
component costs[30].
Critically, enterprise buyers sit at the bottom of NVIDIA’s allocation hierarchy – behind hyperscalers, sovereign AI initiatives, and OEMs[4]. Enterprises are not stepping into an open market; they are competing for residual allocation.
Vendor diversification is no longer optional. OpenAI committed 33 GW of compute across NVIDIA, AMD, Broadcom custom ASICs, and Oracle[29]. Meta is exploring Google TPU use alongside NVIDIA and AMD[29]. For enterprises, qualifying at least two accelerator platforms reduces single-vendor supply risk.
The secondary GPU market is emerging as a strategic procurement lever. Refurbished H100 GPUs trade at 60–85% of new pricing; A100 80 GB units at $12,000–$18,000 versus $25,000+ new[30]. HPC marketplaces aggregate inventory from multiple sellers – verified listings, real-time availability, and dedicated logistical support – offering reserved capacity at 40–70% savings versus on-demand cloud pricing[31].
OEM partners (Dell, HPE, Lenovo, Supermicro) maintain their own NVIDIA allocation and may provide faster access than direct procurement[4].
Lead times are approximate ranges from cited sources. Enterprise buyers sit at the bottom of NVIDIA’s allocation hierarchy – behind hyperscalers, sovereign AI initiatives, and OEMs[4]. Data center GPU lead times range from 36 to 52 weeks[2]. Secondary market pricing from[30].
GPU infrastructure is not a one-time purchase; it is a lifecycle to manage. Effective AI hardware lifecycle management requires a deliberate strategy for when to deploy, repurpose, and retire accelerators. The emerging best practice is the GPU Value Cascade: deploy new-generation GPUs for training (years 1–2), cascade them to production inference (years 3–4), then to batch processing and analytics (years 5–6)[32]. Hyperscalers have converged on 5–6-year depreciation cycles, and extending useful-life assumptions saved the industry approximately $18 billion collectively in 2024[33].
NVIDIA’s roadmap moves from Blackwell Ultra (H2 2025) to Vera Rubin (2026, targeting 5x inference performance over
Blackwell)[34]. AMD’s MI350X (CDNA 4, 2026) claims up to 35x inference improvement over MI300X using FP4 precision[35].
HBM4, with a 2 TB/s target bandwidth and a wider 2,048-bit interface, enters mass production mid-2026 – but is not
backward compatible with HBM3/3E systems, requiring new platform designs[36]. CXL 4.0 memory pooling enables
KV-cache offloading across shared memory pools, with research demonstrating up to 21.9x throughput improvement[37].
For IT leaders, the implication is clear: design for modularity, invest in software portability, and treat hardware procurement
as an ongoing strategic function – not a one-time capital event.
Traditional procurement – bilateral negotiations with a single vendor, 36-to-52-week lead times, and opaque pricing – is
not built for the velocity of AI infrastructure investment. HPC marketplace models address this gap by aggregating supply
from multiple sources, providing transparent pricing against verified listings, and supporting both new and pre-owned
hardware channels.
Nodestream’s HPC marketplace enables enterprises to procure AI-ready infrastructure from verified sellers with end-to-end
logistical support – and to liquidate or resell hardware through the secondary GPU market as workloads evolve. In a market
where GPU scarcity is the binding constraint, access to aggregated, transparent inventory is a strategic advantage.
AI inference is not a simplified version of training. It is a distinct, hardware-driven infrastructure challenge – persistent, latency-sensitive, and directly tied to business outcomes. The organizations that treat it as such will deploy faster, operate more efficiently, and maintain strategic flexibility as the hardware landscape evolves.
The key design principles for inference clusters are:
GPU scarcity and rising costs make every hardware decision consequential. IT leaders who
approach inference infrastructure with the same rigor they apply to any mission-critical
system – grounded in data, diversified in supply, and designed for the long term – will be best
positioned to deliver on AI’s promise.
