Power density is the new constraint
A traditional enterprise rack draws 5-8 kW. A single NVIDIA H100 or H200 GPU server draws 6-10 kW, and a fully populated AI training rack can exceed 120 kW. Legacy raised-floor data centers designed for 4 kW/rack cannot host these workloads without a full electrical and mechanical retrofit.
Plan for 415/240V three-phase distribution at the rack, high-amperage busway (Starline-style) overhead delivery, and rack PDUs rated for the target density. Redundancy at these power levels is expensive — 2N distribution for a 100 kW rack requires two separate 125 A feeds. Model total cost of ownership including UPS, generator and cooling before committing to a topology.
Air cooling stops working at 40 kW/rack
Above roughly 30-40 kW per rack, air cooling becomes physically impractical — the CFM required exceeds what perforated tiles and containment can deliver, and hot-spot temperatures cause GPU thermal throttling that destroys training throughput.
The industry has settled on two options: rear-door heat exchangers (RDHx) for 30-70 kW/rack, and direct-to-chip liquid cooling for anything above. Both require chilled water infrastructure to the rack — CDUs, secondary loops with leak detection, and PUE-optimized cooling towers or dry coolers. Immersion cooling exists but has not seen enterprise adoption at scale.
The fabric is the workload
Distributed AI training is bandwidth-bound and latency-sensitive. All-reduce operations across a GPU cluster generate traffic patterns that break traditional leaf-spine Ethernet designs. Two fabric options dominate: InfiniBand NDR (400 Gb/s) for tightly-coupled training clusters, and 400/800 GbE with RoCEv2 and lossless queuing (PFC, ECN) for organizations that prefer Ethernet operations.
Both approaches require careful design — a single congested link can slow an entire training job by an order of magnitude. Rail-optimized topologies, adaptive routing and hardware telemetry are table stakes at this scale.
What to do if you have legacy space
Not every organization needs to retrofit. For most enterprise AI use cases — fine-tuning, inference, RAG pipelines — GPU capacity from a cloud or colocation provider is faster to deploy and cheaper in year one. Reserve on-premises AI infrastructure for workloads with data-residency constraints, sustained utilization above 60%, or latency requirements that cloud egress cannot meet.
Key takeaways
- Plan for 60-120 kW/rack — legacy 4 kW designs will not host modern GPU nodes.
- Liquid cooling (RDHx or direct-to-chip) is required above 40 kW/rack.
- Choose InfiniBand or lossless Ethernet based on operations expertise, not marketing.
- Colocation or cloud is often the right answer for enterprise AI in year one.
