The AI Fabric Wars: Ethernet vs. InfiniBand for Large-Scale AI Networks

Table of Contents

Frequently Asked Questions - FAQs

Why AI Pushes Networks to the Edge

Artificial intelligence has upended traditional networking assumptions. Large-scale AI models, especially in deep learning and generative AI, generate traffic patterns and performance demands that legacy data center networks simply weren’t designed to handle.

Where traditional enterprise networks mostly move client-to-server “north-south” traffic, AI networks are dominated by “east-west” flows: enormous volumes of data moving laterally among thousands of GPUs, often in highly synchronized bursts. This new reality has sparked a fierce debate between two competing approaches to building AI networks: standards-based Ethernet and specialized InfiniBand.

At stake is more than just network speed. The network has become the lynchpin of AI economics, directly impacting the cost and ROI of multi-million-dollar GPU clusters.

The True Cost Metric: Job Completion Time

In AI infrastructure, Job Completion Time (JCT) reigns supreme. It measures the total time it takes for a distributed AI training job, spanning thousands of GPUs, to finish.

Consider the economics:

A single AI server with eight high-end GPUs can cost over $400,000.
Training a large language model (LLM) might burn through more than 1.7 million GPU-hours.

Because AI workloads are tightly synchronized, the entire cluster moves only as fast as the slowest GPU. If a single GPU is delayed waiting for data, due to network congestion, latency spikes, or packet loss, every other GPU sits idle. Those idle seconds become a massive financial drain.

Studies show that 20% to 50% of total AI job time is spent on inter-GPU data movement. Even small network improvements—an extra 8-10% throughput—can translate into millions saved by shaving time off JCT.

What Makes AI Traffic Unique

AI’s traffic patterns are fundamentally different from traditional applications:

Elephant Flows: Rather than many small, random flows, AI generates a few massive, long-lived flows during processes like gradient exchange in distributed training.
Incast Congestion: Many nodes sending large data bursts simultaneously to a single destination can overwhelm switch buffers, causing queuing delays or packet loss.
Low Entropy: Many AI flows share identical source/destination addresses, making it harder for traditional load-balancing algorithms to spread the traffic across multiple links.

Traditional enterprise networks, built around shallow buffers, basic load balancing, and reactive congestion management, struggle under these conditions. AI workloads need a network that’s not just fast, but lossless.

Why Lossless Matters So Much

In typical enterprise apps, occasional packet loss isn’t catastrophic. TCP simply retransmits, and the application carries on. AI workloads can’t tolerate this.

Distributed AI training uses collective communication operations like All-Reduce, which rely on perfect data delivery. A single dropped packet can force retransmission of huge data blocks, increasing JCT. In some cases, a packet drop might force the entire training job to restart from the last checkpoint, an expensive setback.

InfiniBand: A Proactive HPC Fabric

InfiniBand originated in the world of supercomputing. It’s engineered for deterministic, lossless performance.

Centralized Subnet Manager

InfiniBand operates like an SDN. A Subnet Manager discovers the fabric topology, programs all routing tables, and manages congestion controls from a single point. This centralized control removes the broadcast storms and distributed complexity found in traditional Ethernet.

Credit-Based Flow Control

InfiniBand’s standout feature is credit-based flow control.
    •    A sender verifies whether the downstream device has enough buffer space (credits) before transmitting packets.
    •    No credits? The sender waits.
    •    This guarantees that congestion is avoided before it happens.

Result: lossless operation and low, predictable latency.

Native RDMA and In-Network Computing

InfiniBand was designed with RDMA (Remote Direct Memory Access) from day one. It allows GPUs to move data directly between memory spaces, skipping the CPU and kernel. This reduces latency and frees CPU cycles for compute tasks.

InfiniBand also pushes computation into the network itself. For example, NVIDIA’s SHARP technology enables switches to perform collective operations, such as All-Reduce, directly, thereby reducing traffic and accelerating training.

Ethernet: Evolved for AI

Ethernet’s journey into AI networking is one of adaptation rather than reinvention. While originally a best-effort protocol, Ethernet has evolved dramatically through the introduction of new technologies.

The key to making Ethernet viable for AI is the RoCEv2 stack:

RoCEv2: RDMA Over Ethernet

RoCEv2 allows RDMA traffic to ride over standard IP networks. It wraps InfiniBand transport packets inside UDP/IP headers, letting data traverse large Layer 3 networks while retaining RDMA’s low-latency advantages.

Priority Flow Control (PFC)

To create lossless Ethernet, PFC is used:

It pauses only specific traffic classes when buffers fill up rather than freezing all traffic.
For AI, RDMA traffic sits in a dedicated lossless class.

Explicit Congestion Notification (ECN)

PFC can be blunt and dangerous if overused. To avoid this, Ethernet fabrics employ ECN:

Instead of dropping packets, ECN marks packets as congestion builds.
Receivers notify senders to reduce transmission rates.

DCQCN

The Data Center Quantized Congestion Notification algorithm manages how senders throttle their rates based on ECN signals. This reactive strategy tries to stabilize the fabric and avoid resorting to disruptive PFC pauses.

The Challenges of Ethernet for AI

Ethernet’s “lossless” behavior is an engineered solution, not a native feature:

Tuning Complexity: Building a stable RoCEv2 fabric demands careful tuning—PFC thresholds, buffer sizes, QoS settings, and ECN parameters all must be dialed in precisely.
PFC Storms: Excessive PFC signaling can create cascading pauses across the network, sometimes locking up entire segments.
Buffer Dependency: PFC shifts congestion upstream, making deep buffers a critical hardware requirement.

Nonetheless, Ethernet’s strengths lie in its open standards and massive vendor ecosystem, which drive down costs and foster innovation.

Vendor Strategies: NVIDIA vs. Arista

The Ethernet vs. InfiniBand debate is as much a competition of business models as technologies.

NVIDIA: Vertical Integration

NVIDIA offers:

Quantum-2 InfiniBand: High-performance, native lossless fabric for pure HPC and AI clusters.
Spectrum-X Ethernet: Vertically integrated Ethernet solution pairing Spectrum-4 switches with BlueField-3 DPUs. Delivers Ethernet performance with InfiniBand-like features like adaptive routing and congestion management.

NVIDIA’s pitch: unified, turnkey performance—but with proprietary integration and potential vendor lock-in.

Arista: Open Standards and Software Intelligence

Arista’s strategy:

Builds on merchant silicon from Broadcom (Jericho3-AI for spines, Tomahawk 5 for leaves).
Offers platforms like the 7800R4 AI Spine and 7700R4 DES, capable of scaling from medium clusters to massive AI factories.
Software: EOS and CloudVision brings:
Cluster Load Balancing (CLB): Intelligent, RDMA-aware traffic engineering to avoid hotspots.
AI Analyzer: Deep observability to correlate network behavior with AI job metrics.

Arista’s pitch: performance with open ecosystems, lower TCO, and freedom from lock-in.

Real-World Performance

Perhaps the biggest blow to the old InfiniBand supremacy myth came from Meta’s Llama 3 deployment. Meta built twin AI clusters—one on InfiniBand, the other on Arista Ethernet. Meta publicly reported no network bottlenecks on either fabric. That’s a profound endorsement of modern Ethernet’s viability.

Lab tests (e.g., MLPerf benchmarks) reinforce the point:

Differences in job times between InfiniBand and RoCE Ethernet are now often measured in fractions of a percent.
In NCCL tests, RoCE Ethernet matches InfiniBand for collective communication bandwidth.

Scalability: A Key Differentiator

InfiniBand:

Limited to ~48,000 nodes per subnet.
Larger clusters require routing between subnets, introducing complexity.

Ethernet:

Built on routable IP, scales virtually without bounds.
Arista’s DES architecture can support tens of thousands of 800G ports in a single logical fabric, removing routing complexity while maintaining scalability.

TCO and Strategic Implications

With performance near parity, the decision shifts to economics and operational strategy:

CapEx: InfiniBand hardware costs roughly double that of Ethernet equivalents.
OpEx: InfiniBand demands specialized expertise. Ethernet leverages existing network talent and tools.
Unified Operations: Ethernet enables a single operational model across AI, storage, and general compute networks. InfiniBand creates a silo.

For many enterprises, especially those already deeply invested in Ethernet, sticking with an open Ethernet fabric is the pragmatic choice.

The Road Ahead: Ultra Ethernet Consortium

While RoCEv2 has proven effective, it’s ultimately a retrofit. The future lies in standards like the Ultra Ethernet Consortium (UEC), aiming to design Ethernet protocols purpose-built for AI’s demands:

Ultra Ethernet Transport (UET): Replaces TCP for AI workloads.
Link-Level Retry (LLR): Moves retransmission into hardware for faster recovery.
Advanced Congestion Control: Smarter algorithms for balancing massive AI traffic.

Both Arista and NVIDIA are positioning their solutions as “UEC-ready,” signaling that the future competition won’t be Ethernet vs. InfiniBand, but different flavors of Ethernet.

A Strategic Choice

InfiniBand remains excellent for highly specialized HPC use cases where ultimate performance is non-negotiable. However, for the vast majority of AI deployments, Ethernet has proven itself capable, cost-effective, and future-ready.

The real battle has shifted: it’s no longer about protocols but about choosing between a proprietary vertical stack or an open ecosystem. The evidence suggests that for most organizations, Ethernet is the network fabric of the AI era, and the ecosystem war is only just beginning.