Datacenter Interconnects: Copper vs. Optics, Nvidia's 78-Layer PCB, Co-Packaged Optics (CPO)

Summary

Vik Sekar and Austin Lyons frame datacenter networking as the next major bottleneck for AI: chips are getting faster faster than the fabric connecting them. They break the datacenter into three layers—scale up, scale out, and scale across—and argue that the scale-up layer is where connection density, latency, and power pressure are most extreme. The core rule is “copper when you can, optics when you must,” because every optical hop adds transceiver, DSP, and cooling overhead. Nvidia’s response is to stretch copper further with a highly complex 78-layer PCB in next-generation racks, while the industry continues chasing near-package and co-packaged optics as the long-term answer. The big blocker for CPO is serviceability: if an optical engine or laser fails, replacing it may require sacrificing an expensive advanced-node package.

Notes

• Datacenter networking is presented as the main bottleneck for AI systems, with compute scaling faster than the fabric that connects chips.
• The hosts define three networking tiers: scale up within a rack, scale out across racks, and scale across between campuses or data centers.
• Austin describes scale up as low-latency GPU-to-GPU communication that should feel like shared memory.
• The episode repeatedly returns to the principle that copper should be used wherever possible, with optics reserved for cases where reach forces the switch.
• The hosts note competing scale-up ecosystems including NVLink, UALink, Infiniband, Ethernet, and Nvidia’s Spectrum-X Ethernet push.
• Front-end networking is separated from the backend GPU fabric: client traffic enters from outside, while GPUs communicate internally and then send results back out.
• Inference can become network-limited because message passing and hop latency across an MoE model accumulate into a minimum response time.
• A 72-GPU rack is described as requiring all-to-all connectivity, which translates into a little over 5,000 cables and about 2 km of cabling.
• Blackwell-era interconnects are described as running at 200 gigabits per lane, with Rubin-era links framed as even faster bidirectional signaling on one cable.
• The Nvidia GB200 deployment in xAI’s Colossus is cited as an example of carefully managed optical cabling inside a datacenter.
• Cable density is highest in scale up and falls as you move outward, which the hosts treat as a proxy for optical vendors’ addressable market.
• Nvidia’s next-generation rack design is said to use a gigantic PCB to route signals internally and avoid paying the optics power tax.
• The episode contrasts short-reach PAM4 links with longer-reach coherent optics, plus WDM, CWDM, and DWDM for data-center and inter-data-center transport.
• The optical transceiver explanation includes lasers, photodiodes, driver chips, amplifiers, DSPs, parity, and error correction.
• Austin emphasizes that optics adds power, latency, and error-correction overhead that compounds when every GPU in a scale-up fabric needs transceivers.
• Nvidia is said to oppose pluggable optics for scale up because each module draws about 30 W, which scales badly in high-density racks.
• The episode places current and upcoming networking generations on a transition curve from 100/200G to 400G, then 800G, then 1.6T.
• A possible 2.4T generation using 300 Gbps per lane is discussed as a potential step before 3.2T.
• Pluggable optics are criticized as power hungry because the long PCB path from front panel to switch silicon hurts signal integrity at 200 Gbps/lane and above.
• Near-package optics is presented as a way to move the optical engine inside the box and closer to the switch to reduce the electrical path.
• Co-packaged optics is described as the ideal end state: the switch or GPU is packaged next to the optical engine with almost no copper left in the path.
• The main blocker for CPO is reliability and serviceability: lasers die, and a failed optical engine may force you to discard the whole expensive switch package.
• One mitigation path is to move the laser outside the rack while leaving the rest of the optical subsystem inside.
• Vik notes that near-package and co-packaged optics could cut energy use to about one-third of a pluggable transceiver.
• The long-term vision is an optics-heavy scale-up and scale-out fabric spanning hundreds of thousands of GPUs, but the timeline is still unclear.