Shawn Shuoshuo Chen

Every web search, video stream, and chatbot reply is served not by one computer but by thousands of them, packed into warehouse-sized data centers and stitched together by an internal network. For decades that network was scaled by brute force. When the machines needed to talk faster, operators bought faster switches and wired in more of them. This approach is now failing. The chips inside those switches have stopped getting faster the way they once did, and the standard design insists that every machine be able to reach every other at full speed at once, so keeping up means buying more hardware at prohibitively higher cost and power. Then came AI, whose appetite for shuffling data between chips dwarfs anything before it. The result is a widening gap between the bandwidth a data center needs and the bandwidth anyone can afford to build, a gap we call the data center scaling crisis.

Yet a data center rarely needs all that bandwidth at once. Traffic is often bursty and localized: a few machines communicating heavily while the rest stay quiet, so most of a uniform network's capacity sits idle. The smarter move is to shift bandwidth on the fly to wherever demand actually is, and that is exactly what a reconfigurable data center network (RDCN) does. To do so it relies on a kind of switch very different from the packet switches networks use today, the optical circuit switch (OCS), which is both cheaper and more energy-efficient. The catch is that this is a paradigm shift from today's packet-switched networks: decades of protocols quietly assume what a circuit-switched RDCN cannot promise, uniform bandwidth between every pair of machines and static wiring that never changes. When those assumptions break, the overall performance suffers.

This thesis redesigns the network protocol stack for the RDCN to reclaim the performance that the unmodified stack gives up. It tackles the three core components of that stack in turn: the transport protocol, routing and traffic engineering, and job scheduling. For transport, conventional congestion control stumbles when the path beneath it keeps changing, so our work Time-Division TCP (TDTCP) keeps separate state for each network configuration and reuses it across reconfigurations, delivering 24–41% higher throughput than existing TCP. For traffic engineering, a spine-free optical data center network offers uneven, shifting paths that overwhelm today's switch hardware, so our work PreciseTE computes routes that respect those hardware nlimits, holding precision loss below 7% where the state of the art loses up to 67%, and is now deployed in Google's production network. For job scheduling, ML jobs placed on a static non-reconfigurable torus must trade fragmentation against contention, and our work RFold co-adapts the job's shape and the topology to escape both, cutting average job completion time by more than 4x at high load. Overall, reconfigurability is reshaping data center network infrastructure, but unlocking its performance benefits calls for a systematic redesign of the protocol stack above it.