Decoupled DiLoCo: AI Training with 99% Bandwidth Savings

Key Takeaways & Technical TL;DR

• Unprecedented Bandwidth Reduction: Slashes required inter-datacenter connectivity from 198 Gbps down to just 0.84 Gbps.
• Fault-Isolated Resilience: Achieves 88% goodput during high hardware failure rates, compared to a mere 27% in traditional Data-Parallel setups.
• Heterogeneous Compute: Natively supports mixing different chip generations (e.g., TPU v6e and TPU v5p) in a single training run without performance drops.
• Maintained Accuracy: Matches traditional benchmarks, hitting 64.1% accuracy compared to the conventional 64.4% baseline on Gemma 4 architecture.