Beyond Backpropagation: Scaling Local Learning to LLMs
Abstract: Backpropagation has powered modern deep learning, but its reliance on global gradient propagation creates challenges for memory efficiency and parallelization. Can we train large language models without it? In this talk, I will introduce SOLO (Shared-Output Local Learning), a simple approach that enables neural networks to learn through local objectives rather than global backpropagation. We show that local learning can scale to billion-parameter language models, achieving competitive performance while reducing memory requirements and improving training throughput. Our findings challenge the conventional wisdom that global backpropagation is essential for training large language models, opening new directions for more efficient and scalable AI.