Please note: This distinguished lecture will take place in DC 1302 and online.
Peter Bartlett
Professor, Statistics and Computer Science, UC Berkeley
Principal Scientist, Google DeepMind
Deep learning, the technology underlying the recent progress in AI, has revealed some major surprises from the perspective of theory. Optimization in deep learning relies on simple gradient descent algorithms that are traditionally viewed as a time discretization of gradient flow. However, in practice, large step sizes — large enough to cause oscillation of the loss — exhibit performance advantages.
This talk will review recent results on gradient descent with logistic loss with a step size large enough that the optimization trajectory is at the “edge of stability.” We show the benefits of this initial oscillatory phase for linear functions and for multi-layer networks, and identify an asymptotic implicit bias that gradient descent imposes for a large family of deep networks.
Based on joint work with Yuhang Cai, Michael Lindsey, Song Mei, Matus Telgarsky, Jingfeng Wu, Bin Yu and Kangjie Zhou.
Bio: Peter Bartlett is Professor of Statistics and Computer Science at UC Berkeley and Principal Scientist at Google DeepMind. At Berkeley, he is the Machine Learning Research Director at the Simons Institute for the Theory of Computing, Director of the Foundations of Data Science Institute, and Director of the Collaboration on the Theoretical Foundations of Deep Learning, and he has served as Associate Director of the Simons Institute. He is President of the Association for Computational Learning, Honorary Professor of Mathematical Sciences at the Australian National University, and co-author with Martin Anthony of the book Neural Network Learning: Theoretical Foundations.
He was awarded the Malcolm McIntosh Prize for Physical Scientist of the Year in Australia, and has been an Institute of Mathematical Statistics Medallion Lecturer, an IMS Fellow and Australian Laureate Fellow, a Fellow of the ACM, a recipient of the UC Berkeley Chancellor’s Distinguished Service Award, and a Fellow of the Australian Academy of Science.
To attend this distinguished lecture in person, please go to DC 1302. You can also attend virtually on Zoom.