"Theory for Deep Learning, and Deep Learning for Theory"
Preetum Nakkiran (UCSD)
Monday, October 25th 2021, 2-3pm
Abstract:
The goal of this talk is to discuss the interaction between Deep Learning (DL) and Theory. In the first part, I will talk about motivations and methods. Why do we study DL? (1. To better understand the real world. 2. For what DL can teach us about the theoretical world. 3...) How should we study DL? (Is the "TCS approach" appropriate? Why/not?) I will argue for an "empirical science" approach, but guided by a "TCS aesthetic" – closer to the approach of physics than of pure mathematics. This part based loosely on my thesis.
In the second part, I will talk about more concrete results. I'll discuss the Deep Bootstrap Framework, a framework for understanding generalization in deep learning (and beyond). It is often claimed that optimization is an insufficient language to capture generalization. In contrast, I will show how to "reduce" generalization to two questions in optimization (online optimization & empirical optimization). This part based on joint work with Behnam Neyshabur, Hanie Sedghi.