SFU Mathematics of Computation, Application and Data ("MOCAD") Seminar: Liam Madden

The memory capacity of a statistical model is the largest size of generic data that the model can memorize and has important implications for both training and generalization. In this talk, we will prove a tight memory capacity result for two-layer neural networks with polynomial or real analytic activations. In order to do so, we will use tools from linear algebra, combinatorics, differential topology, and the theory of real analytic functions of several variables. In particular, we will show how to get memorization if the model is a local submersion and we will show that the Jacobian has generically full rank. The perspective that is developed also opens up a path towards deeper architectures, alternative models, and training.