Notes on Deep Learning Theory

TL;DR

This paper provides lecture notes on deep learning theory, covering topics like initialization, loss landscape, generalization, and neural tangent kernels, offering insights into the theoretical foundations of neural networks.

Contribution

It compiles and presents key theoretical aspects of deep learning, including neural tangent kernel theory, in a comprehensive lecture note format.

Findings

Insights into neural tangent kernel behavior

Analysis of loss landscape properties

Discussion on generalization in deep networks

Abstract

These are the notes for the lectures that I was giving during Fall 2020 at the Moscow Institute of Physics and Technology (MIPT) and at the Yandex School of Data Analysis (YSDA). The notes cover some aspects of initialization, loss landscape, generalization, and a neural tangent kernel theory. While many other topics (e.g. expressivity, a mean-field theory, a double descent phenomenon) are missing in the current version, we plan to add them in future revisions.