High Dimensional Optimization through the Lens of Machine Learning

TL;DR

This thesis explores high-dimensional optimization techniques in machine learning, providing theoretical insights into why common heuristics like stochastic gradient descent are effective, and analyzing the role of popular optimizers in practice.

Contribution

It offers a theoretical foundation for understanding high-dimensional optimization methods used in machine learning, especially explaining the success of popular heuristics and default optimizers.

Findings

Convergence proofs for stochastic gradient descent and momentum methods.

Intuition on quadratic models for non-convex optimization.

Explanation of why common heuristics are effective in practice.

Abstract

This thesis reviews numerical optimization methods with machine learning problems in mind. Since machine learning models are highly parametrized, we focus on methods suited for high dimensional optimization. We build intuition on quadratic models to figure out which methods are suited for non-convex optimization, and develop convergence proofs on convex functions for this selection of methods. With this theoretical foundation for stochastic gradient descent and momentum methods, we try to explain why the methods used commonly in the machine learning field are so successful. Besides explaining successful heuristics, the last chapter also provides a less extensive review of more theoretical methods, which are not quite as popular in practice. So in some sense this work attempts to answer the question: Why are the default Tensorflow optimizers included in the defaults?