Statistical Foundation Behind Machine Learning and Its Impact on Computer Vision

TL;DR

This paper explores the statistical principles underlying machine learning, especially uniform convergence, and discusses how large-scale data and future research directions impact computer vision and model robustness.

Contribution

It provides a theoretical perspective on the foundation of machine learning and highlights the importance of structure and knowledge for future advancements.

Findings

Large-scale data reduces empirical and expected loss discrepancy.

Pre-training improves representation learning in computer vision.

Fundamental research needed on robustness, interpretability, and reasoning.

Abstract

This paper revisits the principle of uniform convergence in statistical learning, discusses how it acts as the foundation behind machine learning, and attempts to gain a better understanding of the essential problem that current deep learning algorithms are solving. Using computer vision as an example domain in machine learning, the discussion shows that recent research trends in leveraging increasingly large-scale data to perform pre-training for representation learning are largely to reduce the discrepancy between a practically tractable empirical loss and its ultimately desired but intractable expected loss. Furthermore, this paper suggests a few future research directions, predicts the continued increase of data, and argues that more fundamental research is needed on robustness, interpretability, and reasoning capabilities of machine learning by incorporating structure and knowledge.