Transgressing the boundaries: towards a rigorous understanding of deep learning and its (non-)robustness

TL;DR

This paper reviews the theoretical understanding of deep learning, focusing on robustness, interpretability, and uncertainty quantification, highlighting the gap between practical success and mathematical foundations.

Contribution

It systematically bridges approximation theory and statistical learning theory to address robustness and interpretability issues in deep learning.

Findings

Analysis of robustness challenges in DL

Review of Bayesian methods for uncertainty quantification

Connections between approximation theory and DL robustness

Abstract

The recent advances in machine learning in various fields of applications can be largely attributed to the rise of deep learning (DL) methods and architectures. Despite being a key technology behind autonomous cars, image processing, speech recognition, etc., a notorious problem remains the lack of theoretical understanding of DL and related interpretability and (adversarial) robustness issues. Understanding the specifics of DL, as compared to, say, other forms of nonlinear regression methods or statistical learning, is interesting from a mathematical perspective, but at the same time it is of crucial importance in practice: treating neural networks as mere black boxes might be sufficient in certain cases, but many applications require waterproof performance guarantees and a deeper understanding of what could go wrong and why it could go wrong. It is probably fair to say that, despite being mathematically well founded as a method to approximate complicated functions, DL is mostly still more like modern alchemy that is firmly in the hands of engineers and computer scientists. Nevertheless, it is evident that certain specifics of DL that could explain its success in applications demands systematic mathematical approaches. In this work, we review robustness issues of DL and particularly bridge concerns and attempts from approximation theory to statistical learning theory. Further, we review Bayesian Deep Learning as a means for uncertainty quantification and rigorous explainability.