The Boundaries of Verifiable Accuracy, Robustness, and Generalisation in Deep Learning

TL;DR

This paper explores the fundamental theoretical limits of verifying the stability, accuracy, and robustness of neural networks in classification, highlighting significant challenges in certifying ideal models within common training frameworks.

Contribution

It identifies the inherent computational difficulties in verifying neural network properties under classical risk minimization and regularization, revealing fundamental limitations.

Findings

Verifying ideal neural networks is computationally hard in many cases.

There exist tasks where stable and accurate networks cannot be efficiently verified.

Verification challenges persist even when ideal solutions are within the neural network class.

Abstract

In this work, we assess the theoretical limitations of determining guaranteed stability and accuracy of neural networks in classification tasks. We consider classical distribution-agnostic framework and algorithms minimising empirical risks and potentially subjected to some weights regularisation. We show that there is a large family of tasks for which computing and verifying ideal stable and accurate neural networks in the above settings is extremely challenging, if at all possible, even when such ideal solutions exist within the given class of neural architectures.