Computability of Classification and Deep Learning: From Theoretical Limits to Practical Feasibility through Quantization

TL;DR

This paper explores the theoretical limits of deep learning computability in classification and training, revealing fundamental restrictions and proposing that quantization can mitigate these limitations, impacting the feasibility of guarantees in safety-critical applications.

Contribution

It extends computability analysis to deep learning, identifying inherent limitations and demonstrating how quantization can overcome some of these restrictions.

Findings

Computability restrictions hinder classification and training in deep learning.

Detecting failures algorithmically in deep learning computations is infeasible.

Quantization can alleviate some computability limitations.

Abstract

The unwavering success of deep learning in the past decade led to the increasing prevalence of deep learning methods in various application fields. However, the downsides of deep learning, most prominently its lack of trustworthiness, may not be compatible with safety-critical or high-responsibility applications requiring stricter performance guarantees. Recently, several instances of deep learning applications have been shown to be subject to theoretical limitations of computability, undermining the feasibility of performance guarantees when employed on real-world computers. We extend the findings by studying computability in the deep learning framework from two perspectives: From an application viewpoint in the context of classification problems and a general limitation viewpoint in the context of training neural networks. In particular, we show restrictions on the algorithmic solvability of classification problems that also render the algorithmic detection of failure in computations in a general setting infeasible. Subsequently, we prove algorithmic limitations in training deep neural networks even in cases where the underlying problem is well-behaved. Finally, we end with a positive observation, showing that in quantized versions of classification and deep network training, computability restrictions do not arise or can be overcome to a certain degree.