Deep Learning Works in Practice. But Does it Work in Theory?

TL;DR

This paper explores the theoretical foundations of deep learning's success, proposing that the universe's complex data structure with large logical depth explains why deep neural networks outperform shallower ones.

Contribution

It introduces a novel conjecture linking the universe's computational complexity to the effectiveness of deep learning, connecting it with fundamental computer science hypotheses.

Findings

Deep learning's success may be due to the universe's large logical depth.

Theoretical connection between deep learning performance and P vs NC conjecture.

Provides a new perspective on why deeper networks outperform shallower ones.

Abstract

Deep learning relies on a very specific kind of neural networks: those superposing several neural layers. In the last few years, deep learning achieved major breakthroughs in many tasks such as image analysis, speech recognition, natural language processing, and so on. Yet, there is no theoretical explanation of this success. In particular, it is not clear why the deeper the network, the better it actually performs. We argue that the explanation is intimately connected to a key feature of the data collected from our surrounding universe to feed the machine learning algorithms: large non-parallelizable logical depth. Roughly speaking, we conjecture that the shortest computational descriptions of the universe are algorithms with inherently large computation times, even when a large number of computers are available for parallelization. Interestingly, this conjecture, combined with the folklore conjecture in theoretical computer science that $ P \neq NC$, explains the success of deep learning.