On the Learnability of Deep Random Networks

TL;DR

This paper investigates the learnability challenges of deep random networks, revealing that their ability to be learned diminishes exponentially with depth, both theoretically and practically.

Contribution

It provides a theoretical analysis showing exponential decay in learnability with depth and empirically confirms this trend with modern training methods.

Findings

Learnability drops exponentially with network depth

Practical training methods do not overcome depth-related learnability issues

Theoretical results align closely with empirical observations

Abstract

In this paper we study the learnability of deep random networks from both theoretical and practical points of view. On the theoretical front, we show that the learnability of random deep networks with sign activation drops exponentially with its depth. On the practical front, we find that the learnability drops sharply with depth even with the state-of-the-art training methods, suggesting that our stylized theoretical results are closer to reality.