Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity

TL;DR

This paper explores the duality between neural networks and compositional kernels, revealing that initial random weights provide rich representations that facilitate learning and highlight the networks' expressive power.

Contribution

It introduces a dual framework linking neural networks to kernels, showing initializations are sufficiently expressive to approximate functions in the dual space.

Findings

Initial random weights generate rich, expressive representations.

The dual view clarifies neural networks' expressive power.

Training is challenging in worst-case scenarios, but initializations are promising.

Abstract

We develop a general duality between neural networks and compositional kernels, striving towards a better understanding of deep learning. We show that initial representations generated by common random initializations are sufficiently rich to express all functions in the dual kernel space. Hence, though the training objective is hard to optimize in the worst case, the initial weights form a good starting point for optimization. Our dual view also reveals a pragmatic and aesthetic perspective of neural networks and underscores their expressive power.