Reframing Neural Networks: Deep Structure in Overcomplete Representations

TL;DR

This paper introduces a theoretical framework called deep frame approximation to analyze deep neural networks' effectiveness, linking their structure to representation stability and generalization, and exploring recurrent networks for robustness.

Contribution

It proposes a unifying theory connecting deep network architecture hyperparameters to overcomplete frame structures and introduces the deep frame potential as a measure for model selection.

Findings

Deep frame potential correlates with generalization error.

Recurrent networks can match feed-forward performance and enhance robustness.

The framework guides principled deep network architecture design.

Abstract

In comparison to classical shallow representation learning techniques, deep neural networks have achieved superior performance in nearly every application benchmark. But despite their clear empirical advantages, it is still not well understood what makes them so effective. To approach this question, we introduce deep frame approximation: a unifying framework for constrained representation learning with structured overcomplete frames. While exact inference requires iterative optimization, it may be approximated by the operations of a feed-forward deep neural network. We indirectly analyze how model capacity relates to frame structures induced by architectural hyperparameters such as depth, width, and skip connections. We quantify these structural differences with the deep frame potential, a data-independent measure of coherence linked to representation uniqueness and stability. As a criterion for model selection, we show correlation with generalization error on a variety of common deep network architectures and datasets. We also demonstrate how recurrent networks implementing iterative optimization algorithms can achieve performance comparable to their feed-forward approximations while improving adversarial robustness. This connection to the established theory of overcomplete representations suggests promising new directions for principled deep network architecture design with less reliance on ad-hoc engineering.