Deep Networks Learn Deep Hierarchical Models

TL;DR

This paper demonstrates that layerwise SGD on residual networks can efficiently learn complex hierarchical models, surpassing previous models in depth and providing insights into deep learning's ability to learn structured, layered representations.

Contribution

It introduces a class of hierarchical models that are learnable by deep networks at the depth limit of efficient algorithms, extending prior work on deep learning capabilities.

Findings

Layerwise SGD effectively learns hierarchical models.

Models in this class require polynomial depth, exceeding previous log-depth models.

Hierarchical structures may underpin deep learning success.

Abstract

We consider supervised learning with $n$ labels and show that layerwise SGD on residual networks can efficiently learn a class of hierarchical models. This model class assumes the existence of an (unknown) label hierarchy $L_1 \subseteq L_2 \subseteq \dots \subseteq L_r = [n]$, where labels in $L_1$ are simple functions of the input, while for $i > 1$, labels in $L_i$ are simple functions of simpler labels. Our class surpasses models that were previously shown to be learnable by deep learning algorithms, in the sense that it reaches the depth limit of efficient learnability. That is, there are models in this class that require polynomial depth to express, whereas previous models can be computed by log-depth circuits. Furthermore, we suggest that learnability of such hierarchical models might eventually form a basis for understanding deep learning. Beyond their natural fit for domains where deep learning excels, we argue that the mere existence of human ``teachers" supports the hypothesis that hierarchical structures are inherently available. By providing granular labels, teachers effectively reveal ``hints'' or ``snippets'' of the internal algorithms used by the brain. We formalize this intuition, showing that in a simplified model where a teacher is partially aware of their internal logic, a hierarchical structure emerges that facilitates efficient learnability.