Analysis and Design of Convolutional Networks via Hierarchical Tensor Decompositions

TL;DR

This paper reviews a series of works that analyze convolutional networks' expressiveness and inductive bias through hierarchical tensor decompositions, offering insights into their effectiveness and guiding network design.

Contribution

It introduces a formal framework linking convolutional network features to hierarchical tensor decompositions, providing new tools for analyzing and designing neural networks.

Findings

Deeper networks have greater expressive efficiency.

Strides and width influence inductive bias.

Hierarchical tensor decompositions explain convolutional networks' success.

Abstract

The driving force behind convolutional networks - the most successful deep learning architecture to date, is their expressive power. Despite its wide acceptance and vast empirical evidence, formal analyses supporting this belief are scarce. The primary notions for formally reasoning about expressiveness are efficiency and inductive bias. Expressive efficiency refers to the ability of a network architecture to realize functions that require an alternative architecture to be much larger. Inductive bias refers to the prioritization of some functions over others given prior knowledge regarding a task at hand. In this paper we overview a series of works written by the authors, that through an equivalence to hierarchical tensor decompositions, analyze the expressive efficiency and inductive bias of various convolutional network architectural features (depth, width, strides and more). The results presented shed light on the demonstrated effectiveness of convolutional networks, and in addition, provide new tools for network design.