Building Sparse Deep Feedforward Networks using Tree Receptive Fields

TL;DR

This paper introduces TRF-net, a method for constructing sparse deep feedforward networks using tree-structured receptive fields, leading to models that are more efficient, interpretable, and maintain competitive performance.

Contribution

The paper proposes a novel approach to learn sparse connectivity in FNNs using Chow-Liu trees to identify strongly correlated units, resulting in more efficient and interpretable networks.

Findings

TRF-net achieves comparable or better accuracy than dense FNNs.

TRF-net uses significantly fewer parameters and has sparser structures.

The resulting models are more interpretable.

Abstract

Sparse connectivity is an important factor behind the success of convolutional neural networks and recurrent neural networks. In this paper, we consider the problem of learning sparse connectivity for feedforward neural networks (FNNs). The key idea is that a unit should be connected to a small number of units at the next level below that are strongly correlated. We use Chow-Liu's algorithm to learn a tree-structured probabilistic model for the units at the current level, use the tree to identify subsets of units that are strongly correlated, and introduce a new unit with receptive field over the subsets. The procedure is repeated on the new units to build multiple layers of hidden units. The resulting model is called a TRF-net. Empirical results show that, when compared to dense FNNs, TRF-net achieves better or comparable classification performance with much fewer parameters and sparser structures. They are also more interpretable.