Deep Expander Networks: Efficient Deep Networks from Graph Theory

TL;DR

This paper introduces X-Nets, a novel CNN architecture using expander graphs to connect filters efficiently, resulting in improved accuracy and model size without pruning, inspired by graph theory principles.

Contribution

The paper proposes a new CNN design called X-Nets that leverages expander graphs for efficient, well-connected filter connections, enhancing performance and scalability.

Findings

X-Nets improve MobileNet accuracy by 4% over grouped convolutions.

X-Nets outperform ResNet and DenseNet-BC in performance trade-offs.

Achieves competitive model sizes comparable to pruning techniques without pruning.

Abstract

Efficient CNN designs like ResNets and DenseNet were proposed to improve accuracy vs efficiency trade-offs. They essentially increased the connectivity, allowing efficient information flow across layers. Inspired by these techniques, we propose to model connections between filters of a CNN using graphs which are simultaneously sparse and well connected. Sparsity results in efficiency while well connectedness can preserve the expressive power of the CNNs. We use a well-studied class of graphs from theoretical computer science that satisfies these properties known as Expander graphs. Expander graphs are used to model connections between filters in CNNs to design networks called X-Nets. We present two guarantees on the connectivity of X-Nets: Each node influences every node in a layer in logarithmic steps, and the number of paths between two sets of nodes is proportional to the product of their sizes. We also propose efficient training and inference algorithms, making it possible to train deeper and wider X-Nets effectively. Expander based models give a 4% improvement in accuracy on MobileNet over grouped convolutions, a popular technique, which has the same sparsity but worse connectivity. X-Nets give better performance trade-offs than the original ResNet and DenseNet-BC architectures. We achieve model sizes comparable to state-of-the-art pruning techniques using our simple architecture design, without any pruning. We hope that this work motivates other approaches to utilize results from graph theory to develop efficient network architectures.