Learning Connectivity of Neural Networks from a Topological Perspective

TL;DR

This paper introduces a topological approach to optimize neural network connectivity by learning the importance of connections, leading to improved performance in image classification and object detection.

Contribution

It proposes a differentiable method to learn neural network topologies using a graph representation with learnable edge parameters and sparsity constraints.

Findings

Learned connectivity outperforms rule-based topologies.

Significant improvements in image classification accuracy.

Enhanced object detection results without extra computational cost.

Abstract

Seeking effective neural networks is a critical and practical field in deep learning. Besides designing the depth, type of convolution, normalization, and nonlinearities, the topological connectivity of neural networks is also important. Previous principles of rule-based modular design simplify the difficulty of building an effective architecture, but constrain the possible topologies in limited spaces. In this paper, we attempt to optimize the connectivity in neural networks. We propose a topological perspective to represent a network into a complete graph for analysis, where nodes carry out aggregation and transformation of features, and edges determine the flow of information. By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner. We further attach auxiliary sparsity constraint to the distribution of connectedness, which promotes the learned topology focus on critical connections. This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks. Quantitative results of experiments reflect the learned connectivity is superior to traditional rule-based ones, such as random, residual, and complete. In addition, it obtains significant improvements in image classification and object detection without introducing excessive computation burden.