Exploring Randomly Wired Neural Networks for Image Recognition

TL;DR

This paper investigates the use of randomly generated connectivity patterns in neural networks for image recognition, revealing that such diverse wiring can achieve competitive accuracy and open new avenues for network design.

Contribution

It introduces a stochastic network generator framework that unifies NAS and random wiring, demonstrating the effectiveness of classical random graph models in neural network connectivity.

Findings

Randomly wired networks achieve competitive ImageNet accuracy.

Diverse wiring patterns can outperform manually designed structures.

Exploring less constrained wiring spaces may lead to new breakthroughs.

Abstract

Neural networks for image recognition have evolved through extensive manual design from simple chain-like models to structures with multiple wiring paths. The success of ResNets and DenseNets is due in large part to their innovative wiring plans. Now, neural architecture search (NAS) studies are exploring the joint optimization of wiring and operation types, however, the space of possible wirings is constrained and still driven by manual design despite being searched. In this paper, we explore a more diverse set of connectivity patterns through the lens of randomly wired neural networks. To do this, we first define the concept of a stochastic network generator that encapsulates the entire network generation process. Encapsulation provides a unified view of NAS and randomly wired networks. Then, we use three classical random graph models to generate randomly wired graphs for networks. The results are surprising: several variants of these random generators yield network instances that have competitive accuracy on the ImageNet benchmark. These results suggest that new efforts focusing on designing better network generators may lead to new breakthroughs by exploring less constrained search spaces with more room for novel design.