Discovering Neural Wirings
Mitchell Wortsman, Ali Farhadi, Mohammad Rastegari

TL;DR
This paper introduces a method for discovering neural wirings that learns network connectivity during training, leading to improved performance and a unified approach to neural architecture search and sparse neural network learning.
Contribution
It proposes a novel approach to learn neural network connectivity independently of fixed layers, expanding the search space and improving performance over traditional hand-engineered networks.
Findings
Learned connectivity outperforms hand-engineered and random wiring.
Boosts ImageNet accuracy of MobileNetV1 by 10% at ~41M FLOPs.
Generalizes to recurrent and continuous time networks.
Abstract
The success of neural networks has driven a shift in focus from feature engineering to architecture engineering. However, successful networks today are constructed using a small and manually defined set of building blocks. Even in methods of neural architecture search (NAS) the network connectivity patterns are largely constrained. In this work we propose a method for discovering neural wirings. We relax the typical notion of layers and instead enable channels to form connections independent of each other. This allows for a much larger space of possible networks. The wiring of our network is not fixed during training -- as we learn the network parameters we also learn the structure itself. Our experiments demonstrate that our learned connectivity outperforms hand engineered and randomly wired networks. By learning the connectivity of MobileNetV1we boost the ImageNet accuracy by 10% at…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsCell Image Analysis Techniques · Neural Networks and Applications · Neural dynamics and brain function
MethodsSigmoid Activation · Tanh Activation · Softmax · Long Short-Term Memory
