BARS: Joint Search of Cell Topology and Layout for Accurate and Efficient Binary ARchitectures

TL;DR

This paper introduces BARS, a neural architecture search method for binary neural networks that optimizes both topology and layout, leading to more accurate and efficient BNNs on CIFAR-10 and ImageNet.

Contribution

BARS proposes a novel two-level search space and differentiable NAS approach to automatically discover superior binary architectures, addressing information flow and stability challenges.

Findings

Achieves 1.5% higher accuracy on CIFAR-10 with fewer binary operations.

Attains 6% accuracy gain over hand-crafted binary ResNet-18 on ImageNet.

Outperforms existing BNN NAS methods in accuracy and efficiency.

Abstract

Binary Neural Networks (BNNs) have received significant attention due to their promising efficiency. Currently, most BNN studies directly adopt widely-used CNN architectures, which can be suboptimal for BNNs. This paper proposes a novel Binary ARchitecture Search (BARS) flow to discover superior binary architecture in a large design space. Specifically, we analyze the information bottlenecks that are related to both the topology and layout architecture design choices. And we propose to automatically search for the optimal information flow. To achieve that, we design a two-level (Macro & Micro) search space tailored for BNNs and apply a differentiable neural architecture search (NAS) to explore this search space efficiently. The macro-level search space includes width and depth decisions, which is required for better balancing the model performance and complexity. We also design the micro-level search space to strengthen the information flow for BNN. %A notable challenge of BNN architecture search lies in that binary operations exacerbate the "collapse" problem of differentiable NAS, for which we incorporate various search and derive strategies to stabilize the search process. On CIFAR-10, BARS achieves 1.5% higher accuracy with 2/3 binary operations and 1/10 floating-point operations comparing with existing BNN NAS studies. On ImageNet, with similar resource consumption, BARS-discovered architecture achieves a 6% accuracy gain than hand-crafted binary ResNet-18 architectures and outperforms other binary architectures while fully binarizing the architecture backbone.