Learning Architectures for Binary Networks

TL;DR

This paper introduces a neural architecture search method tailored for binary networks, optimizing architecture design specifically for binary quantization to improve performance and training stability.

Contribution

It proposes a new search space and objective for binary network architecture search, including a novel cell template and the use of Zeroise layers, leading to better-performing binary architectures.

Findings

Searched architectures outperform state-of-the-art binary networks.

The method achieves stable training curves despite quantization errors.

Binary architectures found are on par or better than existing methods.

Abstract

Backbone architectures of most binary networks are well-known floating point architectures such as the ResNet family. Questioning that the architectures designed for floating point networks would not be the best for binary networks, we propose to search architectures for binary networks (BNAS) by defining a new search space for binary architectures and a novel search objective. Specifically, based on the cell based search method, we define the new search space of binary layer types, design a new cell template, and rediscover the utility of and propose to use the Zeroise layer instead of using it as a placeholder. The novel search objective diversifies early search to learn better performing binary architectures. We show that our proposed method searches architectures with stable training curves despite the quantization error inherent in binary networks. Quantitative analyses demonstrate that our searched architectures outperform the architectures used in state-of-the-art binary networks and outperform or perform on par with state-of-the-art binary networks that employ various techniques other than architectural changes.