DCP-NAS: Discrepant Child-Parent Neural Architecture Search for 1-bit CNNs

TL;DR

This paper introduces DCP-NAS, a novel neural architecture search framework that efficiently designs 1-bit CNNs by leveraging a real-valued parent model to guide the search of a binary child model, reducing computational costs.

Contribution

The paper proposes a new framework for searching 1-bit CNN architectures using a parent-child model approach with tangent propagation and decoupled optimization, improving efficiency and performance.

Findings

DCP-NAS outperforms prior methods on CIFAR-10 and ImageNet.

The method achieves strong generalization on re-identification and object detection.

Efficient search of 1-bit CNNs reduces resource consumption.

Abstract

Neural architecture search (NAS) proves to be among the effective approaches for many tasks by generating an application-adaptive neural architecture, which is still challenged by high computational cost and memory consumption. At the same time, 1-bit convolutional neural networks (CNNs) with binary weights and activations show their potential for resource-limited embedded devices. One natural approach is to use 1-bit CNNs to reduce the computation and memory cost of NAS by taking advantage of the strengths of each in a unified framework, while searching the 1-bit CNNs is more challenging due to the more complicated processes involved. In this paper, we introduce Discrepant Child-Parent Neural Architecture Search (DCP-NAS) to efficiently search 1-bit CNNs, based on a new framework of searching the 1-bit model (Child) under the supervision of a real-valued model (Parent). Particularly, we first utilize a Parent model to calculate a tangent direction, based on which the tangent propagation method is introduced to search the optimized 1-bit Child. We further observe a coupling relationship between the weights and architecture parameters existing in such differentiable frameworks. To address the issue, we propose a decoupled optimization method to search an optimized architecture. Extensive experiments demonstrate that our DCP-NAS achieves much better results than prior arts on both CIFAR-10 and ImageNet datasets. In particular, the backbones achieved by our DCP-NAS achieve strong generalization performance on person re-identification and object detection.