Resource-Efficient Neural Architect

TL;DR

RENA is a reinforcement learning-based neural architecture search method that efficiently finds high-performing, resource-constrained neural networks for image recognition and keyword spotting tasks.

Contribution

It introduces a novel resource-efficient NAS approach using network embeddings and reinforcement learning, addressing computational resource constraints.

Findings

Achieves 2.95% test error on CIFAR10 with >100 FLOPs/byte

Attains state-of-the-art accuracy on Google Speech Commands Dataset

Outperforms optimized architectures under tight resource constraints

Abstract

Neural Architecture Search (NAS) is a laborious process. Prior work on automated NAS targets mainly on improving accuracy, but lacks consideration of computational resource use. We propose the Resource-Efficient Neural Architect (RENA), an efficient resource-constrained NAS using reinforcement learning with network embedding. RENA uses a policy network to process the network embeddings to generate new configurations. We demonstrate RENA on image recognition and keyword spotting (KWS) problems. RENA can find novel architectures that achieve high performance even with tight resource constraints. For CIFAR10, it achieves 2.95% test error when compute intensity is greater than 100 FLOPs/byte, and 3.87% test error when model size is less than 3M parameters. For Google Speech Commands Dataset, RENA achieves the state-of-the-art accuracy without resource constraints, and it outperforms the optimized architectures with tight resource constraints.