AnalogNAS-Bench: A NAS Benchmark for Analog In-Memory Computing

TL;DR

AnalogNAS-Bench introduces a specialized NAS benchmark for analog in-memory computing, enabling the discovery of neural architectures optimized for AIMC's unique hardware non-idealities, thus advancing analog-aware neural architecture search.

Contribution

It presents the first NAS benchmark tailored for AIMC, revealing key insights into robust architecture design under analog hardware constraints.

Findings

Standard quantization techniques do not capture AIMC-specific noises.

Robust architectures tend to have wider and branched blocks.

Skip connections enhance resilience to temporal drift noise.

Abstract

Analog In-memory Computing (AIMC) has emerged as a highly efficient paradigm for accelerating Deep Neural Networks (DNNs), offering significant energy and latency benefits over conventional digital hardware. However, state-of-the-art neural networks are not inherently designed for AIMC, as they fail to account for its unique non-idealities. Neural Architecture Search (NAS) is thus needed to systematically discover neural architectures optimized explicitly for AIMC constraints. However, comparing NAS methodologies and extracting insights about robust architectures for AIMC requires a dedicated NAS benchmark that explicitly accounts for AIMC-specific hardware non-idealities. To address this, we introduce AnalogNAS-Bench, the first NAS benchmark tailored specifically for AIMC. Our study reveals three key insights: (1) standard quantization techniques fail to capture AIMC-specific noises, (2) robust architectures tend to feature wider and branched blocks, (3) skip connections improve resilience to temporal drift noise. These insights highlight the limitations of current NAS benchmarks for AIMC and pave the way for future analog-aware NAS. All the implementations used in this paper can be found at https://github.com/IBM/analog-nas/tree/main/analognasbench.