ROME: Robustifying Memory-Efficient NAS via Topology Disentanglement and Gradient Accumulation

TL;DR

This paper introduces ROME, a novel method for memory-efficient neural architecture search that disentangles topology from operations and employs gradient accumulation to improve robustness and performance across multiple benchmarks.

Contribution

The paper proposes ROME, a new NAS algorithm that addresses performance collapse in single-path DARTS by disentangling topology and operation search and using gradient accumulation for robustness.

Findings

Effective across 15 benchmarks

Reduces memory cost in NAS

Improves robustness and performance

Abstract

Albeit being a prevalent architecture searching approach, differentiable architecture search (DARTS) is largely hindered by its substantial memory cost since the entire supernet resides in the memory. This is where the single-path DARTS comes in, which only chooses a single-path submodel at each step. While being memory-friendly, it also comes with low computational costs. Nonetheless, we discover a critical issue of single-path DARTS that has not been primarily noticed. Namely, it also suffers from severe performance collapse since too many parameter-free operations like skip connections are derived, just like DARTS does. In this paper, we propose a new algorithm called RObustifying Memory-Efficient NAS (ROME) to give a cure. First, we disentangle the topology search from the operation search to make searching and evaluation consistent. We then adopt Gumbel-Top2 reparameterization and gradient accumulation to robustify the unwieldy bi-level optimization. We verify ROME extensively across 15 benchmarks to demonstrate its effectiveness and robustness.