Towards a Robust Differentiable Architecture Search under Label Noise

TL;DR

This paper introduces a noise-robust differentiable neural architecture search method that employs an information bottleneck principle and noise injection to maintain performance in noisy label scenarios, outperforming existing methods without prior noise knowledge.

Contribution

It proposes a novel noise injection regularizer for differentiable NAS that enhances robustness against label noise without needing prior noise property knowledge.

Findings

The noise injection method maintains performance on clean data.

The method outperforms existing noise-robust NAS algorithms on noisy datasets.

It does not require prior knowledge of noise characteristics.

Abstract

Neural Architecture Search (NAS) is the game changer in designing robust neural architectures. Architectures designed by NAS outperform or compete with the best manual network designs in terms of accuracy, size, memory footprint and FLOPs. That said, previous studies focus on developing NAS algorithms for clean high quality data, a restrictive and somewhat unrealistic assumption. In this paper, focusing on the differentiable NAS algorithms, we show that vanilla NAS algorithms suffer from a performance loss if class labels are noisy. To combat this issue, we make use of the principle of information bottleneck as a regularizer. This leads us to develop a noise injecting operation that is included during the learning process, preventing the network from learning from noisy samples. Our empirical evaluations show that the noise injecting operation does not degrade the performance of the NAS algorithm if the data is indeed clean. In contrast, if the data is noisy, the architecture learned by our algorithm comfortably outperforms algorithms specifically equipped with sophisticated mechanisms to learn in the presence of label noise. In contrast to many algorithms designed to work in the presence of noisy labels, prior knowledge about the properties of the noise and its characteristics are not required for our algorithm.