Towards Regression-Free Neural Networks for Diverse Compute Platforms

TL;DR

This paper introduces REG-NAS, a neural architecture search method that designs models with fewer negative flips, ensuring more consistent predictions across diverse compute platforms, especially for on-device deep learning.

Contribution

The paper proposes a novel architecture constraint and search reward in neural architecture search to reduce negative flips and improve prediction consistency across platforms.

Findings

REG-NAS achieves 33-48% reduction in negative flips.

It finds architectures with fewer negative flips in multiple search spaces.

The approach improves prediction consistency for on-device deep learning.

Abstract

With the shift towards on-device deep learning, ensuring a consistent behavior of an AI service across diverse compute platforms becomes tremendously important. Our work tackles the emergent problem of reducing predictive inconsistencies arising as negative flips: test samples that are correctly predicted by a less accurate model, but incorrectly by a more accurate one. We introduce REGression constrained Neural Architecture Search (REG-NAS) to design a family of highly accurate models that engender fewer negative flips. REG-NAS consists of two components: (1) A novel architecture constraint that enables a larger model to contain all the weights of the smaller one thus maximizing weight sharing. This idea stems from our observation that larger weight sharing among networks leads to similar sample-wise predictions and results in fewer negative flips; (2) A novel search reward that incorporates both Top-1 accuracy and negative flips in the architecture search metric. We demonstrate that \regnas can successfully find desirable architectures with few negative flips in three popular architecture search spaces. Compared to the existing state-of-the-art approach, REG-NAS enables 33-48% relative reduction of negative flips.