Is Differentiable Architecture Search truly a One-Shot Method?

TL;DR

This paper critically examines differentiable architecture search (DAS) in inverse problems, revealing significant variability and hyperparameter sensitivity, and questioning its effectiveness as a true one-shot method beyond image classification.

Contribution

The study extends DAS analysis to inverse problems, highlighting fundamental evaluation challenges and questioning its status as a one-shot approach.

Findings

High variance in DAS results across tests

Performance heavily depends on optimizer hyperparameters

Weight-sharing architecture does not reliably predict final performance

Abstract

Differentiable architecture search (DAS) is a widely researched tool for the discovery of novel architectures, due to its promising results for image classification. The main benefit of DAS is the effectiveness achieved through the weight-sharing one-shot paradigm, which allows efficient architecture search. In this work, we investigate DAS in a systematic case study of inverse problems, which allows us to analyze these potential benefits in a controlled manner. We demonstrate that the success of DAS can be extended from image classification to signal reconstruction, in principle. However, our experiments also expose three fundamental difficulties in the evaluation of DAS-based methods in inverse problems: First, the results show a large variance in all test cases. Second, the final performance is strongly dependent on the hyperparameters of the optimizer. And third, the performance of the weight-sharing architecture used during training does not reflect the final performance of the found architecture well. While the results on image reconstruction confirm the potential of the DAS paradigm, they challenge the common understanding of DAS as a one-shot method.