Towards Robust Deep Active Learning for Scientific Computing

TL;DR

This paper investigates the robustness of deep active learning methods in scientific computing, highlighting the impact of unknown hyperparameters and proposing a new query synthesis approach that outperforms existing methods and random sampling.

Contribution

It identifies the hyperparameter sensitivity in current DAL methods and introduces NA-QBC, the first query synthesis DAL method for regression that is hyperparameter-free and more robust.

Findings

Modern DAL methods' performance drops without known pool ratio

NA-QBC outperforms other DAL methods on benchmark problems

NA-QBC consistently beats random sampling in performance

Abstract

Deep learning (DL) is revolutionizing the scientific computing community. To reduce the data gap, active learning has been identified as a promising solution for DL in the scientific computing community. However, the deep active learning (DAL) literature is dominated by image classification problems and pool-based methods. Here we investigate the robustness of pool-based DAL methods for scientific computing problems (dominated by regression) where DNNs are increasingly used. We show that modern pool-based DAL methods all share an untunable hyperparameter, termed the pool ratio, denoted $\gamma$, which is often assumed to be known apriori in the literature. We evaluate the performance of five state-of-the-art DAL methods on six benchmark problems if we assume $\gamma$ is \textit{not} known - a more realistic assumption for scientific computing problems. Our results indicate that this reduces the performance of modern DAL methods and that they sometimes can even perform worse than random sampling, creating significant uncertainty when used in real-world settings. To overcome this limitation we propose, to our knowledge, the first query synthesis DAL method for regression, termed NA-QBC. NA-QBC removes the sensitive $\gamma$ hyperparameter and we find that, on average, it outperforms the other DAL methods on our benchmark problems. Crucially, NA-QBC always outperforms random sampling, providing more robust performance benefits.