Experimental Design for Overparameterized Learning with Application to Single Shot Deep Active Learning

TL;DR

This paper develops a new experimental design strategy tailored for overparameterized models like deep neural networks, addressing the limitations of classical methods and demonstrating its effectiveness in single shot deep active learning.

Contribution

It introduces a novel design approach for overparameterized models and applies it to create a new algorithm for deep active learning.

Findings

The proposed method effectively selects data points for overparameterized models.

The new algorithm improves data efficiency in deep active learning.

Classical experimental design is inadequate for modern overparameterized models.

Abstract

The impressive performance exhibited by modern machine learning models hinges on the ability to train such models on a very large amounts of labeled data. However, since access to large volumes of labeled data is often limited or expensive, it is desirable to alleviate this bottleneck by carefully curating the training set. Optimal experimental design is a well-established paradigm for selecting data point to be labeled so to maximally inform the learning process. Unfortunately, classical theory on optimal experimental design focuses on selecting examples in order to learn underparameterized (and thus, non-interpolative) models, while modern machine learning models such as deep neural networks are overparameterized, and oftentimes are trained to be interpolative. As such, classical experimental design methods are not applicable in many modern learning setups. Indeed, the predictive performance of underparameterized models tends to be variance dominated, so classical experimental design focuses on variance reduction, while the predictive performance of overparameterized models can also be, as is shown in this paper, bias dominated or of mixed nature. In this paper we propose a design strategy that is well suited for overparameterized regression and interpolation, and we demonstrate the applicability of our method in the context of deep learning by proposing a new algorithm for single shot deep active learning.