Addressing Bias in Active Learning with Depth Uncertainty Networks... or Not

TL;DR

This paper investigates why bias correction in active learning does not improve overparameterised neural networks, proposing that depth uncertainty networks operate in a low overfitting regime but still do not benefit from bias correction, explained through bias-variance decomposition.

Contribution

The paper provides a new perspective on active learning bias correction by analyzing depth uncertainty networks and explaining their behavior via bias-variance decomposition.

Findings

Depth uncertainty networks operate in a low overfitting regime.

Bias correction does not improve performance in overparameterised models.

Bias-variance decomposition explains the observed phenomena.

Abstract

Farquhar et al. [2021] show that correcting for active learning bias with underparameterised models leads to improved downstream performance. For overparameterised models such as NNs, however, correction leads either to decreased or unchanged performance. They suggest that this is due to an "overfitting bias" which offsets the active learning bias. We show that depth uncertainty networks operate in a low overfitting regime, much like underparameterised models. They should therefore see an increase in performance with bias correction. Surprisingly, they do not. We propose that this negative result, as well as the results Farquhar et al. [2021], can be explained via the lens of the bias-variance decomposition of generalisation error.