Depth Uncertainty Networks for Active Learning

TL;DR

This paper introduces Depth Uncertainty Networks (DUNs), a Bayesian neural network variant that infers network depth to adapt complexity during active learning, reducing overfitting and improving performance.

Contribution

The paper proposes DUNs, a novel BNN approach that dynamically infers network depth to better handle changing data complexity in active learning.

Findings

DUNs outperform other BNN variants on active learning tasks.

DUNs exhibit less overfitting compared to baselines.

DUNs adapt their complexity effectively during active learning.

Abstract

In active learning, the size and complexity of the training dataset changes over time. Simple models that are well specified by the amount of data available at the start of active learning might suffer from bias as more points are actively sampled. Flexible models that might be well suited to the full dataset can suffer from overfitting towards the start of active learning. We tackle this problem using Depth Uncertainty Networks (DUNs), a BNN variant in which the depth of the network, and thus its complexity, is inferred. We find that DUNs outperform other BNN variants on several active learning tasks. Importantly, we show that on the tasks in which DUNs perform best they present notably less overfitting than baselines.