Efficient Bayesian Updates for Deep Active Learning via Laplace Approximations

TL;DR

This paper introduces an efficient Bayesian update method using Laplace approximations to improve deep active learning, reducing computational costs while maintaining performance, and enabling sequential batch selection strategies.

Contribution

It presents a novel second-order Bayesian update for deep neural networks that approximates retraining efficiently, facilitating better batch selection in active learning.

Findings

The proposed update closely matches retraining performance.

It significantly reduces computational complexity.

Enables sequential and look-ahead batch selection strategies.

Abstract

Deep active learning (AL) selects batches of instances for annotation to avoid retraining deep neural networks (DNNs) after each new label. Employing a naive top-$b$ selection can result in a batch of redundant (similar) instances. To address this, various AL strategies employ clustering techniques that ensure diversity within a batch. We approach this issue by substituting the costly retraining with an efficient Bayesian update. Our proposed update represents a second-order optimization step using the Gaussian posterior from a last-layer Laplace approximation. Thereby, we achieve low computational complexity by computing the inverse Hessian in closed form. We demonstrate that in typical AL settings, our update closely approximates retraining while being considerably faster. Leveraging our update, we introduce a new framework for batch selection through sequential construction, updating the DNN after each label acquisition. Furthermore, we incorporate our update into a look-ahead selection strategy as a feasible upper baseline approximating optimal batch selection. Our results highlight the potential of efficient updates to advance deep AL research.