A Planning Framework for Adaptive Labeling

TL;DR

This paper introduces an adaptive labeling framework using a Markov decision process to efficiently allocate measurement efforts, improving label collection strategies for scientific and engineering applications.

Contribution

It proposes a novel planning-based approach for adaptive labeling that is compatible with various uncertainty quantification methods and introduces a low-variance backpropagation estimator.

Findings

One-step lookahead policy outperforms common heuristics.

Smoothed-Autodiff reduces estimator variance with manageable bias.

Framework is effective on real and synthetic datasets.

Abstract

Ground truth labels/outcomes are critical for advancing scientific and engineering applications, e.g., evaluating the treatment effect of an intervention or performance of a predictive model. Since randomly sampling inputs for labeling can be prohibitively expensive, we introduce an adaptive labeling framework where measurement effort can be reallocated in batches. We formulate this problem as a Markov decision process where posterior beliefs evolve over time as batches of labels are collected (state transition), and batches (actions) are chosen to minimize uncertainty at the end of data collection. We design a computational framework that is agnostic to different uncertainty quantification approaches including those based on deep learning, and allows a diverse array of policy gradient approaches by relying on continuous policy parameterizations. On real and synthetic datasets, we demonstrate even a one-step lookahead policy can substantially outperform common adaptive labeling heuristics, highlighting the virtue of planning. On the methodological side, we note that standard REINFORCE-style policy gradient estimators can suffer high variance since they rely only on zeroth order information. We propose a direct backpropagation-based approach, Smoothed-Autodiff, based on a carefully smoothed version of the original non-differentiable MDP. Our method enjoys low variance at the price of introducing bias, and we theoretically and empirically show that this trade-off can be favorable.