Active Learning for Cost-Sensitive Classification

TL;DR

This paper introduces COAL, an active learning algorithm for cost-sensitive multiclass classification that efficiently reduces labeling effort by selectively querying labels based on cost predictions, with proven guarantees and empirical success.

Contribution

The paper presents COAL, a novel active learning algorithm tailored for cost-sensitive classification that leverages regression to minimize label queries and optimize cost prediction accuracy.

Findings

COAL outperforms passive learning and baselines in reducing labeling effort.

The algorithm has strong theoretical guarantees for predictive performance.

Empirical results show significant cost savings on real-world datasets.

Abstract

We design an active learning algorithm for cost-sensitive multiclass classification: problems where different errors have different costs. Our algorithm, COAL, makes predictions by regressing to each label's cost and predicting the smallest. On a new example, it uses a set of regressors that perform well on past data to estimate possible costs for each label. It queries only the labels that could be the best, ignoring the sure losers. We prove COAL can be efficiently implemented for any regression family that admits squared loss optimization; it also enjoys strong guarantees with respect to predictive performance and labeling effort. We empirically compare COAL to passive learning and several active learning baselines, showing significant improvements in labeling effort and test cost on real-world datasets.