Committee-Based Sample Selection for Probabilistic Classifiers

TL;DR

This paper introduces a committee-based sample selection method for probabilistic classifiers that reduces annotation costs by selecting the most informative examples based on model disagreement, demonstrated on NLP tagging tasks.

Contribution

It extends query-by-committee methods to probabilistic models, proposing empirical strategies for sample selection that effectively reduce labeling effort.

Findings

Significant reduction in annotation costs achieved

Simple two-member committee performs well

Sample selection reduces model complexity

Abstract

In many real-world learning tasks, it is expensive to acquire a sufficient number of labeled examples for training. This paper investigates methods for reducing annotation cost by `sample selection'. In this approach, during training the learning program examines many unlabeled examples and selects for labeling only those that are most informative at each stage. This avoids redundantly labeling examples that contribute little new information. Our work follows on previous research on Query By Committee, extending the committee-based paradigm to the context of probabilistic classification. We describe a family of empirical methods for committee-based sample selection in probabilistic classification models, which evaluate the informativeness of an example by measuring the degree of disagreement between several model variants. These variants (the committee) are drawn randomly from a probability distribution conditioned by the training set labeled so far. The method was applied to the real-world natural language processing task of stochastic part-of-speech tagging. We find that all variants of the method achieve a significant reduction in annotation cost, although their computational efficiency differs. In particular, the simplest variant, a two member committee with no parameters to tune, gives excellent results. We also show that sample selection yields a significant reduction in the size of the model used by the tagger.