Beyond Labels: Information-Efficient Human-in-the-Loop Learning using Ranking and Selection Queries

TL;DR

This paper introduces a human-in-the-loop learning framework that uses rich query types like ranking and exemplar selection, significantly reducing sample complexity and improving efficiency over traditional label-only methods.

Contribution

It develops probabilistic response models and active learning algorithms for rich queries, providing theoretical bounds and practical algorithms that outperform traditional approaches.

Findings

Reduces sample complexity by over 57% in sentiment classification.

Demonstrates significant efficiency gains with rich query strategies.

Provides theoretical bounds and efficient algorithms for human-in-the-loop learning.

Abstract

Integrating human expertise into machine learning systems often reduces the role of experts to labeling oracles, a paradigm that limits the amount of information exchanged and fails to capture the nuances of human judgment. We address this challenge by developing a human-in-the-loop framework to learn binary classifiers with rich query types, consisting of item ranking and exemplar selection. We first introduce probabilistic human response models for these rich queries motivated by the relationship experimentally observed between the perceived implicit score of an item and its distance to the unknown classifier. Using these models, we then design active learning algorithms that leverage the rich queries to increase the information gained per interaction. We provide theoretical bounds on sample complexity and develop a tractable and computationally efficient variational approximation. Through experiments with simulated annotators derived from crowdsourced word-sentiment and image-aesthetic datasets, we demonstrate significant reductions on sample complexity. We further extend active learning strategies to select queries that maximize information rate, explicitly balancing informational value against annotation cost. This algorithm in the word sentiment classification task reduces learning time by more than 57\% compared to traditional label-only active learning.