Learning User's confidence for active learning

TL;DR

This paper introduces a filtering scheme that learns user confidence to improve active learning efficiency in image labeling, reducing unhelpful queries and enhancing practical applicability.

Contribution

It proposes a novel classifier-based filtering method to estimate user confidence, addressing the contradiction between uncertainty-based heuristics and user labeling ability.

Findings

The filtering scheme maximizes useful queries in active learning.

Experiments show improved efficiency on multi-resolution QuickBird images.

The method adapts to different user confidence levels and image resolutions.

Abstract

In this paper, we study the applicability of active learning in operative scenarios: more particularly, we consider the well-known contradiction between the active learning heuristics, which rank the pixels according to their uncertainty, and the user's confidence in labeling, which is related to both the homogeneity of the pixel context and user's knowledge of the scene. We propose a filtering scheme based on a classifier that learns the confidence of the user in labeling, thus minimizing the queries where the user would not be able to provide a class for the pixel. The capacity of a model to learn the user's confidence is studied in detail, also showing the effect of resolution is such a learning task. Experiments on two QuickBird images of different resolutions (with and without pansharpening) and considering committees of users prove the efficiency of the filtering scheme proposed, which maximizes the number of useful queries with respect to traditional active learning.