Addressing practical challenges in Active Learning via a hybrid query strategy

TL;DR

This paper introduces a hybrid query strategy for Active Learning that effectively addresses cold-start, oracle uncertainty, and performance evaluation challenges in practical machine learning applications.

Contribution

It proposes a novel AL framework combining pre-clustering and heuristics to handle real-world challenges, validated across diverse industrial environments.

Findings

Successfully mitigates cold-start problem using pre-clustering

Incorporates labeler uncertainty to improve query reliability

Demonstrates robustness across multiple real-world settings

Abstract

Active Learning (AL) is a powerful tool to address modern machine learning problems with significantly fewer labeled training instances. However, implementation of traditional AL methodologies in practical scenarios is accompanied by multiple challenges due to the inherent assumptions. There are several hindrances, such as unavailability of labels for the AL algorithm at the beginning; unreliable external source of labels during the querying process; or incompatible mechanisms to evaluate the performance of Active Learner. Inspired by these practical challenges, we present a hybrid query strategy-based AL framework that addresses three practical challenges simultaneously: cold-start, oracle uncertainty and performance evaluation of Active Learner in the absence of ground truth. While a pre-clustering approach is employed to address the cold-start problem, the uncertainty surrounding the expertise of labeler and confidence in the given labels is incorporated to handle oracle uncertainty. The heuristics obtained during the querying process serve as the fundamental premise for accessing the performance of Active Learner. The robustness of the proposed AL framework is evaluated across three different environments and industrial settings. The results demonstrate the capability of the proposed framework to tackle practical challenges during AL implementation in real-world scenarios.