MCAL: Minimum Cost Human-Machine Active Labeling

TL;DR

This paper introduces MCAL, an iterative hybrid human-machine labeling method that significantly reduces the total cost of dataset annotation by optimally combining human and machine labeling strategies.

Contribution

It presents a novel cost-minimization approach for hybrid labeling that adaptively chooses between human and machine labels to reduce overall annotation costs.

Findings

Achieves up to 6x cost reduction compared to full human labeling.

Always cheaper than the best existing strategies.

Validated on multiple public datasets including ImageNet.

Abstract

Today, ground-truth generation uses data sets annotated by cloud-based annotation services. These services rely on human annotation, which can be prohibitively expensive. In this paper, we consider the problem of hybrid human-machine labeling, which trains a classifier to accurately auto-label part of the data set. However, training the classifier can be expensive too. We propose an iterative approach that minimizes total overall cost by, at each step, jointly determining which samples to label using humans and which to label using the trained classifier. We validate our approach on well known public data sets such as Fashion-MNIST, CIFAR-10, CIFAR-100, and ImageNet. In some cases, our approach has 6x lower overall cost relative to human labeling the entire data set, and is always cheaper than the cheapest competing strategy.