Enhanced Nearest Neighbor Classification for Crowdsourcing

TL;DR

This paper introduces an enhanced nearest neighbor classifier tailored for crowdsourcing scenarios, effectively handling noisy labels by estimating worker quality and achieving near-oracle performance.

Contribution

It proposes two algorithms for estimating worker quality in crowdsourcing, improving classification accuracy despite label noise, with theoretical regret guarantees.

Findings

Algorithms achieve regret comparable to oracle methods.

Proven lower bound on sample size for optimal regret convergence.

Numerical experiments validate the effectiveness of the methods.

Abstract

In machine learning, crowdsourcing is an economical way to label a large amount of data. However, the noise in the produced labels may deteriorate the accuracy of any classification method applied to the labelled data. We propose an enhanced nearest neighbor classifier (ENN) to overcome this issue. Two algorithms are developed to estimate the worker quality (which is often unknown in practice): one is to construct the estimate based on the denoised worker labels by applying the $k$NN classifier to the expert data; the other is an iterative algorithm that works even without access to the expert data. Other than strong numerical evidence, our proposed methods are proven to achieve the same regret as its oracle version based on high-quality expert data. As a technical by-product, a lower bound on the sample size assigned to each worker to reach the optimal convergence rate of regret is derived.