A Two-Stage Active Learning Algorithm for $k$-Nearest Neighbors

TL;DR

This paper introduces the first active learning algorithm specifically designed for $k$-nearest neighbor classifiers, improving training efficiency while maintaining the classifier's core voting mechanism and achieving faster convergence under certain conditions.

Contribution

It presents a novel active learning algorithm for $k$-nearest neighbors that retains the voting property and provides theoretical guarantees of improved convergence rates.

Findings

The algorithm retains the $k$-nearest neighbor voting mechanism.

Active training leads to faster convergence to the Bayes optimal classifier.

Theoretical guarantees are provided under smoothness and noise conditions.

Abstract

$k$-nearest neighbor classification is a popular non-parametric method because of desirable properties like automatic adaption to distributional scale changes. Unfortunately, it has thus far proved difficult to design active learning strategies for the training of local voting-based classifiers that naturally retain these desirable properties, and hence active learning strategies for $k$-nearest neighbor classification have been conspicuously missing from the literature. In this work, we introduce a simple and intuitive active learning algorithm for the training of $k$-nearest neighbor classifiers, the first in the literature which retains the concept of the $k$-nearest neighbor vote at prediction time. We provide consistency guarantees for a modified $k$-nearest neighbors classifier trained on samples acquired via our scheme, and show that when the conditional probability function $\mathbb{P}(Y=y|X=x)$ is sufficiently smooth and the Tsybakov noise condition holds, our actively trained classifiers converge to the Bayes optimal classifier at a faster asymptotic rate than passively trained $k$-nearest neighbor classifiers.