Minimax Optimal Algorithms with Fixed-$k$-Nearest Neighbors

TL;DR

This paper develops distributed minimax optimal algorithms for classification, regression, and density estimation using fixed-$k$-nearest neighbors, achieving near-optimal error rates with scalable data splitting.

Contribution

It introduces optimal aggregation rules for fixed-$k$-NN in distributed settings, attaining minimax optimal rates for various estimation tasks.

Findings

Distributed fixed-$k$-NN algorithms achieve minimax optimal error rates.

Performance comparable to standard $ heta(kM)$-NN rules with fixed $k$.

Algorithms are effective in large-scale, distributed data scenarios.

Abstract

This paper presents how to perform minimax optimal classification, regression, and density estimation based on fixed-$k$ nearest neighbor (NN) searches. We consider a distributed learning scenario, in which a massive dataset is split into smaller groups, where the $k$-NNs are found for a query point with respect to each subset of data. We propose \emph{optimal} rules to aggregate the fixed-$k$-NN information for classification, regression, and density estimation that achieve minimax optimal rates for the respective problems. We show that the distributed algorithm with a fixed $k$ over a sufficiently large number of groups attains a minimax optimal error rate up to a multiplicative logarithmic factor under some regularity conditions. Roughly speaking, distributed $k$-NN rules with $M$ groups has a performance comparable to the standard $\Theta(kM)$-NN rules even for fixed $k$.