On the underestimation of model uncertainty by Bayesian K-nearest neighbors

TL;DR

This paper investigates Bayesian K-nearest neighbors (BKNN) and finds that it still significantly underestimates model uncertainty despite its Bayesian framework, highlighting limitations in uncertainty quantification.

Contribution

The study provides evidence that BKNN underestimates uncertainty, revealing limitations in its ability to accurately quantify model uncertainty.

Findings

BKNN underestimates model uncertainty

Evidence shows limitations in BKNN's uncertainty quantification

Highlights need for improved uncertainty estimation methods

Abstract

When using the K-nearest neighbors method, one often ignores uncertainty in the choice of K. To account for such uncertainty, Holmes and Adams (2002) proposed a Bayesian framework for K-nearest neighbors (KNN). Their Bayesian KNN (BKNN) approach uses a pseudo-likelihood function, and standard Markov chain Monte Carlo (MCMC) techniques to draw posterior samples. Holmes and Adams (2002) focused on the performance of BKNN in terms of misclassification error but did not assess its ability to quantify uncertainty. We present some evidence to show that BKNN still significantly underestimates model uncertainty.