Optimal Extended Neighbourhood Rule $k$ Nearest Neighbours Ensemble

TL;DR

This paper introduces an optimal extended neighborhood rule ensemble for kNN that improves neighbor selection and ensemble accuracy by considering neighbors beyond the spherical region and selecting models based on out-of-bag performance.

Contribution

It proposes a novel neighbor selection rule and an ensemble method that enhances kNN performance by optimizing neighbor inclusion and model selection criteria.

Findings

Outperforms state-of-the-art methods on 17 benchmark datasets.

Improves accuracy, Cohen's kappa, and Brier score.

Robust to added contrived features.

Abstract

The traditional k nearest neighbor (kNN) approach uses a distance formula within a spherical region to determine the k closest training observations to a test sample point. However, this approach may not work well when test point is located outside this region. Moreover, aggregating many base kNN learners can result in poor ensemble performance due to high classification errors. To address these issues, a new optimal extended neighborhood rule based ensemble method is proposed in this paper. This rule determines neighbors in k steps starting from the closest sample point to the unseen observation and selecting subsequent nearest data points until the required number of observations is reached. Each base model is constructed on a bootstrap sample with a random subset of features, and optimal models are selected based on out-of-bag performance after building a sufficient number of models. The proposed ensemble is compared with state-of-the-art methods on 17 benchmark datasets using accuracy, Cohen's kappa, and Brier score (BS). The performance of the proposed method is also assessed by adding contrived features in the original data.