Improving classifier decision boundaries using nearest neighbors

TL;DR

This paper introduces a simple nearest neighbors-based method to improve neural network decision boundaries, enhancing robustness, accuracy, and interpretability without altering the model architecture or training process.

Contribution

The authors propose a straightforward algorithm that averages predictions with nearest neighbors in latent space, improving multiple neural network performance metrics.

Findings

Enhanced resistance to label noise

Improved robustness against adversarial attacks

Increased classification accuracy

Abstract

Neural networks are not learning optimal decision boundaries. We show that decision boundaries are situated in areas of low training data density. They are impacted by few training samples which can easily lead to overfitting. We provide a simple algorithm performing a weighted average of the prediction of a sample and its nearest neighbors' (computed in latent space) leading to a minor favorable outcomes for a variety of important measures for neural networks. In our evaluation, we employ various self-trained and pre-trained convolutional neural networks to show that our approach improves (i) resistance to label noise, (ii) robustness against adversarial attacks, (iii) classification accuracy, and to some degree even (iv) interpretability. While improvements are not necessarily large in all four areas, our approach is conceptually simple, i.e., improvements come without any modification to network architecture, training procedure or dataset. Furthermore, they are in stark contrast to prior works that often require trade-offs among the four objectives or provide valuable, but non-actionable insights.