Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Later

TL;DR

This paper revisits a classical differentiable K-nearest neighbors method, enhancing it with modern deep learning techniques to establish a strong baseline for tabular data that rivals state-of-the-art models.

Contribution

It demonstrates that a simple, deep learning-enhanced NCA approach can outperform existing deep models and match top tree-based methods on diverse tabular datasets.

Findings

Deep NCA achieves performance comparable to CatBoost.

Modern training techniques significantly improve NCA results.

Deep architectures and loss functions are key to performance gains.

Abstract

The widespread enthusiasm for deep learning has recently expanded into the domain of tabular data. Recognizing that the advancement in deep tabular methods is often inspired by classical methods, e.g., integration of nearest neighbors into neural networks, we investigate whether these classical methods can be revitalized with modern techniques. We revisit a differentiable version of $K$-nearest neighbors (KNN) -- Neighbourhood Components Analysis (NCA) -- originally designed to learn a linear projection to capture semantic similarities between instances, and seek to gradually add modern deep learning techniques on top. Surprisingly, our implementation of NCA using SGD and without dimensionality reduction already achieves decent performance on tabular data, in contrast to the results of using existing toolboxes like scikit-learn. Further equipping NCA with deep representations and additional training stochasticity significantly enhances its capability, being on par with the leading tree-based method CatBoost and outperforming existing deep tabular models in both classification and regression tasks on 300 datasets. We conclude our paper by analyzing the factors behind these improvements, including loss functions, prediction strategies, and deep architectures. The code is available at https://github.com/qile2000/LAMDA-TALENT.