Large-Margin kNN Classification Using a Deep Encoder Network

TL;DR

This paper introduces DNet-kNN, a scalable deep neural network-based non-linear feature mapping method that enhances large-margin kNN classification and dimensionality reduction, outperforming previous linear and autoencoder-based approaches on benchmark datasets.

Contribution

It presents a novel deep neural network approach for non-linear feature transformation that improves large-margin kNN classification and scalability.

Findings

DNet-kNN outperforms linear mapping large-margin kNN.

DNet-kNN surpasses autoencoder-based kNN on benchmarks.

The method is scalable to large datasets.

Abstract

KNN is one of the most popular classification methods, but it often fails to work well with inappropriate choice of distance metric or due to the presence of numerous class-irrelevant features. Linear feature transformation methods have been widely applied to extract class-relevant information to improve kNN classification, which is very limited in many applications. Kernels have been used to learn powerful non-linear feature transformations, but these methods fail to scale to large datasets. In this paper, we present a scalable non-linear feature mapping method based on a deep neural network pretrained with restricted boltzmann machines for improving kNN classification in a large-margin framework, which we call DNet-kNN. DNet-kNN can be used for both classification and for supervised dimensionality reduction. The experimental results on two benchmark handwritten digit datasets show that DNet-kNN has much better performance than large-margin kNN using a linear mapping and kNN based on a deep autoencoder pretrained with retricted boltzmann machines.