Metric Embedding for Nearest Neighbor Classification

TL;DR

This paper introduces a theoretical framework for embedding arbitrary metric spaces into Euclidean space to enhance nearest neighbor classification accuracy, using regularization in a reproducing kernel Hilbert space and semidefinite programming.

Contribution

It presents a novel approach to metric embedding for NN classification, connecting it to SVMs and providing a method that outperforms Mahalanobis metric learning.

Findings

Outperforms Mahalanobis metric learning on benchmark datasets

Provides a semidefinite programming solution for embedding functions

Establishes a theoretical link between metric embedding and SVMs

Abstract

The distance metric plays an important role in nearest neighbor (NN) classification. Usually the Euclidean distance metric is assumed or a Mahalanobis distance metric is optimized to improve the NN performance. In this paper, we study the problem of embedding arbitrary metric spaces into a Euclidean space with the goal to improve the accuracy of the NN classifier. We propose a solution by appealing to the framework of regularization in a reproducing kernel Hilbert space and prove a representer-like theorem for NN classification. The embedding function is then determined by solving a semidefinite program which has an interesting connection to the soft-margin linear binary support vector machine classifier. Although the main focus of this paper is to present a general, theoretical framework for metric embedding in a NN setting, we demonstrate the performance of the proposed method on some benchmark datasets and show that it performs better than the Mahalanobis metric learning algorithm in terms of leave-one-out and generalization errors.