Neighborhood and Graph Constructions using Non-Negative Kernel Regression

TL;DR

This paper introduces Non-Negative Kernel Regression (NNK), a novel method for neighborhood and graph construction that improves robustness and adaptivity over traditional kNN and epsilon-neighborhood methods, enhancing machine learning performance.

Contribution

The paper presents a new perspective linking neighborhood construction to sparse signal approximation and proposes NNK, an algorithm with theoretical advantages and improved practical performance.

Findings

NNK provides more robust neighborhood and graph constructions.

NNK adapts the number of neighbors based on data properties.

NNK outperforms traditional methods in machine learning tasks.

Abstract

Data-driven neighborhood definitions and graph constructions are often used in machine learning and signal processing applications. k-nearest neighbor~(kNN) and $\epsilon$-neighborhood methods are among the most common methods used for neighborhood selection, due to their computational simplicity. However, the choice of parameters associated with these methods, such as k and $\epsilon$, is still ad hoc. We make two main contributions in this paper. First, we present an alternative view of neighborhood selection, where we show that neighborhood construction is equivalent to a sparse signal approximation problem. Second, we propose an algorithm, non-negative kernel regression~(NNK), for obtaining neighborhoods that lead to better sparse representation. NNK draws similarities to the orthogonal matching pursuit approach to signal representation and possesses desirable geometric and theoretical properties. Experiments demonstrate (i) the robustness of the NNK algorithm for neighborhood and graph construction, (ii) its ability to adapt the number of neighbors to the data properties, and (iii) its superior performance in local neighborhood and graph-based machine learning tasks.