Simultaneously Learning Neighborship and Projection Matrix for Supervised Dimensionality Reduction

TL;DR

This paper introduces a novel supervised dimensionality reduction method that jointly learns neighborhood relationships and the projection matrix by optimizing a unified objective, improving classification and generalization performance.

Contribution

It proposes a method that simultaneously learns neighborhood similarity and the projection matrix, addressing limitations of fixed neighborhood assumptions in high-dimensional spaces.

Findings

Effective on YALE B, COIL-100, and MNIST datasets.

Outperforms traditional methods in classification tasks.

Adaptive regularization parameter improves model flexibility.

Abstract

Explicitly or implicitly, most of dimensionality reduction methods need to determine which samples are neighbors and the similarity between the neighbors in the original highdimensional space. The projection matrix is then learned on the assumption that the neighborhood information (e.g., the similarity) is known and fixed prior to learning. However, it is difficult to precisely measure the intrinsic similarity of samples in high-dimensional space because of the curse of dimensionality. Consequently, the neighbors selected according to such similarity might and the projection matrix obtained according to such similarity and neighbors are not optimal in the sense of classification and generalization. To overcome the drawbacks, in this paper we propose to let the similarity and neighbors be variables and model them in low-dimensional space. Both the optimal similarity and projection matrix are obtained by minimizing a unified objective function. Nonnegative and sum-to-one constraints on the similarity are adopted. Instead of empirically setting the regularization parameter, we treat it as a variable to be optimized. It is interesting that the optimal regularization parameter is adaptive to the neighbors in low-dimensional space and has intuitive meaning. Experimental results on the YALE B, COIL-100, and MNIST datasets demonstrate the effectiveness of the proposed method.