Classification Constrained Dimensionality Reduction

TL;DR

This paper introduces the CCDR algorithm, which incorporates label information into dimensionality reduction, improving classification performance especially in semi-supervised and multi-class settings, with demonstrated gains on satellite imagery data.

Contribution

The paper proposes the CCDR algorithm that integrates label information into dimensionality reduction, including out-of-sample extensions for labeled and unlabeled data, and demonstrates its effectiveness.

Findings

10% improvement in k-NN classification accuracy

Effective handling of semi-supervised and multi-class data

Connection established between intrinsic dimension and embedding dimension

Abstract

Dimensionality reduction is a topic of recent interest. In this paper, we present the classification constrained dimensionality reduction (CCDR) algorithm to account for label information. The algorithm can account for multiple classes as well as the semi-supervised setting. We present an out-of-sample expressions for both labeled and unlabeled data. For unlabeled data, we introduce a method of embedding a new point as preprocessing to a classifier. For labeled data, we introduce a method that improves the embedding during the training phase using the out-of-sample extension. We investigate classification performance using the CCDR algorithm on hyper-spectral satellite imagery data. We demonstrate the performance gain for both local and global classifiers and demonstrate a 10% improvement of the $k$-nearest neighbors algorithm performance. We present a connection between intrinsic dimension estimation and the optimal embedding dimension obtained using the CCDR algorithm.