Clustering, multicollinearity, and singular vectors

TL;DR

This paper presents a mathematical approach to identify redundant features in data matrices by analyzing the structure of a specific matrix related to the pseudo-inverse, with applications in clustering and feature selection.

Contribution

It proves that the matrix S can be block-diagonalized to reveal linearly dependent column groups, aiding in feature redundancy detection.

Findings

Matrix S has a block-diagonal form after column reordering.

The method helps identify linearly dependent columns in data matrices.

Applications include improved feature selection and clustering techniques.

Abstract

Let $A$ be a matrix with its pseudo-matrix $A^{\dagger}$ and set $S=I-A^{\dagger}A$. We prove that, after re-ordering the columns of $A$, the matrix $S$ has a block-diagonal form where each block corresponds to a set of linearly dependent columns. This allows us to identify redundant columns in $A$. We explore some applications in supervised and unsupervised learning, specially feature selection, clustering, and sensitivity of solutions of least squares solutions.