The extension of Pearson correlation coefficient, measuring noise, and selecting features

TL;DR

This paper extends Pearson's correlation coefficient to multiple variables using random matrix theory, demonstrating its effectiveness in noise measurement and feature selection for classification tasks.

Contribution

It introduces a novel multivariable extension of Pearson's correlation coefficient based on random matrix theory, addressing limitations of the traditional measure.

Findings

Extended correlation measure effectively gauges noise in multivariable data

Improves feature selection accuracy in classification tasks

Provides a new tool for risk assessment in multi-asset portfolios

Abstract

Not a matter of serious contention, Pearson's correlation coefficient is still the most important statistical association measure. Restricted to just two variables, this measure sometimes doesn't live up to users' needs and expectations. Specifically, a multivariable version of the correlation coefficient can greatly contribute to better assessment of the risk in a multi-asset investment portfolio. Needless to say, the correlation coefficient is derived from another concept: covariance. Even though covariance can be extended naturally by its mathematical formula, such an extension is to no use. Making matters worse, the correlation coefficient can never be extended based on its mathematical definition. In this article, we briefly explore random matrix theory to extend the notion of Pearson's correlation coefficient to an arbitrary number of variables. Then, we show that how useful this measure is at gauging noise, thereby selecting features particularly in classification.