Big Variates: Visualizing and identifying key variables in a multivariate world

TL;DR

This paper introduces a novel information-theoretic approach with algorithms to visualize, identify key variables, and assess classification limits in high-dimensional Big Data, enhancing multivariate data analysis.

Contribution

It presents new information-theoretic statistics and a histogram algorithm to identify key variables and evaluate classification potential in large multivariate datasets.

Findings

Key variables identified using the proposed statistics

Histogram algorithm quantifies data information content

Class Distance Indicator predicts maximum classification performance

Abstract

Big Data involves both a large number of events but also many variables. This paper will concentrate on the challenge presented by the large number of variables in a Big Dataset. It will start with a brief review of exploratory data visualisation for large dimensional datasets and the use of parallel coordinates. This motivates the use of information theoretic ideas to understand multivariate data. Two key information-theoretic statistics (Similarity Index and Class Distance Indicator) will be described which are used to identify the key variables and then guide the user in a subsequent machine learning analysis. Key to the approach is a novel algorithm to histogram data which quantifies the information content of the data. The Class Distance Indicator also sets a limit on the classification performance of machine learning algorithms for the specific dataset.