Basic statistics for probabilistic symbolic variables: a novel metric-based approach

TL;DR

This paper introduces a novel metric-based approach using Wasserstein distance to compute basic statistics for distributional data, enabling improved analysis of multi-valued variables in data mining.

Contribution

It extends classic inertia measures with Wasserstein distance for distributional data, providing new tools for variability and association analysis in multi-valued variables.

Findings

Proves properties of the Wasserstein distance in this context

Demonstrates the approach with a clustering algorithm on real data

Shows the effectiveness of the new statistics for distributional data

Abstract

In data mining, it is usually to describe a set of individuals using some summaries (means, standard deviations, histograms, confidence intervals) that generalize individual descriptions into a typology description. In this case, data can be described by several values. In this paper, we propose an approach for computing basic statics for such data, and, in particular, for data described by numerical multi-valued variables (interval, histograms, discrete multi-valued descriptions). We propose to treat all numerical multi-valued variables as distributional data, i.e. as individuals described by distributions. To obtain new basic statistics for measuring the variability and the association between such variables, we extend the classic measure of inertia, calculated with the Euclidean distance, using the squared Wasserstein distance defined between probability measures. The distance is a generalization of the Wasserstein distance, that is a distance between quantile functions of two distributions. Some properties of such a distance are shown. Among them, we prove the Huygens theorem of decomposition of the inertia. We show the use of the Wasserstein distance and of the basic statistics presenting a k-means like clustering algorithm, for the clustering of a set of data described by modal numerical variables (distributional variables), on a real data set. Keywords: Wasserstein distance, inertia, dependence, distributional data, modal variables.