Variability of qualitative variables: A Hilbert space formulation

TL;DR

This paper introduces a novel Hilbert space formalism for analyzing qualitative variable diversity, enabling robust, interpretable category reduction and handling simultaneous categories in data analysis.

Contribution

It presents a new mathematical framework using Hilbert spaces for diversity measures, including strategies for category reduction and robustness against uncertainty.

Findings

Formalism effectively addresses category reduction issues.

Normalization of diversity measures improves interpretability.

Framework accommodates simultaneous categories in data analysis.

Abstract

A new formalism to express and operate on diversity measures of qualitative variables, built in a Hilbert space, is presented. The abstract character of the Hilbert space naturally incorporates the equivalence between qualitative variables and is utilized here to (i) represent the binary character of answers to categories and (ii) introduce a new criterium for choosing between different measures of diversity, namely, robustness against uncertainty. The full potential of the formulation on a Hilbert space comes to play when addressing the reduction of categories problem, a common problem in data analysis. The present formalism solves the problem by incorporating strategies inspired by mathematical and physical statistics, specifically, it makes use of the concept of partial trace. In solving this problem, it is shown that properly normalizing the diversity measures is instrumental to provide a sensible interpretation of the results when the reduction of categories is performed. Finally, the approach presented here also allows for straightforwardly measuring diversity and performing category reduction in situations when simultaneous categories could be chosen.