Unraveling the graph structure of tabular data through Bayesian and spectral analysis

TL;DR

This paper introduces a graph-based approach combining Bayesian and spectral analysis to uncover complex relationships in tabular data, demonstrated through a socioeconomic survey case study.

Contribution

The paper presents a novel method that maps tabular data into a graph using Shapley explanations and analyzes its structure with spectral and hierarchical models.

Findings

Spectral embedding identified distinct question groups.

Hierarchical modular analysis corroborated survey design insights.

Revealed unexpected question groupings suggesting new interpretations.

Abstract

In the big-data age, tabular data are being generated and analyzed everywhere. As a consequence, finding and understanding the relationships between the features in these data are of great relevance. Here, to encompass these relationships, we propose a graph-based method that allows individual, group and multi-scale analyses. The method starts by mapping the tabular data into a weighted directed graph using the Shapley additive explanations technique. With this graph of relationships, we show that the inference of the hierarchical modular structure obtained by the Nested Stochastic Block Model (nSBM) as well as the study of the spectral space of the magnetic Laplacian can help us identify the classes of features and unravel non-trivial relationships. As a case study, we analyzed a socioeconomic survey conducted with students in Brazil: the PeNSE survey. The spectral embedding of the columns suggested that questions related to physical activities form a separate group. The application of the nSBM approach not only corroborated with that but allowed complementary findings about the modular structure: some groups of questions showed a high adherence with the divisions qualitatively defined by the designers of the survey. As opposed to the structure obtained by the spectrum, questions from the class Safety were partly grouped by our method in the class Drugs. Surprisingly, by inspecting these questions, we observed that they were related to both these topics, suggesting an alternative interpretation of these questions. These results show how our method can provide guidance for tabular data analysis as well as the design of future surveys.