From Features to Graphs: Exploring Graph Structures and Pairwise Interactions via GNNs

TL;DR

This paper investigates how the structure of feature graphs influences GNN performance in modeling feature interactions, emphasizing the importance of selecting relevant edges for efficiency and interpretability.

Contribution

It introduces a theoretical framework for sparse feature graph selection using MDL, demonstrating that retaining only necessary interaction edges improves GNN effectiveness.

Findings

Edges between interacting features are crucial for GNN performance.

Non-interaction edges can introduce noise and reduce accuracy.

Sparse feature graphs based on MDL are more efficient and interpretable.

Abstract

Feature interaction is crucial in predictive machine learning models, as it captures the relationships between features that influence model performance. In this work, we focus on pairwise interactions and investigate their importance in constructing feature graphs for Graph Neural Networks (GNNs). We leverage existing GNN models and tools to explore the relationship between feature graph structures and their effectiveness in modeling interactions. Through experiments on synthesized datasets, we uncover that edges between interacting features are important for enabling GNNs to model feature interactions effectively. We also observe that including non-interaction edges can act as noise, degrading model performance. Furthermore, we provide theoretical support for sparse feature graph selection using the Minimum Description Length (MDL) principle. We prove that feature graphs retaining only necessary interaction edges yield a more efficient and interpretable representation than complete graphs, aligning with Occam's Razor. Our findings offer both theoretical insights and practical guidelines for designing feature graphs that improve the performance and interpretability of GNN models.