TabGLM: Tabular Graph Language Model for Learning Transferable Representations Through Multi-Modal Consistency Minimization

TL;DR

TabGLM is a multi-modal model that combines graph and text representations to improve learning from heterogeneous tabular data, achieving significant performance gains over existing methods.

Contribution

Introduces TabGLM, a novel multi-modal architecture that models structural and semantic information from tables using graph neural networks and text encoders with self-supervised alignment.

Findings

Achieves up to 5.56% AUC-ROC improvement over state-of-the-art methods.

Effectively processes heterogeneous datasets with fewer parameters.

Demonstrates strong performance across 25 benchmark datasets.

Abstract

Handling heterogeneous data in tabular datasets poses a significant challenge for deep learning models. While attention-based architectures and self-supervised learning have achieved notable success, their application to tabular data remains less effective over linear and tree based models. Although several breakthroughs have been achieved by models which transform tables into uni-modal transformations like image, language and graph, these models often underperform in the presence of feature heterogeneity. To address this gap, we introduce TabGLM (Tabular Graph Language Model), a novel multi-modal architecture designed to model both structural and semantic information from a table. TabGLM transforms each row of a table into a fully connected graph and serialized text, which are then encoded using a graph neural network (GNN) and a text encoder, respectively. By aligning these representations through a joint, multi-modal, self-supervised learning objective, TabGLM leverages complementary information from both modalities, thereby enhancing feature learning. TabGLM's flexible graph-text pipeline efficiently processes heterogeneous datasets with significantly fewer parameters over existing Deep Learning approaches. Evaluations across 25 benchmark datasets demonstrate substantial performance gains, with TabGLM achieving an average AUC-ROC improvement of up to 5.56% over State-of-the-Art (SoTA) tabular learning methods.