Relational Learning in Pre-Trained Models: A Theory from Hypergraph Recovery Perspective

TL;DR

This paper introduces a hypergraph recovery framework to analyze how pre-trained foundation models learn relational structures, providing theoretical insights into their data efficiency and capabilities.

Contribution

It formalizes relational learning in foundation models as hypergraph recovery, integrating graph theory to analyze pre-training and extending to multimodal entity alignment.

Findings

Theoretically assesses the feasibility of hypergraph recovery by pre-trained models.

Analyzes data efficiency in hypergraph recovery with minimax optimality.

Extends the framework to entity alignment in multimodal learning.

Abstract

Foundation Models (FMs) have demonstrated remarkable insights into the relational dynamics of the world, leading to the crucial question: how do these models acquire an understanding of world hybrid relations? Traditional statistical learning, particularly for prediction problems, may overlook the rich and inherently structured information from the data, especially regarding the relationships between objects. We introduce a mathematical model that formalizes relational learning as hypergraph recovery to study pre-training of FMs. In our framework, the world is represented as a hypergraph, with data abstracted as random samples from hyperedges. We theoretically examine the feasibility of a Pre-Trained Model (PTM) to recover this hypergraph and analyze the data efficiency in a minimax near-optimal style. By integrating rich graph theories into the realm of PTMs, our mathematical framework offers powerful tools for an in-depth understanding of pre-training from a unique perspective and can be used under various scenarios. As an example, we extend the framework to entity alignment in multimodal learning.