Beyond Graphs: Can Large Language Models Comprehend Hypergraphs?

TL;DR

This paper introduces a comprehensive benchmark and novel techniques to evaluate and enhance large language models' ability to understand and reason about hypergraphs, which model complex high-order relationships.

Contribution

The work presents the first extensive hypergraph benchmark for LLMs, along with new prompting methods Hyper-BAG and Hyper-COT that improve high-order reasoning performance.

Findings

Benchmark with 21,500 problems across various hypergraph tasks.

Evaluation of six prominent LLMs demonstrating model strengths and weaknesses.

Performance improvements of up to 9% on structure classification tasks.

Abstract

Existing benchmarks like NLGraph and GraphQA evaluate LLMs on graphs by focusing mainly on pairwise relationships, overlooking the high-order correlations found in real-world data. Hypergraphs, which can model complex beyond-pairwise relationships, offer a more robust framework but are still underexplored in the context of LLMs. To address this gap, we introduce LLM4Hypergraph, the first comprehensive benchmark comprising 21,500 problems across eight low-order, five high-order, and two isomorphism tasks, utilizing both synthetic and real-world hypergraphs from citation networks and protein structures. We evaluate six prominent LLMs, including GPT-4o, demonstrating our benchmark's effectiveness in identifying model strengths and weaknesses. Our specialized prompting framework incorporates seven hypergraph languages and introduces two novel techniques, Hyper-BAG and Hyper-COT, which enhance high-order reasoning and achieve an average 4% (up to 9%) performance improvement on structure classification tasks. This work establishes a foundational testbed for integrating hypergraph computational capabilities into LLMs, advancing their comprehension. The source codes are at https://github.com/iMoonLab/LLM4Hypergraph.