FoGE: Fock Space inspired encoding for graph prompting

TL;DR

This paper introduces FoGE, a novel graph encoding method inspired by Fock space from physics, which enhances large language models' ability to understand and answer questions about various structured graph data with minimal modifications.

Contribution

The work presents a simple, versatile, and parameter-free graph encoder based on Fock space representations that improves graph question-answering across diverse graph types using pre-trained LLMs.

Findings

Effective encoding of various graph types including proteins and hypergraphs.

Simplifies existing graph prompting methods with minimal architectural changes.

Achieves strong performance without extensive fine-tuning or supervision.

Abstract

Recent results show that modern Large Language Models (LLM) are indeed capable of understanding and answering questions about structured data such as graphs. This new paradigm can lead to solutions that require less supervision while, at the same time, providing a model that can generalize and answer questions beyond the training labels. Existing proposals often use some description of the graph to create an ``augmented'' prompt fed to the LLM. For a chosen class of graphs, if a well-tailored graph encoder is deployed to play together with a pre-trained LLM, the model can answer graph-related questions well. Existing solutions to graph-based prompts range from graph serialization to graph transformers. In this work, we show that the use of a parameter-free graph encoder based on Fock space representations, a concept borrowed from mathematical physics, is remarkably versatile in this problem setting. The simple construction, inherited directly from the theory with a few small adjustments, can provide rich and informative graph encodings, for a wide range of different graphs. We investigate the use of this idea for prefix-tuned prompts leveraging the capabilities of a pre-trained, frozen LLM. The modifications lead to a model that can answer graph-related questions -- from simple graphs to proteins to hypergraphs -- effectively and with minimal, if any, adjustments to the architecture. Our work significantly simplifies existing solutions and generalizes well to multiple different graph-based structures effortlessly.