Lost in Serialization: Invariance and Generalization of LLM Graph Reasoners

TL;DR

This paper investigates how Large Language Model graph reasoners are sensitive to serialization variations, analyzing their robustness and generalization, and proposing benchmarks and spectral tasks for comprehensive evaluation.

Contribution

It provides a systematic analysis of serialization invariance issues in LLM graph reasoners, introduces a decomposition framework, and evaluates robustness and generalization with new spectral tasks.

Findings

Larger models are more robust to serialization variations.

Fine-tuning reduces sensitivity to node relabeling but can increase sensitivity to structure and format changes.

Fine-tuning does not consistently improve generalization to unseen tasks.

Abstract

While promising, graph reasoners based on Large Language Models (LLMs) lack built-in invariance to symmetries in graph representations. Operating on sequential graph serializations, LLMs can produce different outputs under node reindexing, edge reordering, or formatting changes, raising robustness concerns. We systematically analyze these effects, studying how fine-tuning impacts encoding sensitivity as well generalization on unseen tasks. We propose a principled decomposition of graph serializations into node labeling, edge encoding, and syntax, and evaluate LLM robustness to variations of each of these factors on a comprehensive benchmarking suite. We also contribute a novel set of spectral tasks to further assess generalization abilities of fine-tuned reasoners. Results show that larger (non-fine-tuned) models are more robust. Fine-tuning reduces sensitivity to node relabeling but may increase it to variations in structure and format, while it does not consistently improve performance on unseen tasks.