Graph Property Inference in Small Language Models: Effects of Representation and Reasoning Strategy

TL;DR

This study investigates how small language models infer graph properties from text, revealing limitations in reliability but also identifying how input representation and reasoning strategies can improve performance.

Contribution

It systematically analyzes the effects of input encoding and reasoning strategies on graph property inference in small instruction-tuned language models.

Findings

Small models fail to reliably estimate graph properties.

Adjacency-list encoding reduces error compared to edge-list.

Multi-branch reasoning yields measurable improvements.

Abstract

Recent progress in language modeling has expanded the range of tasks that can be approached through natural language interfaces, including problems that require structured reasoning. However, it remains unclear how effectively limited-capacity language models can infer formal properties of relational structures when those structures are presented in textual form. We conduct a systematic study of graph-theoretic property inference in small instruction-tuned language models, isolating the roles of input representation and reasoning strategy. Across a diverse set of local and global graph metrics evaluated on three models, we find that small language models fail to achieve reliable graph property estimation: normalized errors consistently exceed the intrinsic dispersion of target properties, and rank correlations remain weak across all configurations. However, the failure is structured rather than uniform. Adjacency-list encodings consistently reduce error and improve ordinal consistency relative to edge-lists, and multi-branch reasoning yields measurable aggregate gains across configurations. These results show that without task-specific fine-tuning or architectural adaptation, graph property inference in pretrained small language models remains fundamentally unreliable, but that representational organization and inference design produce consistent differences. The findings characterize the conditions under which structured inference degrades and identify which design choices yield improvements even under constrained model capacity.