SLASH the Sink: Sharpening Structural Attention Inside LLMs

TL;DR

This paper reveals that LLMs internally reconstruct graph structures and introduces a training-free method, SLASH, to enhance their structural understanding, leading to improved performance on graph-related tasks.

Contribution

The paper uncovers the intrinsic graph reconstruction ability of LLMs and proposes SLASH, a novel attention sharpening technique that boosts structural reasoning without additional training.

Findings

SLASH significantly improves LLMs' performance on graph tasks.

LLMs naturally reconstruct graph topology internally.

Attention redistribution enhances structural understanding in LLMs.

Abstract

Large Language Models (LLMs) show remarkable semantic understanding but often struggle with structural understanding when processing graph topologies in a serialized format. Existing solutions rely on training external graph-based adapters or fine-tuning, which incur high costs and lost generalizability. In this work, we investigate the internal mechanisms of LLMs and present a critical finding: LLMs spontaneously reconstruct the graph's topology internally, evidenced by a distinct "sawtooth" pattern in their attention maps that structurally aligns with the "token-level adjacency matrix". However, this intrinsic structural understanding is diluted by the attention sink. We theoretically formalize this dilution as a representation bottleneck, stemming from a fundamental conflict: the model's anisotropic bias, essential for language tasks, suppresses the topology-aware local aggregation required for graph reasoning. To address this, we propose a training-free solution, named StructuraL Attention SHarpening (SLASH), which amplifies this internal structural understanding via a plug-and-play attention redistribution. Experiments on pure graph tasks and molecular prediction validate that SLASH delivers significant and consistent performance gains across diverse LLMs.