SAFT: Structure-Aware Fine-Tuning of LLMs for AMR-to-Text Generation

TL;DR

SAFT is a novel fine-tuning method that incorporates graph structure into large language models for improved AMR-to-text generation, achieving state-of-the-art results without changing the model architecture.

Contribution

Introduces SAFT, a structure-aware fine-tuning technique that injects graph topology into pretrained LLMs using magnetic Laplacian-based positional encodings, enhancing structured data processing.

Findings

Sets new state-of-the-art on AMR 3.0 with 3.5 BLEU improvement

Gains increase with graph complexity

Effective without architectural modifications

Abstract

Large Language Models (LLMs) are increasingly applied to tasks involving structured inputs such as graphs. Abstract Meaning Representations (AMRs), which encode rich semantics as directed graphs, offer a rigorous testbed for evaluating LLMs on text generation from such structures. Yet, current methods often arbitrarily linearize AMRs, discarding key structural cues, or rely on architectures incompatible with standard LLMs. We introduce SAFT, a structure-aware fine-tuning approach that injects graph topology into pretrained LLMs without architectural changes. We compute direction-sensitive positional encodings from the magnetic Laplacian of transformed AMRs and project them into the embedding space of the LLM. While possibly applicable to any graph-structured inputs, we focus on AMR-to-text generation as a representative and challenging benchmark. SAFT sets a new state-of-the-art on AMR 3.0 with a 3.5 BLEU improvement over baselines. Gains scale with graph complexity, highlighting the value of structure-aware representations in enhancing LLM performance. SAFT offers a general and effective pathway for bridging structured data and language models.