Gated Tree Cross-Attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMs

TL;DR

This paper introduces GTCA, a method to improve the syntactic robustness of decoder-only large language models by injecting explicit structure without altering the original checkpoint.

Contribution

GTCA provides a checkpoint-compatible approach to enhance syntactic robustness in decoder-only LLMs through gated cross-attention and staged training.

Findings

GTCA improves syntactic robustness across benchmarks.

GTCA does not harm QA or reasoning performance.

GTCA is compatible with existing checkpoints.

Abstract

Decoder-only large language models achieve strong broad performance but are brittle to minor grammatical perturbations, undermining reliability for downstream reasoning. However, directly injecting explicit syntactic structure into an existing checkpoint can interfere with its pretrained competence. We introduce a checkpoint-compatible gated tree cross-attention (GTCA) branch that reads precomputed constituency chunk memory while leaving backbone architecture unchanged. Our design uses a token update mask and staged training to control the scope and timing of structural updates. Across benchmarks and Transformer backbones, GTCA strengthens syntactic robustness beyond continued-training baselines without compromising Multiple-Choice QA performance or commonsense reasoning, providing a practical checkpoint-compatible route to more syntax-robust decoder-only LLMs.