Sneaking Syntax into Transformer Language Models with Tree Regularization

TL;DR

This paper introduces TreeReg, a regularizer that injects syntactic structure into transformer language models, improving their syntactic understanding and out-of-distribution performance without altering model architecture.

Contribution

The work presents a novel differentiable regularizer, TreeReg, that softly encodes syntactic tree information into transformers, enhancing their linguistic generalization capabilities.

Findings

Up to 10% lower perplexity on out-of-distribution data.

Up to 9.5 point improvement in syntactic generalization.

Mitigates performance degradation on adversarial NLI benchmarks by 41.2 points.

Abstract

While compositional accounts of human language understanding are based on a hierarchical tree-like process, neural models like transformers lack a direct inductive bias for such tree structures. Introducing syntactic inductive biases could unlock more robust and data-efficient learning in transformer language models (LMs), but existing methods for incorporating such structure greatly restrict models, either limiting their expressivity or increasing inference complexity. This work instead aims to softly inject syntactic inductive biases into given transformer circuits, through a structured regularizer. We introduce TreeReg, an auxiliary loss function that converts bracketing decisions from silver parses into a set of differentiable orthogonality constraints on vector hidden states. TreeReg integrates seamlessly with the standard LM objective, requiring no architectural changes. LMs pre-trained with TreeReg on natural language corpora such as WikiText-103 achieve up to 10% lower perplexities on out-of-distribution data and up to 9.5 point improvements in syntactic generalization, requiring less than half the training data to outperform standard LMs. TreeReg still provides gains for pre-trained LLMs: Continued pre-training of Sheared Llama with TreeReg results in improved syntactic generalization, and fine-tuning on MultiNLI with TreeReg mitigates degradation of performance on adversarial NLI benchmarks by 41.2 points. We release all code to guide future research.