GiLT: Augmenting Transformer Language Models with Dependency Graphs

TL;DR

GiLT introduces a novel method to enhance Transformer language models by integrating dependency graphs directly into the attention mechanism, improving syntactic generalization and downstream task performance.

Contribution

GiLT is the first to incorporate dependency graphs into Transformers without adding structural tokens, modulating attention weights with graph features during language modeling.

Findings

GiLT with semantic dependency graphs outperforms baseline models in syntactic generalization.

GiLT maintains competitive perplexity while enhancing syntactic understanding.

Fine-tuning GiLT improves downstream task performance.

Abstract

Augmenting Transformers with linguistic structures effectively enhances the syntactic generalization performance of language models. Previous work in this direction focuses on syntactic tree structures of languages, in particular constituency tree structures. We propose Graph-Infused Layers Transformer Language Model (GiLT) which leverages dependency graphs for augmenting Transformer language models. Unlike most previous work, GiLT does not insert extra structural tokens in language modeling; instead, it injects structural information into language modeling by modulating attention weights in the Transformer with features extracted from the dependency graph that is incrementally constructed along with token prediction. In our experiments, GiLT with semantic dependency graphs achieves better syntactic generalization while maintaining competitive perplexity in comparison with Transformer language model baselines. In addition, GiLT can be finetuned from a pretrained language model to achieve improved downstream task performance. Our code is released at https://github.com/cookie-pie-oops/GiLT-LM.