Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models

TL;DR

Dependency Transformer Grammars (DTGs) introduce explicit dependency-based structures into Transformer language models, improving syntactic generalization and outperforming constituency-based models by integrating dependency trees through modified attention and encoding strategies.

Contribution

The paper presents Dependency Transformer Grammars, a novel Transformer architecture that incorporates dependency structures via attention masks and encoding, enhancing syntactic modeling.

Findings

DTGs achieve better generalization on dependency-annotated datasets.

DTGs outperform recent constituency-based models in syntactic tasks.

DTGs maintain comparable perplexity with baseline Transformer models.

Abstract

Syntactic Transformer language models aim to achieve better generalization through simultaneously modeling syntax trees and sentences. While prior work has been focusing on adding constituency-based structures to Transformers, we introduce Dependency Transformer Grammars (DTGs), a new class of Transformer language model with explicit dependency-based inductive bias. DTGs simulate dependency transition systems with constrained attention patterns by modifying attention masks, incorporate the stack information through relative positional encoding, and augment dependency arc representation with a combination of token embeddings and operation embeddings. When trained on a dataset of sentences annotated with dependency trees, DTGs achieve better generalization while maintaining comparable perplexity with Transformer language model baselines. DTGs also outperform recent constituency-based models, showing that dependency can better guide Transformer language models. Our code is released at https://github.com/zhaoyd1/Dep_Transformer_Grammars.