Retrofitting Structure-aware Transformer Language Model for End Tasks

TL;DR

This paper introduces a retrofitted structure-aware Transformer language model that incorporates syntactic distances to improve performance on end tasks, achieving better perplexity and syntactic phrase accuracy.

Contribution

It proposes a novel method to integrate syntactic structure into Transformer models using syntactic distances and a middle-layer structural learning strategy.

Findings

Improved perplexity in language modeling.

Enhanced syntactic phrase induction.

Significant gains in semantic and syntactic tasks.

Abstract

We consider retrofitting structure-aware Transformer-based language model for facilitating end tasks by proposing to exploit syntactic distance to encode both the phrasal constituency and dependency connection into the language model. A middle-layer structural learning strategy is leveraged for structure integration, accomplished with main semantic task training under multi-task learning scheme. Experimental results show that the retrofitted structure-aware Transformer language model achieves improved perplexity, meanwhile inducing accurate syntactic phrases. By performing structure-aware fine-tuning, our model achieves significant improvements for both semantic- and syntactic-dependent tasks.