Improving BERT Pretraining with Syntactic Supervision

TL;DR

This paper enhances BERT pretraining by integrating syntactic supervision through token-level supertagging, aiming to improve syntactic generalization without significant computational costs.

Contribution

It introduces a simple, effective method to incorporate syntactic biases into BERT pretraining using token-level supertagging on Dutch data.

Findings

Syntax-aware model matches baseline performance.

Method adds minimal computational overhead.

Effective on smaller, automatically annotated corpora.

Abstract

Bidirectional masked Transformers have become the core theme in the current NLP landscape. Despite their impressive benchmarks, a recurring theme in recent research has been to question such models' capacity for syntactic generalization. In this work, we seek to address this question by adding a supervised, token-level supertagging objective to standard unsupervised pretraining, enabling the explicit incorporation of syntactic biases into the network's training dynamics. Our approach is straightforward to implement, induces a marginal computational overhead and is general enough to adapt to a variety of settings. We apply our methodology on Lassy Large, an automatically annotated corpus of written Dutch. Our experiments suggest that our syntax-aware model performs on par with established baselines, despite Lassy Large being one order of magnitude smaller than commonly used corpora.