Discontinuous Constituency and BERT: A Case Study of Dutch

TL;DR

This study evaluates BERT's ability to handle complex Dutch syntactic patterns that are mildly context-sensitive, revealing limitations in its implicit understanding of certain linguistic dependencies.

Contribution

It introduces a novel test suite based on formal grammars to assess BERT's syntactic capabilities on non-context free Dutch structures.

Findings

BERT struggles with control verb nesting.

BERT fails to capture verb raising dependencies.

Extensive analysis shows limitations in BERT's syntactic understanding.

Abstract

In this paper, we set out to quantify the syntactic capacity of BERT in the evaluation regime of non-context free patterns, as occurring in Dutch. We devise a test suite based on a mildly context-sensitive formalism, from which we derive grammars that capture the linguistic phenomena of control verb nesting and verb raising. The grammars, paired with a small lexicon, provide us with a large collection of naturalistic utterances, annotated with verb-subject pairings, that serve as the evaluation test bed for an attention-based span selection probe. Our results, backed by extensive analysis, suggest that the models investigated fail in the implicit acquisition of the dependencies examined.