Assessing BERT's Syntactic Abilities

TL;DR

This paper evaluates BERT's ability to understand English syntax through various stimuli, showing that BERT performs remarkably well across different syntactic phenomena and stimuli types.

Contribution

The study provides a comprehensive assessment of BERT's syntactic capabilities using natural, modified, and manually crafted stimuli, highlighting its strong performance.

Findings

BERT performs well on natural subject-verb agreement stimuli

BERT maintains high accuracy on artificially modified stimuli

BERT effectively captures reflexive anaphora phenomena

Abstract

I assess the extent to which the recently introduced BERT model captures English syntactic phenomena, using (1) naturally-occurring subject-verb agreement stimuli; (2) "coloreless green ideas" subject-verb agreement stimuli, in which content words in natural sentences are randomly replaced with words sharing the same part-of-speech and inflection; and (3) manually crafted stimuli for subject-verb agreement and reflexive anaphora phenomena. The BERT model performs remarkably well on all cases.