BERT Rediscovers the Classical NLP Pipeline

TL;DR

This paper demonstrates that BERT internally encodes the traditional NLP pipeline stages in an interpretable manner, dynamically adjusting its processing based on contextual information, which enhances understanding of its linguistic capabilities.

Contribution

The study shows that BERT's internal representations align with classical NLP pipeline steps and can adaptively revise lower-level decisions using higher-level context.

Findings

BERT encodes POS tagging, parsing, NER, semantic roles, and coreference in sequence.

The model dynamically revises lower-level decisions based on higher-level information.

Representations are interpretable and localized within the network.

Abstract

Pre-trained text encoders have rapidly advanced the state of the art on many NLP tasks. We focus on one such model, BERT, and aim to quantify where linguistic information is captured within the network. We find that the model represents the steps of the traditional NLP pipeline in an interpretable and localizable way, and that the regions responsible for each step appear in the expected sequence: POS tagging, parsing, NER, semantic roles, then coreference. Qualitative analysis reveals that the model can and often does adjust this pipeline dynamically, revising lower-level decisions on the basis of disambiguating information from higher-level representations.