Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction

TL;DR

This paper introduces a simple method to extract constituency trees from pre-trained language models, revealing their ability to capture syntactic structures like adverb phrases without additional training.

Contribution

The authors propose a novel, training-free approach for analyzing the syntactic knowledge of pre-trained LMs, demonstrating their effectiveness in constituency parsing tasks.

Findings

Pre-trained LMs outperform other methods in identifying adverb phrases.

The method effectively extracts syntactic trees without additional training.

Pre-trained LMs show significant syntactic awareness in constituency boundaries.

Abstract

With the recent success and popularity of pre-trained language models (LMs) in natural language processing, there has been a rise in efforts to understand their inner workings. In line with such interest, we propose a novel method that assists us in investigating the extent to which pre-trained LMs capture the syntactic notion of constituency. Our method provides an effective way of extracting constituency trees from the pre-trained LMs without training. In addition, we report intriguing findings in the induced trees, including the fact that pre-trained LMs outperform other approaches in correctly demarcating adverb phrases in sentences.