Unsupervised and Few-shot Parsing from Pretrained Language Models

TL;DR

This paper introduces unsupervised and few-shot parsing methods leveraging pretrained language models' self-attention, achieving competitive results on English and cross-lingual datasets with minimal supervision.

Contribution

It proposes UPOA and UPIO models that use self-attention weights for unsupervised parsing, and extends them to few-shot parsing with limited annotated data.

Findings

UPIO achieves state-of-the-art results on short sentences.

FPIO outperforms previous few-shot methods with only 20 annotated trees.

Both methods outperform previous approaches on most cross-lingual datasets.

Abstract

Pretrained language models are generally acknowledged to be able to encode syntax [Tenney et al., 2019, Jawahar et al., 2019, Hewitt and Manning, 2019]. In this article, we propose UPOA, an Unsupervised constituent Parsing model that calculates an Out Association score solely based on the self-attention weight matrix learned in a pretrained language model as the syntactic distance for span segmentation. We further propose an enhanced version, UPIO, which exploits both inside association and outside association scores for estimating the likelihood of a span. Experiments with UPOA and UPIO disclose that the linear projection matrices for the query and key in the self-attention mechanism play an important role in parsing. We therefore extend the unsupervised models to few-shot parsing models (FPOA, FPIO) that use a few annotated trees to learn better linear projection matrices for parsing. Experiments on the Penn Treebank demonstrate that our unsupervised parsing model UPIO achieves results comparable to the state of the art on short sentences (length <= 10). Our few-shot parsing model FPIO trained with only 20 annotated trees outperforms a previous few-shot parsing method trained with 50 annotated trees. Experiments on cross-lingual parsing show that both unsupervised and few-shot parsing methods are better than previous methods on most languages of SPMRL [Seddah et al., 2013].