Exploiting Pre-trained Encoder-Decoder Transformers for Sequence-to-Sequence Constituent Parsing

TL;DR

This paper explores using pre-trained encoder-decoder models like BART, mBART, and T5 for sequence-to-sequence constituent parsing, showing they outperform previous models and compete with task-specific parsers.

Contribution

It extends the sequence-to-sequence parsing framework by thoroughly investigating pre-trained encoder-decoder models for constituency parsing.

Findings

Pre-trained encoder-decoder models outperform prior sequence-to-sequence parsers.

The approach achieves competitive results with task-specific parsers on continuous benchmarks.

Extensive evaluation across various linearization strategies demonstrates robustness.

Abstract

To achieve deep natural language understanding, syntactic constituent parsing plays a crucial role and is widely required by many artificial intelligence systems for processing both text and speech. A recent approach involves using standard sequence-to-sequence models to handle constituent parsing as a machine translation problem, moving away from traditional task-specific parsers. These models are typically initialized with pre-trained encoder-only language models like BERT or RoBERTa. However, the use of pre-trained encoder-decoder language models for constituency parsing has not been thoroughly explored. To bridge this gap, we extend the sequence-to-sequence framework by investigating parsers built on pre-trained encoder-decoder architectures, including BART, mBART, and T5. We fine-tune them to generate linearized parse trees and extensively evaluate them on different linearization strategies across both continuous treebanks and more complex discontinuous benchmarks. Our results demonstrate that our approach outperforms all prior sequence-to-sequence models and performs competitively with leading task-specific constituent parsers on continuous constituent parsing.