Improving AMR Parsing with Sequence-to-Sequence Pre-training

TL;DR

This paper introduces a sequence-to-sequence pre-training approach for AMR parsing that leverages multi-task learning, significantly improving performance on benchmark datasets and achieving state-of-the-art results with simpler models.

Contribution

It proposes a novel seq2seq pre-training and multi-task fine-tuning method specifically tailored for AMR parsing, outperforming previous models.

Findings

Pre-trained models improve AMR parsing accuracy from 71.5 to 80.2.

Joint pre-training on translation, syntax, and AMR tasks enhances performance.

Achieves state-of-the-art results with simpler seq2seq models.

Abstract

In the literature, the research on abstract meaning representation (AMR) parsing is much restricted by the size of human-curated dataset which is critical to build an AMR parser with good performance. To alleviate such data size restriction, pre-trained models have been drawing more and more attention in AMR parsing. However, previous pre-trained models, like BERT, are implemented for general purpose which may not work as expected for the specific task of AMR parsing. In this paper, we focus on sequence-to-sequence (seq2seq) AMR parsing and propose a seq2seq pre-training approach to build pre-trained models in both single and joint way on three relevant tasks, i.e., machine translation, syntactic parsing, and AMR parsing itself. Moreover, we extend the vanilla fine-tuning method to a multi-task learning fine-tuning method that optimizes for the performance of AMR parsing while endeavors to preserve the response of pre-trained models. Extensive experimental results on two English benchmark datasets show that both the single and joint pre-trained models significantly improve the performance (e.g., from 71.5 to 80.2 on AMR 2.0), which reaches the state of the art. The result is very encouraging since we achieve this with seq2seq models rather than complex models. We make our code and model available at https://github.com/xdqkid/S2S-AMR-Parser.