Structure-aware Fine-tuning of Sequence-to-sequence Transformers for Transition-based AMR Parsing

TL;DR

This paper introduces a structure-aware fine-tuning method for sequence-to-sequence transformers in AMR parsing, improving accuracy while maintaining graph properties and simplifying the transition system.

Contribution

It proposes a simplified transition set and modeling strategies that leverage pre-trained models for better structured AMR parsing.

Findings

Achieves new state-of-the-art on AMR 2.0

Retains graph well-formedness properties

Simplifies transition-based parsing approach

Abstract

Predicting linearized Abstract Meaning Representation (AMR) graphs using pre-trained sequence-to-sequence Transformer models has recently led to large improvements on AMR parsing benchmarks. These parsers are simple and avoid explicit modeling of structure but lack desirable properties such as graph well-formedness guarantees or built-in graph-sentence alignments. In this work we explore the integration of general pre-trained sequence-to-sequence language models and a structure-aware transition-based approach. We depart from a pointer-based transition system and propose a simplified transition set, designed to better exploit pre-trained language models for structured fine-tuning. We also explore modeling the parser state within the pre-trained encoder-decoder architecture and different vocabulary strategies for the same purpose. We provide a detailed comparison with recent progress in AMR parsing and show that the proposed parser retains the desirable properties of previous transition-based approaches, while being simpler and reaching the new parsing state of the art for AMR 2.0, without the need for graph re-categorization.