Transition-based Abstract Meaning Representation Parsing with Contextual Embeddings

TL;DR

This paper investigates integrating pretrained contextual embeddings like BERT into a transition-based AMR parser, finding that while embeddings alone don't improve performance, combined with concept info they enhance robustness and performance.

Contribution

The paper introduces AmrBerger, a new AMR parser that incorporates contextual embeddings, and provides insights into their strengths and limitations in semantic parsing.

Findings

Contextual embeddings alone do not improve SMATCH scores.

Adding concept information boosts parser performance.

Embeddings increase robustness against syntactic feature removal.

Abstract

The ability to understand and generate languages sets human cognition apart from other known life forms'. We study a way of combing two of the most successful routes to meaning of language--statistical language models and symbolic semantics formalisms--in the task of semantic parsing. Building on a transition-based, Abstract Meaning Representation (AMR) parser, AmrEager, we explore the utility of incorporating pretrained context-aware word embeddings--such as BERT and RoBERTa--in the problem of AMR parsing, contributing a new parser we dub as AmrBerger. Experiments find these rich lexical features alone are not particularly helpful in improving the parser's overall performance as measured by the SMATCH score when compared to the non-contextual counterpart, while additional concept information empowers the system to outperform the baselines. Through lesion study, we found the use of contextual embeddings helps to make the system more robust against the removal of explicit syntactical features. These findings expose the strength and weakness of the contextual embeddings and the language models in the current form, and motivate deeper understanding thereof.