Diagnosing Transformers in Task-Oriented Semantic Parsing

TL;DR

This paper investigates the strengths and weaknesses of transformer-based semantic parsers like BART and XLM-R in task-oriented settings, revealing challenges in disambiguation, syntax validity, and span extraction, but also their potential for error detection.

Contribution

It provides a comprehensive analysis of transformer-based parsers in semantic parsing, highlighting specific issues and their implications for deployment.

Findings

Transformers struggle with disambiguating intents and slots.

Transformers often produce syntactically invalid frames.

Span extraction is a major bottleneck for current models.

Abstract

Modern task-oriented semantic parsing approaches typically use seq2seq transformers to map textual utterances to semantic frames comprised of intents and slots. While these models are empirically strong, their specific strengths and weaknesses have largely remained unexplored. In this work, we study BART and XLM-R, two state-of-the-art parsers, across both monolingual and multilingual settings. Our experiments yield several key results: transformer-based parsers struggle not only with disambiguating intents/slots, but surprisingly also with producing syntactically-valid frames. Though pre-training imbues transformers with syntactic inductive biases, we find the ambiguity of copying utterance spans into frames often leads to tree invalidity, indicating span extraction is a major bottleneck for current parsers. However, as a silver lining, we show transformer-based parsers give sufficient indicators for whether a frame is likely to be correct or incorrect, making them easier to deploy in production settings.