On Robustness of Neural Semantic Parsers

TL;DR

This paper empirically investigates the robustness of neural semantic parsers against adversarial attacks, proposing a scalable testing methodology and analyzing the impact of data augmentation on their performance.

Contribution

It introduces a scalable method to create robustness test sets for semantic parsers and provides an empirical evaluation of their vulnerability to adversarial examples.

Findings

State-of-the-art parsers are vulnerable to adversarial perturbations.

Data augmentation can improve robustness but does not eliminate vulnerabilities.

The methodology enables systematic robustness evaluation of semantic parsers.

Abstract

Semantic parsing maps natural language (NL) utterances into logical forms (LFs), which underpins many advanced NLP problems. Semantic parsers gain performance boosts with deep neural networks, but inherit vulnerabilities against adversarial examples. In this paper, we provide the empirical study on the robustness of semantic parsers in the presence of adversarial attacks. Formally, adversaries of semantic parsing are considered to be the perturbed utterance-LF pairs, whose utterances have exactly the same meanings as the original ones. A scalable methodology is proposed to construct robustness test sets based on existing benchmark corpora. Our results answered five research questions in measuring the sate-of-the-art parsers' performance on robustness test sets, and evaluating the effect of data augmentation.