How to Build Robust FAQ Chatbot with Controllable Question Generator?

TL;DR

This paper introduces DCSA, a controllable method for generating diverse, high-quality adversarial question-answer pairs to improve the robustness and domain generalization of FAQ chatbots.

Contribution

The paper presents DCSA, a novel semantic graph-based question generator that produces high-quality adversarial QA pairs for enhancing chatbot robustness.

Findings

Generated data improves QA model robustness to adversarial questions.

Data enhances QA model's generalization across different domains.

The method effectively fools passage retrieval models with diverse QA pairs.

Abstract

Many unanswerable adversarial questions fool the question-answer (QA) system with some plausible answers. Building a robust, frequently asked questions (FAQ) chatbot needs a large amount of diverse adversarial examples. Recent question generation methods are ineffective at generating many high-quality and diverse adversarial question-answer pairs from unstructured text. We propose the diversity controllable semantically valid adversarial attacker (DCSA), a high-quality, diverse, controllable method to generate standard and adversarial samples with a semantic graph. The fluent and semantically generated QA pairs fool our passage retrieval model successfully. After that, we conduct a study on the robustness and generalization of the QA model with generated QA pairs among different domains. We find that the generated data set improves the generalizability of the QA model to the new target domain and the robustness of the QA model to detect unanswerable adversarial questions.