"I'd rather just go to bed": Understanding Indirect Answers

TL;DR

This paper introduces 'Circa', a large-scale corpus of indirect answers in English dialog, and develops BERT-based models to classify these pragmatic responses, highlighting current limitations and progress in understanding indirect communication.

Contribution

The paper provides the first large-scale corpus for indirect answers and proposes neural models for classifying pragmatic responses in dialog systems.

Findings

Models achieve 82-88% accuracy on 4-class classification.

Transfer learning from entailment improves performance.

Current models are not yet robust enough for practical dialog systems.

Abstract

We revisit a pragmatic inference problem in dialog: understanding indirect responses to questions. Humans can interpret 'I'm starving.' in response to 'Hungry?', even without direct cue words such as 'yes' and 'no'. In dialog systems, allowing natural responses rather than closed vocabularies would be similarly beneficial. However, today's systems are only as sensitive to these pragmatic moves as their language model allows. We create and release the first large-scale English language corpus 'Circa' with 34,268 (polar question, indirect answer) pairs to enable progress on this task. The data was collected via elaborate crowdsourcing, and contains utterances with yes/no meaning, as well as uncertain, middle-ground, and conditional responses. We also present BERT-based neural models to predict such categories for a question-answer pair. We find that while transfer learning from entailment works reasonably, performance is not yet sufficient for robust dialog. Our models reach 82-88% accuracy for a 4-class distinction, and 74-85% for 6 classes.