Interviewer-Candidate Role Play: Towards Developing Real-World NLP Systems

TL;DR

This paper introduces a multi-stage question-answering task simulating real-world interview scenarios, aiming to improve NLP systems' ability to handle uncertainty and out-of-domain inputs through staged information provision.

Contribution

It proposes a novel multi-stage task framework for NLP that incorporates real-world interaction features like clarifications and abstentions, and evaluates its effectiveness on in-domain and out-of-domain data.

Findings

Multi-stage formulation improves OOD generalization up to 72% in later stages.

Significant OOD performance gains at each stage compared to standard prediction.

Challenges remain in further enhancing OOD robustness across all stages.

Abstract

Standard NLP tasks do not incorporate several common real-world scenarios such as seeking clarifications about the question, taking advantage of clues, abstaining in order to avoid incorrect answers, etc. This difference in task formulation hinders the adoption of NLP systems in real-world settings. In this work, we take a step towards bridging this gap and present a multi-stage task that simulates a typical human-human questioner-responder interaction such as an interview. Specifically, the system is provided with question simplifications, knowledge statements, examples, etc. at various stages to improve its prediction when it is not sufficiently confident. We instantiate the proposed task in Natural Language Inference setting where a system is evaluated on both in-domain and out-of-domain (OOD) inputs. We conduct comprehensive experiments and find that the multi-stage formulation of our task leads to OOD generalization performance improvement up to 2.29% in Stage 1, 1.91% in Stage 2, 54.88% in Stage 3, and 72.02% in Stage 4 over the standard unguided prediction. However, our task leaves a significant challenge for NLP researchers to further improve OOD performance at each stage.