Interview: A Large-Scale Open-Source Corpus of Media Dialog

TL;DR

This paper introduces 'Interview', a large-scale media dialog dataset from news interviews, which improves dialog modeling and response specificity in conversational AI systems.

Contribution

The paper presents a new extensive media dialog dataset with speaker role annotations, enhancing the training and performance of dialog systems in real-world scenarios.

Findings

Models trained on 'Interview' outperform others in zero-shot out-of-domain tasks.

Speaker role annotations improve dialog system responsiveness and specificity.

The dataset enables more natural and engaging interview-style conversations.

Abstract

Existing conversational datasets consist either of written proxies for dialog or small-scale transcriptions of natural speech. We introduce 'Interview': a large-scale (105K conversations) media dialog dataset collected from news interview transcripts. Compared to existing large-scale proxies for conversational data, language models trained on our dataset exhibit better zero-shot out-of-domain performance on existing spoken dialog datasets, demonstrating its usefulness in modeling real-world conversations. 'Interview' contains speaker role annotations for each turn, facilitating the development of engaging, responsive dialog systems. In fact, experiments on two dialog tasks show that leveraging such labels improves performance over strong speaker-agnostic baselines, and enabling models to generate more specific and inquisitive responses in interview-style conversations.