NewsInterview: a Dataset and a Playground to Evaluate LLMs' Ground Gap via Informational Interviews

TL;DR

This paper introduces a large dataset of journalistic interviews and a simulated environment to evaluate and improve LLMs' strategic dialogue and grounding abilities, highlighting current limitations and future directions.

Contribution

It provides a new dataset of 40,000 interviews and a simulated environment to assess and enhance LLMs' strategic and grounding dialogue skills.

Findings

LLMs are less likely than humans to use acknowledgements and pivot to higher-level questions.

Source LLMs mimic human information sharing, but interviewer LLMs struggle with question recognition and persuasion.

Models show suboptimal information extraction, indicating a need for improved strategic dialogue capabilities.

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities in generating coherent text but often struggle with grounding language and strategic dialogue. To address this gap, we focus on journalistic interviews, a domain rich in grounding communication and abundant in data. We curate a dataset of 40,000 two-person informational interviews from NPR and CNN, and reveal that LLMs are significantly less likely than human interviewers to use acknowledgements and to pivot to higher-level questions. Realizing that a fundamental deficit exists in multi-turn planning and strategic thinking, we develop a realistic simulated environment, incorporating source personas and persuasive elements, in order to facilitate the development of agents with longer-horizon rewards. Our experiments show that while source LLMs mimic human behavior in information sharing, interviewer LLMs struggle with recognizing when questions are answered and engaging persuasively, leading to suboptimal information extraction across model size and capability. These findings underscore the need for enhancing LLMs' strategic dialogue capabilities.