"I Never Said That": A dataset, taxonomy and baselines on response clarity classification

TL;DR

This paper introduces a new dataset, taxonomy, and baseline models for classifying response clarity in political interview QA pairs, addressing ambiguity and evasion in political discourse using LLMs and human annotation.

Contribution

It presents a novel two-level taxonomy and a curated dataset for response clarity classification, combining LLMs and human annotations to establish baselines.

Findings

Established baseline performance for response clarity classification.

Analyzed effectiveness of different model architectures and sizes.

Provided insights into evasion techniques in political responses.

Abstract

Equivocation and ambiguity in public speech are well-studied discourse phenomena, especially in political science and analysis of political interviews. Inspired by the well-grounded theory on equivocation, we aim to resolve the closely related problem of response clarity in questions extracted from political interviews, leveraging the capabilities of Large Language Models (LLMs) and human expertise. To this end, we introduce a novel taxonomy that frames the task of detecting and classifying response clarity and a corresponding clarity classification dataset which consists of question-answer (QA) pairs drawn from political interviews and annotated accordingly. Our proposed two-level taxonomy addresses the clarity of a response in terms of the information provided for a given question (high-level) and also provides a fine-grained taxonomy of evasion techniques that relate to unclear, ambiguous responses (lower-level). We combine ChatGPT and human annotators to collect, validate and annotate discrete QA pairs from political interviews, to be used for our newly introduced response clarity task. We provide a detailed analysis and conduct several experiments with different model architectures, sizes and adaptation methods to gain insights and establish new baselines over the proposed dataset and task.