Interpreting Answers to Yes-No Questions in User-Generated Content

TL;DR

This paper introduces a new Twitter-based corpus of yes-no question-answer pairs and analyzes linguistic features to interpret answers, revealing that large language models still struggle with this task.

Contribution

The paper presents a novel social media dataset and examines linguistic cues for interpreting yes-no answers, highlighting the limitations of current language models.

Findings

Large language models perform poorly on this task.

Linguistic characteristics can help distinguish yes, no, and unknown answers.

The new corpus enables future research in social media answer interpretation.

Abstract

Interpreting answers to yes-no questions in social media is difficult. Yes and no keywords are uncommon, and the few answers that include them are rarely to be interpreted what the keywords suggest. In this paper, we present a new corpus of 4,442 yes-no question-answer pairs from Twitter. We discuss linguistic characteristics of answers whose interpretation is yes or no, as well as answers whose interpretation is unknown. We show that large language models are far from solving this problem, even after fine-tuning and blending other corpora for the same problem but outside social media.