Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agents

TL;DR

This paper introduces Researchy Questions, a new dataset of challenging, multi-perspective, non-factoid questions derived from search logs, highlighting their complexity and the benefits of decompositional answering methods for large language models.

Contribution

The paper presents a novel dataset of 100,000 multi-perspective, non-factoid questions from search logs, emphasizing their difficulty and proposing decomposition techniques to improve LLM responses.

Findings

Researchy Questions dataset contains ~100k complex, multi-perspective questions.

Decomposition into sub-questions improves LLM answering performance.

Users exhibit significant effort signals on these challenging questions.

Abstract

Existing question answering (QA) datasets are no longer challenging to most powerful Large Language Models (LLMs). Traditional QA benchmarks like TriviaQA, NaturalQuestions, ELI5 and HotpotQA mainly study ``known unknowns'' with clear indications of both what information is missing, and how to find it to answer the question. Hence, good performance on these benchmarks provides a false sense of security. A yet unmet need of the NLP community is a bank of non-factoid, multi-perspective questions involving a great deal of unclear information needs, i.e. ``unknown uknowns''. We claim we can find such questions in search engine logs, which is surprising because most question-intent queries are indeed factoid. We present Researchy Questions, a dataset of search engine queries tediously filtered to be non-factoid, ``decompositional'' and multi-perspective. We show that users spend a lot of ``effort'' on these questions in terms of signals like clicks and session length, and that they are also challenging for GPT-4. We also show that ``slow thinking'' answering techniques, like decomposition into sub-questions shows benefit over answering directly. We release $\sim$ 100k Researchy Questions, along with the Clueweb22 URLs that were clicked.