Augmenting Researchy Questions with Sub-question Judgments

TL;DR

This paper enhances the Researchy Questions dataset by adding LLM-judged labels for sub-questions, aiming to improve retrieval models for complex information needs.

Contribution

It introduces a method to augment existing datasets with LLM-generated labels for sub-questions, facilitating better training of retrieval systems.

Findings

Sub-question labels generated using Llama3.3 70B model.

Enhanced dataset supports training for complex retrieval tasks.

Improved annotation quality for sub-questions in research datasets.

Abstract

The Researchy Questions dataset provides about 100k question queries with complex information needs that require retrieving information about several aspects of a topic. Each query in ResearchyQuestions is associated with sub-questions that were produced by prompting GPT-4. While ResearchyQuestions contains labels indicating what documents were clicked after issuing the query, there are no associations in the dataset between sub-questions and relevant documents. In this work, we augment the Researchy Questions dataset with LLM-judged labels for each sub-question using a Llama3.3 70B model. We intend these sub-question labels to serve as a resource for training retrieval models that better support complex information needs.