HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attribution

TL;DR

HAGRID is a new publicly available dataset designed to facilitate the development of generative information-seeking models that can retrieve relevant quotes and generate explanations with proper attributions, built through human and LLM collaboration.

Contribution

This paper introduces HAGRID, a novel dataset for training and evaluating attribution-capable generative information-seeking models, based on human-LLM collaboration and automatic data collection.

Findings

HAGRID enables the development of models with improved attribution capabilities.

The dataset includes human-annotated evaluations of LLM-generated explanations.

HAGRID is built on the MIRACL dataset, ensuring accessibility and relevance.

Abstract

The rise of large language models (LLMs) had a transformative impact on search, ushering in a new era of search engines that are capable of generating search results in natural language text, imbued with citations for supporting sources. Building generative information-seeking models demands openly accessible datasets, which currently remain lacking. In this paper, we introduce a new dataset, HAGRID (Human-in-the-loop Attributable Generative Retrieval for Information-seeking Dataset) for building end-to-end generative information-seeking models that are capable of retrieving candidate quotes and generating attributed explanations. Unlike recent efforts that focus on human evaluation of black-box proprietary search engines, we built our dataset atop the English subset of MIRACL, a publicly available information retrieval dataset. HAGRID is constructed based on human and LLM collaboration. We first automatically collect attributed explanations that follow an in-context citation style using an LLM, i.e. GPT-3.5. Next, we ask human annotators to evaluate the LLM explanations based on two criteria: informativeness and attributability. HAGRID serves as a catalyst for the development of information-seeking models with better attribution capabilities.