The Chronicles of RiDiC: Generating Datasets with Controlled Popularity Distribution for Long-form Factuality Evaluation

TL;DR

This paper introduces RiDiC, a multilingual dataset generator for evaluating large language models' factuality in long-form generation, focusing on controlled entity characteristics like domain, location, and popularity.

Contribution

It presents a configurable pipeline for creating multilingual datasets with specified entity attributes, and demonstrates its use with the RiDiC dataset for factuality assessment.

Findings

LLMs hallucinate even on curated RiDiC entities.

RiDiC covers diverse domains and popularity levels.

Code and data are publicly available for further research.

Abstract

We present a configurable pipeline for generating multilingual sets of entities with specified characteristics, such as domain, geographical location and popularity, using data from Wikipedia and Wikidata. These datasets are intended for evaluating the factuality of LLMs' long-form generation, thereby complementing evaluation based on short-form QA datasets. We present the RiDiC dataset as an example of this approach. RiDiC contains 3,000 entities from three domains -- rivers, natural disasters, and car models -- spanning different popularity tiers. Each entity is accompanied by its geographical location, English and Chinese names (if available) and relevant English and Chinese Wikipedia content, which is used to evaluate LLMs' responses. Generations about RiDiC entities were obtained from three LLMs in English and Chinese. These were then evaluated using a third-party factuality checker, which showed that entities from our dataset caused even frontier models to hallucinate. To facilitate the evaluation of LLMs' long-form factuality in multiple languages, the code, data, and generation/evaluation scripts have been released.