Is This Collection Worth My LLM's Time? Automatically Measuring Information Potential in Text Corpora

TL;DR

This paper introduces an automated, model-agnostic method to evaluate the potential informational value of text collections for large language models by measuring performance differences on generated MCQs.

Contribution

It presents a novel pipeline that estimates information gain from text corpora without training or fine-tuning LLMs, aiding data prioritization.

Findings

Effectively identifies valuable information-rich collections

Correlates performance gaps with information potential

Validated on diverse datasets including historical and Wikipedia texts

Abstract

As large language models (LLMs) converge towards similar capabilities, the key to advancing their performance lies in identifying and incorporating valuable new information sources. However, evaluating which text collections are worth the substantial investment required for digitization, preprocessing, and integration into LLM systems remains a significant challenge. We present a novel approach to this challenge: an automated pipeline that evaluates the potential information gain from text collections without requiring model training or fine-tuning. Our method generates multiple choice questions (MCQs) from texts and measures an LLM's performance both with and without access to the source material. The performance gap between these conditions serves as a proxy for the collection's information potential. We validate our approach using five strategically selected datasets: EPFL PhD manuscripts, a private collection of Venetian historical records, two sets of Wikipedia articles on related topics, and a synthetic baseline dataset. Our results demonstrate that this method effectively identifies collections containing valuable novel information, providing a practical tool for prioritizing data acquisition and integration efforts.