AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models

TL;DR

This paper introduces AA-Omniscience, a benchmark for evaluating large language models' factual accuracy and knowledge calibration across multiple domains, revealing persistent weaknesses and domain-specific performance variability.

Contribution

The paper presents AA-Omniscience, a novel benchmark that measures factual recall and calibration in language models across 42 topics and 6,000 questions, with a new Omniscience Index metric.

Findings

Claude 4.1 Opus scores highest among evaluated models.

Models exhibit significant factuality and calibration weaknesses.

Performance varies notably across different domains.

Abstract

Existing language model evaluations primarily measure general capabilities, yet reliable use of these models across a range of domains demands factual accuracy and recognition of knowledge gaps. We introduce AA-Omniscience, a benchmark designed to measure both factual recall and knowledge calibration across 6,000 questions. Questions are derived from authoritative academic and industry sources, and cover 42 economically relevant topics within six different domains. The evaluation measures a model's Omniscience Index, a bounded metric (-100 to 100) measuring factual recall that jointly penalizes hallucinations and rewards abstention when uncertain, with 0 equating to a model that answers questions correctly as much as it does incorrectly. Among evaluated models, Claude 4.1 Opus attains the highest score (4.8), making it one of only three models to score above zero. These results reveal persistent factuality and calibration weaknesses across frontier models. Performance also varies by domain, with the models from three different research labs leading across the six domains. This performance variability suggests models should be chosen according to the demands of the use case rather than general performance for tasks where knowledge is important.