Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universe

TL;DR

This paper introduces RIKER, a scalable, contamination-resistant evaluation framework for AI knowledge retrieval systems that uses synthetic document generation from known ground truth, enabling deterministic and scalable assessment.

Contribution

The paper presents RIKER, a novel benchmark and methodology that inverts the evaluation paradigm, allowing for scalable, contamination-free, and deterministic assessment of knowledge retrieval models.

Findings

Context length beyond 32K tokens degrades performance.

Cross-document aggregation is more challenging than single-document extraction.

Models can find facts but still hallucinate or fabricate information.

Abstract

Evaluating knowledge systems (LLMs, RAG, knowledge graphs, etc) faces fundamental challenges: static benchmarks are vulnerable to contamination, LLM-based judges exhibit systematic biases, and ground truth extraction requires expensive human annotation. We present RIKER (Retrieval Intelligence and Knowledge Extraction Rating), both a benchmark and a replicable methodology based on paradigm inversion - generating documents from known ground truth rather than extracting ground truth from documents. This approach enables deterministic scoring and scalable evaluation without human annotation or reference models, and contamination resistance through regenerable corpora. Our evaluation of 33 models using over 21 billion tokens reveals that context length claims frequently exceed usable capacity, with significant degradation beyond 32K tokens; cross-document aggregation proves substantially harder than single-document extraction; and grounding ability and hallucination resistance are distinct capabilities - models excelling at finding facts that exist may still fabricate facts that do not. Beyond the specific benchmark, we contribute a domain-agnostic methodology for constructing scalable and contamination-resistant evaluations wherever synthetic documents can be generated from structured ground truth.