TruthTensor: Evaluating LLMs through Human Imitation on Prediction Market under Drift and Holistic Reasoning

TL;DR

TruthTensor is a comprehensive evaluation framework for language models that assesses their reasoning and human imitation capabilities in dynamic, real-world environments like prediction markets, emphasizing robustness, calibration, and decision-making under drift.

Contribution

The paper introduces TruthTensor, a novel, reproducible evaluation paradigm that measures LLMs as human-imitation systems in socially-grounded, high-entropy environments, integrating probabilistic scoring and drift diagnostics.

Findings

Models with similar accuracy can differ in calibration and risk sensitivity.

Evaluation reveals divergence in model behavior under distribution shifts.

TruthTensor provides a holistic, reproducible assessment of LLMs in real-world scenarios.

Abstract

Evaluating language models and AI agents remains fundamentally challenging because static benchmarks fail to capture real-world uncertainty, distribution shift, and the gap between isolated task accuracy and human-aligned decision-making under evolving conditions. This paper introduces TruthTensor, a novel, reproducible evaluation paradigm that measures reasoning models not only as prediction engines but as human-imitation systems operating in socially-grounded, high-entropy environments. Building on forward-looking, contamination-free tasks, our framework anchors evaluation to live prediction markets and combines probabilistic scoring to provide a holistic view of model behavior. TruthTensor complements traditional correctness metrics with drift-centric diagnostics and explicit robustness checks for reproducibility. It specify human vs. automated evaluation roles, annotation protocols, and statistical testing procedures to ensure interpretability and replicability of results. In experiments across 500+ real markets (political, economic, cultural, technological), TruthTensor demonstrates that models with similar forecast accuracy can diverge markedly in calibration, drift, and risk-sensitivity, underscoring the need to evaluate models along multiple axes (accuracy, calibration, narrative stability, cost, and resource efficiency). TruthTensor therefore operationalizes modern evaluation best practices, clear hypothesis framing, careful metric selection, transparent compute/cost reporting, human-in-the-loop validation, and open, versioned evaluation contracts, to produce defensible assessments of LLMs in real-world decision contexts. We publicly released TruthTensor at https://truthtensor.com.