No Free Delivery Service: Epistemic limits of passive data collection in complex social systems

TL;DR

This paper demonstrates that the standard train-test paradigm for model validation is fundamentally invalid in complex social systems, posing epistemic limitations for AI validation and deployment.

Contribution

It provides formal impossibility results showing the invalidity of the train-test paradigm in complex social systems, highlighting a critical epistemic challenge for AI validation.

Findings

Train-test paradigm is invalid for risk estimation in social systems.

Naive scaling and benchmarks do not address validation issues.

Formal results apply to recommender systems and large language models.

Abstract

Rapid model validation via the train-test paradigm has been a key driver for the breathtaking progress in machine learning and AI. However, modern AI systems often depend on a combination of tasks and data collection practices that violate all assumptions ensuring test validity. Yet, without rigorous model validation we cannot ensure the intended outcomes of deployed AI systems, including positive social impact, nor continue to advance AI research in a scientifically sound way. In this paper, I will show that for widely considered inference settings in complex social systems the train-test paradigm does not only lack a justification but is indeed invalid for any risk estimator, including counterfactual and causal estimators, with high probability. These formal impossibility results highlight a fundamental epistemic issue, i.e., that for key tasks in modern AI we cannot know whether models are valid under current data collection practices. Importantly, this includes variants of both recommender systems and reasoning via large language models, and neither na\"ive scaling nor limited benchmarks are suited to address this issue. I am illustrating these results via the widely used MovieLens benchmark and conclude by discussing the implications of these results for AI in social systems, including possible remedies such as participatory data curation and open science.