Epistemic Compression: The Case for Deliberate Ignorance in High-Stakes AI

TL;DR

This paper proposes Epistemic Compression, a principle that aligns model complexity with data stability to improve robustness in high-stakes AI applications, challenging the notion that bigger models are always better.

Contribution

It introduces a novel architectural regularization approach called Epistemic Compression and a Regime Index to adapt model complexity based on data stability, improving performance in critical domains.

Findings

The Regime Index aligned with the best modeling strategy in 86.7% of cases.

Architectural parsimony reduces overfitting in unstable, data-scarce environments.

Scaling models alone is insufficient for robustness in high-stakes AI.

Abstract

Foundation models excel in stable environments, yet often fail where reliability matters most: medicine, finance, and policy. This Fidelity Paradox is not just a data problem; it is structural. In domains where rules change over time, extra model capacity amplifies noise rather than capturing signal. We introduce Epistemic Compression: the principle that robustness emerges from matching model complexity to the shelf life of the data, not from scaling parameters. Unlike classical regularization, which penalizes weights post hoc, Epistemic Compression enforces parsimony through architecture: the model structure itself is designed to reduce overfitting by making it architecturally costly to represent variance that exceeds the evidence in the data. We operationalize this with a Regime Index that separates Shifting Regime (unstable, data-poor; simplicity wins) from Stable Regime (invariant, data-rich; complexity viable). In an exploratory synthesis of 15 high-stakes domains, this index was concordant with the empirically superior modeling strategy in 86.7% of cases (13/15). High-stakes AI demands a shift from scaling for its own sake to principled parsimony.