Cognitive Dark Matter: Measuring What AI Misses

TL;DR

This paper introduces the concept of 'cognitive dark matter' (CDM), highlighting unmeasured brain functions in AI that influence behavior but are not captured by current benchmarks, and proposes a new research approach to surface and train these functions.

Contribution

It defines key CDM domains, critiques existing benchmarks, and proposes a multi-faceted data collection strategy to incorporate cognitive processes into AI training.

Findings

Current benchmarks are biased toward mastered capabilities.

Identified key CDM domains like metacognition and reasoning.

Proposed data types can surface unmeasured cognitive functions.

Abstract

We propose that the jagged intelligence landscape of modern AI systems arises from a missing training signal that we call "cognitive dark matter" (CDM): brain functions that meaningfully shape behavior yet are hard to infer from behavior alone. We identify key CDM domains-metacognition, cognitive flexibility, episodic memory, lifelong learning, abductive reasoning, social and common-sense reasoning, and emotional intelligence-and present evidence that current AI benchmarks and large-scale neuroscience datasets are both heavily skewed toward already-mastered capabilities, with CDM-loaded functions largely unmeasured. We then outline a research program centered on three complementary data types designed to surface CDM for model training: (i) latent variables from large-scale cognitive models, (ii) process-tracing data such as eye-tracking and think-aloud protocols, and (iii) paired neural-behavioral data. These data will enable AI training on cognitive process rather than behavioral outcome alone, producing models with more general, less jagged intelligence. As a dual benefit, the same data will advance our understanding of human intelligence itself.