Uneven Evolution of Cognition Across Generations of Generative AI Models

TL;DR

This paper introduces a psychometric framework and the AIQ Benchmark to evaluate and track the uneven cognitive development of generative AI models across different modalities and generations.

Contribution

It presents a novel assessment framework revealing asymmetric cognitive growth and architectural biases in generative AI, highlighting limitations in current scaling approaches.

Findings

Models perform near-ceiling in verbal tasks but near-floor in perceptual reasoning.

Abstract reasoning improves faster in language than in visual formats.

Visual-perceptual organization remains largely stagnant across generations.

Abstract

The pursuit of artificial general intelligence necessitates robust methods for evaluating the cognitive capabilities of models beyond narrow task performance. Here, we introduce a psychometric framework to assess the cognitive profiles of generative AI, comparing them to human norms and tracking their evolution across generations. Initial evaluation of leading multimodal models using tasks adapted from the Wechsler Adult Intelligence Scale revealed a profoundly uneven cognitive architecture: near-ceiling performance in verbal comprehension and working memory (>$98^{\text{th}}$ percentile) contrasted with near-floor performance in perceptual reasoning (<$1^{\text{st}}$ percentile). To track developmental trajectories beyond human-normed limits, we developed the Artificial Intelligence Quotient (AIQ) Benchmark and applied it to six generations and two model families, revealing significant but asymmetric performance gains. Notably, we uncovered a sharp dissociation between modalities; abstract quantitative reasoning matured far more rapidly when presented linguistically compared to a visually analogous format, indicating an architectural bias towards language-based symbolic manipulation. While abstract visual reasoning improved, visual-perceptual organization remained largely stagnant. Collectively, these findings demonstrate that the cognitive abilities of generative models are evolving unevenly, suggesting that scaling and optimization approaches to AGI development alone may be insufficient to overcome fundamental architectural limitations in achieving balanced, human-like general intelligence.