The Cognitive Capabilities of Generative AI: A Comparative Analysis with Human Benchmarks

TL;DR

This study benchmarks large language and vision models against human cognitive performance, revealing strengths in verbal and working memory tasks but significant weaknesses in perceptual reasoning, highlighting progress and limitations of current AI models.

Contribution

It provides a comprehensive comparison of AI models' cognitive abilities with human benchmarks across multiple domains using the WAIS-IV assessment.

Findings

Models outperform humans in working memory and verbal comprehension.

Models perform poorly on perceptual reasoning tasks.

Advances in training data and parameters improve cognitive capabilities.

Abstract

There is increasing interest in tracking the capabilities of general intelligence foundation models. This study benchmarks leading large language models and vision language models against human performance on the Wechsler Adult Intelligence Scale (WAIS-IV), a comprehensive, population-normed assessment of underlying human cognition and intellectual abilities, with a focus on the domains of VerbalComprehension (VCI), Working Memory (WMI), and Perceptual Reasoning (PRI). Most models demonstrated exceptional capabilities in the storage, retrieval, and manipulation of tokens such as arbitrary sequences of letters and numbers, with performance on the Working Memory Index (WMI) greater or equal to the 99.5th percentile when compared to human population normative ability. Performance on the Verbal Comprehension Index (VCI) which measures retrieval of acquired information, and linguistic understanding about the meaning of words and their relationships to each other, also demonstrated consistent performance at or above the 98th percentile. Despite these broad strengths, we observed consistently poor performance on the Perceptual Reasoning Index (PRI; range 0.1-10th percentile) from multimodal models indicating profound inability to interpret and reason on visual information. Smaller and older model versions consistently performed worse, indicating that training data, parameter count and advances in tuning are resulting in significant advances in cognitive ability.