Visual Large Language Models Exhibit Human-Level Cognitive Flexibility in the Wisconsin Card Sorting Test

TL;DR

This paper demonstrates that state-of-the-art Visual Large Language Models can achieve or surpass human-level cognitive flexibility in the Wisconsin Card Sorting Test, highlighting their potential to emulate complex brain functions.

Contribution

It is the first to evaluate VLLMs' cognitive flexibility using WCST and shows they can simulate deficits, revealing parallels to human cognitive architecture.

Findings

VLLMs achieve or surpass human-level set-shifting with chain-of-thought prompting.

Input modality and prompting strategy significantly influence VLLMs' performance.

VLLMs can simulate cognitive deficits through role-playing, indicating a form of cognitive architecture.

Abstract

Cognitive flexibility has been extensively studied in human cognition but remains relatively unexplored in the context of Visual Large Language Models (VLLMs). This study assesses the cognitive flexibility of state-of-the-art VLLMs (GPT-4o, Gemini-1.5 Pro, and Claude-3.5 Sonnet) using the Wisconsin Card Sorting Test (WCST), a classic measure of set-shifting ability. Our results reveal that VLLMs achieve or surpass human-level set-shifting capabilities under chain-of-thought prompting with text-based inputs. However, their abilities are highly influenced by both input modality and prompting strategy. In addition, we find that through role-playing, VLLMs can simulate various functional deficits aligned with patients having impairments in cognitive flexibility, suggesting that VLLMs may possess a cognitive architecture, at least regarding the ability of set-shifting, similar to the brain. This study reveals the fact that VLLMs have already approached the human level on a key component underlying our higher cognition, and highlights the potential to use them to emulate complex brain processes.