CogLM: Tracking Cognitive Development of Large Language Models

TL;DR

This paper introduces CogLM, a comprehensive benchmark based on Piaget's theory to evaluate the cognitive development of large language models, revealing their abilities and influencing factors.

Contribution

It presents a novel cognitive ability benchmark for LLMs grounded in Piaget's theory, enabling systematic assessment of their cognitive development.

Findings

Advanced LLMs like GPT-4 exhibit human-like cognitive abilities.

Parameter size and optimization objectives significantly impact cognitive levels.

Better cognitive abilities correlate with improved downstream task performance.

Abstract

Piaget's Theory of Cognitive Development (PTC) posits that the development of cognitive levels forms the foundation for human learning across various abilities. As Large Language Models (LLMs) have recently shown remarkable abilities across a wide variety of tasks, we are curious about the cognitive levels of current LLMs: to what extent they have developed and how this development has been achieved. To this end, we construct a benchmark CogLM (Cognitive Ability Evaluation for Language Model) based on PTC to assess the cognitive levels of LLMs. CogLM comprises 1,220 questions spanning 10 cognitive abilities crafted by more than 20 human experts, providing a comprehensive testbed for the cognitive levels of LLMs. Through extensive experiments across multiple mainstream LLMs with CogLM, we find that: (1) In our testing framework, advanced LLMs (such as GPT-4) have demonstrated human-like cognitive abilities, comparable to those of a 20-year-old human. (2) The parameter size and optimization objective are two key factors affecting the cognitive levels of LLMs. (3) The performance on downstream tasks is positively correlated with the level of cognitive abilities. These findings fill the gap in research on the cognitive abilities of LLMs, tracing the development of LLMs from a cognitive perspective and guiding the future direction of their evolution.