Development of Cognitive Intelligence in Pre-trained Language Models

TL;DR

This paper investigates how pre-trained language models develop cognitive abilities during training, revealing a consistent window where their performance aligns most closely with human cognitive development, which is crucial for modeling human cognition.

Contribution

It introduces a developmental perspective to analyze PLMs' cognitive abilities during training, highlighting a specific window of maximal cognitive alignment across models.

Findings

PLMs show a consistent developmental window of maximal cognitive alignment.

Training before this window enhances structure for learning, after reduces alignment.

Alignment with human cognition peaks during a specific training phase.

Abstract

Recent studies show evidence for emergent cognitive abilities in Large Pre-trained Language Models (PLMs). The increasing cognitive alignment of these models has made them candidates for cognitive science theories. Prior research into the emergent cognitive abilities of PLMs has largely been path-independent to model training, i.e., has focused on the final model weights and not the intermediate steps. However, building plausible models of human cognition using PLMs would benefit from considering the developmental alignment of their performance during training to the trajectories of children's thinking. Guided by psychometric tests of human intelligence, we choose four sets of tasks to investigate the alignment of ten popular families of PLMs and evaluate their available intermediate and final training steps. These tasks are Numerical ability, Linguistic abilities, Conceptual understanding, and Fluid reasoning. We find a striking regularity: regardless of model size, the developmental trajectories of PLMs consistently exhibit a window of maximal alignment to human cognitive development. Before that window, training appears to endow "blank slate" models with the requisite structure to be poised to rapidly learn from experience. After that window, training appears to serve the engineering goal of reducing loss but not the scientific goal of increasing alignment with human cognition.