Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Study

TL;DR

This paper introduces a cognitive-inspired framework to analyze and evaluate the learning abilities of large language models across three key dimensions, revealing insights into their strengths and limitations.

Contribution

It proposes a novel three-dimensional framework for understanding LLM learning, and develops a benchmark for comprehensive evaluation of their general learning abilities.

Findings

Interaction enhances learning effectiveness

Larger models show emergent conceptual understanding

LLMs excel in few-shot but not many-shot learning

Abstract

Large language models (LLMs) have shown impressive capabilities across tasks such as mathematics, coding, and reasoning, yet their learning ability, which is crucial for adapting to dynamic environments and acquiring new knowledge, remains underexplored. In this work, we address this gap by introducing a framework inspired by cognitive psychology and education. Specifically, we decompose general learning ability into three distinct, complementary dimensions: Learning from Instructor (acquiring knowledge via explicit guidance), Learning from Concept (internalizing abstract structures and generalizing to new contexts), and Learning from Experience (adapting through accumulated exploration and feedback). We conduct a comprehensive empirical study across the three learning dimensions and identify several insightful findings, such as (i) interaction improves learning; (ii) conceptual understanding is scale-emergent and benefits larger models; and (iii) LLMs are effective few-shot learners but not many-shot learners. Based on our framework and empirical findings, we introduce a benchmark that provides a unified and realistic evaluation of LLMs' general learning abilities across three learning cognition dimensions. It enables diagnostic insights and supports evaluation and development of more adaptive and human-like models.