How Far Can LLMs Improve from Experience? Measuring Test-Time Learning Ability in LLMs with Human Comparison

TL;DR

This paper evaluates the ability of large language models to learn during test time through strategic semantic games, revealing their potential and current limitations compared to humans in experience-based reasoning tasks.

Contribution

It introduces a novel framework and semantic game benchmarks for assessing test-time learning in LLMs, highlighting their capabilities and gaps relative to humans.

Findings

LLMs show measurable test-time learning ability

Improvements in LLMs are less stable and slower than humans

Significant gap remains between LLMs and humans in experience-based reasoning

Abstract

As evaluation designs of large language models may shape our trajectory toward artificial general intelligence, comprehensive and forward-looking assessment is essential. Existing benchmarks primarily assess static knowledge, while intelligence also entails the ability to rapidly learn from experience. To this end, we advocate for the evaluation of Test-time Learning, the capacity to improve performance in experience-based, reasoning-intensive tasks during test time. In this work, we propose semantic games as effective testbeds for evaluating test-time learning, due to their resistance to saturation and inherent demand for strategic reasoning. We introduce an objective evaluation framework that compares model performance under both limited and cumulative experience settings, and contains four forms of experience representation. To provide a comparative baseline, we recruit eight human participants to complete the same task. Results show that LLMs exhibit measurable test-time learning capabilities; however, their improvements are less stable under cumulative experience and progress more slowly than those observed in humans. These findings underscore the potential of LLMs as general-purpose learning machines, while also revealing a substantial intellectual gap between models and humans, irrespective of how well LLMs perform on static benchmarks.