Randomly Sampled Language Reasoning Problems Elucidate Limitations of In-Context Learning

TL;DR

This paper investigates the limitations of in-context learning in large language models by testing them on simple, unseen language tasks, revealing they underperform compared to n-gram models even in minimal complexity settings.

Contribution

It introduces a novel experimental setup with randomly sampled simple language tasks to isolate in-context learning performance from model knowledge, highlighting fundamental limitations.

Findings

LLMs underperform n-gram models on simple language tasks

In-context learning does not outperform basic statistical models in this setting

LLMs struggle with unseen language tasks regardless of task simplicity

Abstract

While LLMs have revolutionized the field of machine learning due to their high performance on a strikingly wide range of problems, they are also known to hallucinate false answers and underperform on less canonical versions of the same tasks. There are several emerging theories of LLM performance, among them that LLMs lack world modeling ability, that they have an undesirable bias towards an autoregressive prior, and that they struggle on more novel problems. The existing literature on LLM input novelty has focused on tasks of relatively high complexity, studying perturbations of canonical but complex problems. In this paper, we attempt to minimize complexity in order to isolate novelty as a factor in LLM underperformance and investigate the power of in-context-learning. To this end, we consider an extremely simple domain: next token prediction on simple language tasks. The twist is that these language tasks are wholly unseen, as they are randomly drawn from a large, parsimoniously defined set of languages arising from simple grammar rules. This experimental setup allows us to evaluate ICL independently of models' parametric knowledge. We find that LLMs uniformly underperform n-gram models on this task, both when used as next token predictors and in chain-of-thought.