How Large Language Models Get Stuck: Early structure with persistent errors

TL;DR

This paper investigates why large language models like OPT struggle with certain linguistic tasks, revealing early entrenched errors that persist and proposing the Bigram Hypothesis to explain this phenomenon.

Contribution

It introduces the Bigram Hypothesis, linking early statistical biases to persistent errors in LLMs, supported by qualitative and quantitative analyses.

Findings

OPT fails to consistently prefer grammatical sentences in about one-third of BLiMP classes.

Early biases in likelihood separation tend to persist throughout training.

The Bigram Hypothesis offers a new perspective on entrenched errors in LLMs.

Abstract

Linguistic insights may help make Large Language Model (LLM) training more efficient. We trained Meta's OPT model on the 100M word BabyLM dataset, and evaluated it on the BLiMP benchmark, which consists of 67 classes, each defined by sentence pairs that differ in a targeted syntactic or semantic rule violation. We tested the model's preference for grammatical over ungrammatical sentences across training iterations and grammatical types. In nearly one-third of the BLiMP classes, OPT fails to consistently assign a higher likelihood to grammatical sentences, even after extensive training. When it fails, it often establishes a clear (erroneous) separation of the likelihoods at an early stage of processing and sustains this to the end of our training phase. We hypothesize that this mis-categorization is costly because it creates entrenched biases that must, eventually, be reversed in order for the model to perform well. We probe this phenomenon using a mixture of qualitative (based on linguistic theory and the theory of Deep Learning) and quantitative (based on numerical testing) assessments. Our qualitative assessments indicate that only some BLiMP tests are meaningful guides. We conclude by articulating a hypothesis, the Bigram Hypothesis, which claims that the learning process will exhibit erroneous entrenchment if bigram statistics bias the model toward wrong distinctions early in training, and we describe a method of testing the hypothesis on appropriately selected BLiMP classes.