The Astonishing Ability of Large Language Models to Parse Jabberwockified Language

TL;DR

Large language models can remarkably interpret heavily degraded English texts by leveraging structural cues, revealing insights into linguistic structure and language processing.

Contribution

Demonstrates that LLMs can recover meaning from severely distorted texts, highlighting the importance of structural cues in language understanding.

Findings

LLMs translate nonsense-laden texts to close-to-original English.

Structural cues constrain lexical meaning more than previously thought.

Results suggest tight integration of syntax, semantics, and world knowledge in language processing.

Abstract

We show that large language models (LLMs) have an astonishing ability to recover meaning from severely degraded English texts. Texts in which content words have been randomly substituted by nonsense strings, e.g., "At the ghybe of the swuint, we are haiveed to Wourge Phrear-gwurr, who sproles into an ghitch flount with his crurp", can be translated to conventional English that is, in many cases, close to the original text, e.g., "At the start of the story, we meet a man, Chow, who moves into an apartment building with his wife." These results show that structural cues (e.g., morphosyntax, closed-class words) constrain lexical meaning to a much larger degree than imagined. Although the abilities of LLMs to make sense of "Jabberwockified" English are clearly superhuman, they are highly relevant to understanding linguistic structure and suggest that efficient language processing either in biological or artificial systems likely benefits from very tight integration between syntax, lexical semantics, and general world knowledge.