Beyond Human-Like Processing: Large Language Models Perform Equivalently on Forward and Backward Scientific Text

TL;DR

Large language models perform equally well on forward and backward scientific texts, suggesting their success stems from flexible architectures rather than human-like language processing.

Contribution

The study demonstrates that LLMs' performance is driven by architecture flexibility, not human-like processing, by training on forward and backward scientific texts and comparing results.

Findings

LLMs perform equally well on forward and backward texts.

LLMs outperform human experts on a neuroscience benchmark.

Transformers' success is due to pattern extraction from structured inputs.

Abstract

The impressive performance of large language models (LLMs) has led to their consideration as models of human language processing. Instead, we suggest that the success of LLMs arises from the flexibility of the transformer learning architecture. To evaluate this conjecture, we trained LLMs on scientific texts that were either in a forward or backward format. Despite backward text being inconsistent with the structure of human languages, we found that LLMs performed equally well in either format on a neuroscience benchmark, eclipsing human expert performance for both forward and backward orders. Our results are consistent with the success of transformers across diverse domains, such as weather prediction and protein design. This widespread success is attributable to LLM's ability to extract predictive patterns from any sufficiently structured input. Given their generality, we suggest caution in interpreting LLM's success in linguistic tasks as evidence for human-like mechanisms.