Mind Scramble: Unveiling Large Language Model Psychology Via Typoglycemia

TL;DR

This paper introduces LLM Psychology and uses Typoglycemia experiments to analyze the cognitive behaviors and robustness of large language models, revealing human-like traits and unique model-specific patterns.

Contribution

It pioneers the use of psychology-inspired experiments, like Typoglycemia, to investigate LLM cognition and provides a new benchmark for evaluating model robustness without additional datasets.

Findings

LLMs show human-like behaviors in scrambled text tasks

Typoglycemia serves as a robustness benchmark for LLMs

Different tasks reveal varying impacts and unique model-specific cognitive patterns

Abstract

Research into the external behaviors and internal mechanisms of large language models (LLMs) has shown promise in addressing complex tasks in the physical world. Studies suggest that powerful LLMs, like GPT-4, are beginning to exhibit human-like cognitive abilities, including planning, reasoning, and reflection. In this paper, we introduce a research line and methodology called LLM Psychology, leveraging human psychology experiments to investigate the cognitive behaviors and mechanisms of LLMs. We migrate the Typoglycemia phenomenon from psychology to explore the "mind" of LLMs. Unlike human brains, which rely on context and word patterns to comprehend scrambled text, LLMs use distinct encoding and decoding processes. Through Typoglycemia experiments at the character, word, and sentence levels, we observe: (I) LLMs demonstrate human-like behaviors on a macro scale, such as lower task accuracy and higher token/time consumption; (II) LLMs exhibit varying robustness to scrambled input, making Typoglycemia a benchmark for model evaluation without new datasets; (III) Different task types have varying impacts, with complex logical tasks (e.g., math) being more challenging in scrambled form; (IV) Each LLM has a unique and consistent "cognitive pattern" across tasks, revealing general mechanisms in its psychology process. We provide an in-depth analysis of hidden layers to explain these phenomena, paving the way for future research in LLM Psychology and deeper interpretability.