Lower Perplexity is Not Always Human-Like

TL;DR

This study challenges the assumption that lower perplexity in language models correlates with human-like behavior, showing this is not universally true across languages like English and Japanese, and highlights the need for cross-lingual evaluation.

Contribution

The paper demonstrates that the correlation between perplexity and human-likeness does not hold across languages, emphasizing the importance of cross-lingual assessments in computational psycholinguistics.

Findings

Lower perplexity does not always indicate human-like language modeling.

The universality of perplexity as a human-likeness measure is questioned.

Differences in language structure affect the perplexity-human-likeness relationship.

Abstract

In computational psycholinguistics, various language models have been evaluated against human reading behavior (e.g., eye movement) to build human-like computational models. However, most previous efforts have focused almost exclusively on English, despite the recent trend towards linguistic universal within the general community. In order to fill the gap, this paper investigates whether the established results in computational psycholinguistics can be generalized across languages. Specifically, we re-examine an established generalization -- the lower perplexity a language model has, the more human-like the language model is -- in Japanese with typologically different structures from English. Our experiments demonstrate that this established generalization exhibits a surprising lack of universality; namely, lower perplexity is not always human-like. Moreover, this discrepancy between English and Japanese is further explored from the perspective of (non-)uniform information density. Overall, our results suggest that a cross-lingual evaluation will be necessary to construct human-like computational models.