Computational Sentence-level Metrics Predicting Human Sentence Comprehension

TL;DR

This paper develops multilingual sentence-level metrics using large language models to predict human sentence comprehension, demonstrating high accuracy and interpretability across languages.

Contribution

Introduces novel sentence surprisal and relevance metrics derived from multilingual LLMs for predicting human sentence comprehension.

Findings

Metrics accurately predict human reading speeds

Effective across multiple languages

Enhance understanding of sentence processing difficulties

Abstract

The majority of research in computational psycholinguistics has concentrated on the processing of words. This study introduces innovative methods for computing sentence-level metrics using multilingual large language models. The metrics developed sentence surprisal and sentence relevance and then are tested and compared to validate whether they can predict how humans comprehend sentences as a whole across languages. These metrics offer significant interpretability and achieve high accuracy in predicting human sentence reading speeds. Our results indicate that these computational sentence-level metrics are exceptionally effective at predicting and elucidating the processing difficulties encountered by readers in comprehending sentences as a whole across a variety of languages. Their impressive performance and generalization capabilities provide a promising avenue for future research in integrating LLMs and cognitive science.