Attention-aware semantic relevance predicting Chinese sentence reading

TL;DR

This paper introduces an attention-aware semantic relevance metric inspired by Transformer models and human memory, which improves prediction of eye-movement durations in Chinese reading and supports semantic preview benefits.

Contribution

It proposes a novel attention-aware approach for computing semantic relevance that incorporates contextual contributions and expectation effects, enhancing modeling of Chinese reading.

Findings

Attention-aware metrics outperform existing approaches in predicting fixation durations.

Results support the existence of semantic preview benefits in Chinese reading.

Metrics are highly interpretable from linguistic and cognitive perspectives.

Abstract

In recent years, several influential computational models and metrics have been proposed to predict how humans comprehend and process sentence. One particularly promising approach is contextual semantic similarity. Inspired by the attention algorithm in Transformer and human memory mechanisms, this study proposes an ``attention-aware'' approach for computing contextual semantic relevance. This new approach takes into account the different contributions of contextual parts and the expectation effect, allowing it to incorporate contextual information fully. The attention-aware approach also facilitates the simulation of existing reading models and evaluate them. The resulting ``attention-aware'' metrics of semantic relevance can more accurately predict fixation durations in Chinese reading tasks recorded in an eye-tracking corpus than those calculated by existing approaches. The study's findings further provide strong support for the presence of semantic preview benefits in Chinese naturalistic reading. Furthermore, the attention-aware metrics of semantic relevance, being memory-based, possess high interpretability from both linguistic and cognitive standpoints, making them a valuable computational tool for modeling eye-movements in reading and further gaining insight into the process of language comprehension. Our approach underscores the potential of these metrics to advance our comprehension of how humans understand and process language, ultimately leading to a better understanding of language comprehension and processing.