Does Attention Mechanism Possess the Feature of Human Reading? A Perspective of Sentiment Classification Task

TL;DR

This study investigates whether the attention mechanism in deep learning models for sentiment classification mimics human reading by focusing on important words, using eye-tracking data for analysis and improvement.

Contribution

It demonstrates that attention mechanisms can reflect human reading patterns and shows how eye-tracking data can enhance model interpretability and performance.

Findings

Attention focuses on sentiment-rich words like adjectives and adverbs

Eye-tracking data can correct attention errors and improve accuracy

Attention mechanism exhibits features of human reading behavior

Abstract

[Purpose] To understand the meaning of a sentence, humans can focus on important words in the sentence, which reflects our eyes staying on each word in different gaze time or times. Thus, some studies utilize eye-tracking values to optimize the attention mechanism in deep learning models. But these studies lack to explain the rationality of this approach. Whether the attention mechanism possesses this feature of human reading needs to be explored. [Design/methodology/approach] We conducted experiments on a sentiment classification task. Firstly, we obtained eye-tracking values from two open-source eye-tracking corpora to describe the feature of human reading. Then, the machine attention values of each sentence were learned from a sentiment classification model. Finally, a comparison was conducted to analyze machine attention values and eye-tracking values. [Findings] Through experiments, we found the attention mechanism can focus on important words, such as adjectives, adverbs, and sentiment words, which are valuable for judging the sentiment of sentences on the sentiment classification task. It possesses the feature of human reading, focusing on important words in sentences when reading. Due to the insufficient learning of the attention mechanism, some words are wrongly focused. The eye-tracking values can help the attention mechanism correct this error and improve the model performance. [Originality/value] Our research not only provides a reasonable explanation for the study of using eye-tracking values to optimize the attention mechanism, but also provides new inspiration for the interpretability of attention mechanism.