Graph Representations for Reading Comprehension Analysis using Large Language Model and Eye-Tracking Biomarker

TL;DR

This paper introduces a graph-based text representation method using LLMs and eye-tracking data to analyze reading comprehension, revealing high consistency in language understanding at the graph structure level.

Contribution

It proposes a novel graph representation of text based on semantic meaning and prompts, integrating eye-tracking biomarkers to compare human and AI understanding.

Findings

LLMs show high consistency in graph topological understanding.

Eye fixation patterns align with important graph nodes and edges.

Graph-based analysis enhances insights into human-AI reading comprehension.

Abstract

Reading comprehension is a fundamental skill in human cognitive development. With the advancement of Large Language Models (LLMs), there is a growing need to compare how humans and LLMs understand language across different contexts and apply this understanding to functional tasks such as inference, emotion interpretation, and information retrieval. Our previous work used LLMs and human biomarkers to study the reading comprehension process. The results showed that the biomarkers corresponding to words with high and low relevance to the inference target, as labeled by the LLMs, exhibited distinct patterns, particularly when validated using eye-tracking data. However, focusing solely on individual words limited the depth of understanding, which made the conclusions somewhat simplistic despite their potential significance. This study used an LLM-based AI agent to group words from a reading passage into nodes and edges, forming a graph-based text representation based on semantic meaning and question-oriented prompts. We then compare the distribution of eye fixations on important nodes and edges. Our findings indicate that LLMs exhibit high consistency in language understanding at the level of graph topological structure. These results build on our previous findings and offer insights into effective human-AI co-learning strategies.