Towards Attention-Aware Large Language Models: Integrating Real-Time Eye-Tracking and EEG for Adaptive AI Responses

TL;DR

This paper presents an innovative attention-aware large language model that dynamically monitors user attention through real-time EEG and eye-tracking data, enabling adaptive responses to enhance engagement and reduce cognitive overload.

Contribution

It introduces a novel system integrating EEG and eye-tracking into LLMs for real-time attention state classification and adaptive interaction.

Findings

Successfully classifies five attention states in real-time

Adapts responses based on user attention levels

Improves user engagement and reduces cognitive load

Abstract

This project proposes an attention-aware LLM that integrates EEG and eye tracking to monitor and measure user attention dynamically. To realize this, the project will integrate real-time EEG and eye-tracking data into an LLM-based interactive system and classify the user's attention state on the fly. The system can identify five attention states: High Attention, Stable Attention, Dropping Attention, Cognitive Overload, and Distraction. It responds accordingly to each state, with a particular focus on adapting to decreased attention, distraction, and cognitive overload to improve user engagement and reduce cognitive load.