GazeInterpreter: Parsing Eye Gaze to Generate Eye-Body-Coordinated Narrations

TL;DR

GazeInterpreter is a novel LLM-based system that interprets eye gaze data and integrates it with body motion to generate coherent narrations, advancing human behavior understanding.

Contribution

It introduces a hierarchical, iterative approach combining symbolic gaze parsing and LLMs to produce eye-body-coordinated narrations, a novel method in behavior interpretation.

Findings

Effective in generating integrated eye-body narrations

Improves performance on action anticipation tasks

Enhances behavior summarization accuracy

Abstract

Comprehensively interpreting human behavior is a core challenge in human-aware artificial intelligence. However, prior works typically focused on body behavior, neglecting the crucial role of eye gaze and its synergy with body motion. We present GazeInterpreter - a novel large language model-based (LLM-based) approach that parses eye gaze data to generate eye-body-coordinated narrations. Specifically, our method features 1) a symbolic gaze parser that translates raw gaze signals into symbolic gaze events; 2) a hierarchical structure that first uses an LLM to generate eye gaze narration at semantic level and then integrates gaze with body motion within the same observation window to produce integrated narration; and 3) a self-correcting loop that iteratively refines the modality match, temporal coherence, and completeness of the integrated narration. This hierarchical and iterative processing can effectively align physical values and semantic text in the temporal and spatial domains. We validated the effectiveness of our eye-body-coordinated narrations on the text-driven motion generation task in the large-scale Nymeria benchmark. Moreover, we report significant performance improvements for the sample downstream tasks of action anticipation and behavior summarization. Taken together, these results reveal the significant potential of parsing eye gaze to interpret human behavior and open up a new direction for human behavior understanding.