Gaze-Aware Task Progression Detection Framework for Human-Robot Interaction Using RGB Cameras

TL;DR

This paper introduces a calibration-free, RGB camera-based framework for detecting human task progression in human-robot interaction by analyzing gaze patterns, enabling natural and intuitive communication without specialized hardware.

Contribution

It presents a novel, calibration-free gaze detection framework using standard RGB cameras to monitor attention shifts, enhancing naturalness and accessibility in human-robot interactions.

Findings

Achieved 77.6% task completion detection accuracy.

System improves user comfort and perceived naturalness.

Gaze cues are used intuitively without conscious effort.

Abstract

In human-robot interaction (HRI), detecting a human's gaze helps robots interpret user attention and intent. However, most gaze detection approaches rely on specialized eye-tracking hardware, limiting deployment in everyday settings. Appearance-based gaze estimation methods remove this dependency by using standard RGB cameras, but their practicality in HRI remains underexplored. We present a calibration-free framework for detecting task progression when information is conveyed via integrated display interfaces. The framework uses only the robot's built-in monocular RGB camera (640x480 resolution) and state-of-the-art gaze estimation to monitor attention patterns. It leverages natural behavior, where users shift focus from task interfaces to the robot's face to signal task completion, formalized through three Areas of Interest (AOI): tablet, robot face, and elsewhere. Systematic parameter optimization identifies configurations that balance detection accuracy and interaction latency. We validate our framework in a "First Day at Work" scenario, comparing it to button-based interaction. Results show a task completion detection accuracy of 77.6%. Compared to button-based interaction, the proposed system exhibits slightly higher response latency but preserves information retention and significantly improves comfort, social presence, and perceived naturalness. Notably, most participants reported that they did not consciously use eye movements to guide the interaction, underscoring the intuitive role of gaze as a communicative cue. This work demonstrates the feasibility of intuitive, low-cost, RGB-only gaze-based HRI for natural and engaging interactions.