G-VOILA: Gaze-Facilitated Information Querying in Daily Scenarios

TL;DR

G-VOILA introduces a gaze-voice integrated querying system that enhances natural interaction in daily scenarios, demonstrating improved effectiveness through user studies and a novel design framework.

Contribution

This paper presents the first gaze-facilitated querying paradigm combining gaze, visual context, and voice, with a deep learning implementation and validation in real-world scenarios.

Findings

Gaze-voice coordination patterns were identified in natural queries.

The G-VOILA system outperformed baselines without gaze data in user studies.

A comprehensive design framework for gaze-integrated querying was developed.

Abstract

Modern information querying systems are progressively incorporating multimodal inputs like vision and audio. However, the integration of gaze -- a modality deeply linked to user intent and increasingly accessible via gaze-tracking wearables -- remains underexplored. This paper introduces a novel gaze-facilitated information querying paradigm, named G-VOILA, which synergizes users' gaze, visual field, and voice-based natural language queries to facilitate a more intuitive querying process. In a user-enactment study involving 21 participants in 3 daily scenarios (p = 21, scene = 3), we revealed the ambiguity in users' query language and a gaze-voice coordination pattern in users' natural query behaviors with G-VOILA. Based on the quantitative and qualitative findings, we developed a design framework for the G-VOILA paradigm, which effectively integrates the gaze data with the in-situ querying context. Then we implemented a G-VOILA proof-of-concept using cutting-edge deep learning techniques. A follow-up user study (p = 16, scene = 2) demonstrates its effectiveness by achieving both higher objective score and subjective score, compared to a baseline without gaze data. We further conducted interviews and provided insights for future gaze-facilitated information querying systems.