Gaze-based Object Detection in the Wild

TL;DR

This paper explores gaze-based object detection in realistic human-robot interaction scenarios, using heatmaps derived from gaze data and machine learning to identify objects and their bounding boxes efficiently.

Contribution

It introduces a novel gaze-based detection method utilizing heatmaps with variable temporal windows and grid sizes, demonstrating speed and resource efficiency over traditional detectors.

Findings

Effective object detection from gaze heatmaps in real-world scenarios

Method achieves high speed and low resource usage

Public dataset available for further research

Abstract

In human-robot collaboration, one challenging task is to teach a robot new yet unknown objects enabling it to interact with them. Thereby, gaze can contain valuable information. We investigate if it is possible to detect objects (object or no object) merely from gaze data and determine their bounding box parameters. For this purpose, we explore different sizes of temporal windows, which serve as a basis for the computation of heatmaps, i.e., the spatial distribution of the gaze data. Additionally, we analyze different grid sizes of these heatmaps, and demonstrate the functionality in a proof of concept using different machine learning techniques. Our method is characterized by its speed and resource efficiency compared to conventional object detectors. In order to generate the required data, we conducted a study with five subjects who could move freely and thus, turn towards arbitrary objects. This way, we chose a scenario for our data collection that is as realistic as possible. Since the subjects move while facing objects, the heatmaps also contain gaze data trajectories, complicating the detection and parameter regression. We make our data set publicly available to the research community for download.