Multimodal Across Domains Gaze Target Detection

TL;DR

This paper introduces a novel multimodal deep learning approach for gaze target detection in images that does not require gaze angle supervision or eye location, and effectively handles domain adaptation across datasets.

Contribution

The paper proposes a new multimodal deep architecture for gaze detection that operates without gaze angle supervision and addresses domain adaptation, outperforming prior methods.

Findings

Outperforms previous methods on multiple benchmarks.

Effective domain adaptation across different datasets.

Various model configurations show improved results.

Abstract

This paper addresses the gaze target detection problem in single images captured from the third-person perspective. We present a multimodal deep architecture to infer where a person in a scene is looking. This spatial model is trained on the head images of the person-of- interest, scene and depth maps representing rich context information. Our model, unlike several prior art, do not require supervision of the gaze angles, do not rely on head orientation information and/or location of the eyes of person-of-interest. Extensive experiments demonstrate the stronger performance of our method on multiple benchmark datasets. We also investigated several variations of our method by altering joint-learning of multimodal data. Some variations outperform a few prior art as well. First time in this paper, we inspect domain adaption for gaze target detection, and we empower our multimodal network to effectively handle the domain gap across datasets. The code of the proposed method is available at https://github.com/francescotonini/multimodal-across-domains-gaze-target-detection.