End-to-End Human-Gaze-Target Detection with Transformers

TL;DR

This paper introduces HGTTR, an end-to-end transformer-based method for human-gaze-target detection that simultaneously detects human heads and gaze targets, outperforming existing methods on standard benchmarks.

Contribution

HGTTR is the first end-to-end transformer model for simultaneous human head and gaze target detection, simplifying the pipeline and improving accuracy over prior two-stage approaches.

Findings

HGTTR achieves 6.4 mAP improvement on GazeFollowing dataset.

HGTTR achieves 10.3 mAP improvement on VideoAttentionTarget dataset.

HGTTR outperforms state-of-the-art methods with a simpler architecture.

Abstract

In this paper, we propose an effective and efficient method for Human-Gaze-Target (HGT) detection, i.e., gaze following. Current approaches decouple the HGT detection task into separate branches of salient object detection and human gaze prediction, employing a two-stage framework where human head locations must first be detected and then be fed into the next gaze target prediction sub-network. In contrast, we redefine the HGT detection task as detecting human head locations and their gaze targets, simultaneously. By this way, our method, named Human-Gaze-Target detection TRansformer or HGTTR, streamlines the HGT detection pipeline by eliminating all other additional components. HGTTR reasons about the relations of salient objects and human gaze from the global image context. Moreover, unlike existing two-stage methods that require human head locations as input and can predict only one human's gaze target at a time, HGTTR can directly predict the locations of all people and their gaze targets at one time in an end-to-end manner. The effectiveness and robustness of our proposed method are verified with extensive experiments on the two standard benchmark datasets, GazeFollowing and VideoAttentionTarget. Without bells and whistles, HGTTR outperforms existing state-of-the-art methods by large margins (6.4 mAP gain on GazeFollowing and 10.3 mAP gain on VideoAttentionTarget) with a much simpler architecture.