Gaze Estimation using Transformer

TL;DR

This paper explores the use of transformer architectures for gaze estimation, comparing pure and hybrid models, and demonstrates that hybrid transformers achieve state-of-the-art results with fewer parameters.

Contribution

It introduces a hybrid transformer model combining CNNs and transformers for gaze estimation, showing superior performance over pure transformers.

Findings

Hybrid transformer outperforms pure transformer in all datasets.

Hybrid transformer achieves state-of-the-art performance with fewer parameters.

Self-attention mechanism provides significant advantages in gaze estimation.

Abstract

Recent work has proven the effectiveness of transformers in many computer vision tasks. However, the performance of transformers in gaze estimation is still unexplored. In this paper, we employ transformers and assess their effectiveness for gaze estimation. We consider two forms of vision transformer which are pure transformers and hybrid transformers. We first follow the popular ViT and employ a pure transformer to estimate gaze from images. On the other hand, we preserve the convolutional layers and integrate CNNs as well as transformers. The transformer serves as a component to complement CNNs. We compare the performance of the two transformers in gaze estimation. The Hybrid transformer significantly outperforms the pure transformer in all evaluation datasets with less parameters. We further conduct experiments to assess the effectiveness of the hybrid transformer and explore the advantage of self-attention mechanism. Experiments show the hybrid transformer can achieve state-of-the-art performance in all benchmarks with pre-training.To facilitate further research, we release codes and models in https://github.com/yihuacheng/GazeTR.