Looking at the right stuff: Guided semantic-gaze for autonomous driving

TL;DR

This paper introduces SAGE-Net, a novel approach combining scene semantics and gaze data to improve driver attention prediction, enhancing safety and efficiency in autonomous driving without extra computational costs.

Contribution

The paper presents SAGE-Net, integrating scene semantics with gaze data for more accurate driver attention prediction, a novel approach in autonomous driving research.

Findings

SAGE outperforms existing methods in 87.5% of cases.

The approach improves saliency prediction accuracy.

No additional computational overhead during training.

Abstract

In recent years, predicting driver's focus of attention has been a very active area of research in the autonomous driving community. Unfortunately, existing state-of-the-art techniques achieve this by relying only on human gaze information, thereby ignoring scene semantics. We propose a novel Semantics Augmented GazE (SAGE) detection approach that captures driving specific contextual information, in addition to the raw gaze. Such a combined attention mechanism serves as a powerful tool to focus on the relevant regions in an image frame in order to make driving both safe and efficient. Using this, we design a complete saliency prediction framework - SAGE-Net, which modifies the initial prediction from SAGE by taking into account vital aspects such as distance to objects (depth), ego vehicle speed, and pedestrian crossing intent. Exhaustive experiments conducted through four popular saliency algorithms show that on $\mathbf{49/56\text{ }(87.5\%)}$ cases - considering both the overall dataset and crucial driving scenarios, SAGE outperforms existing techniques without any additional computational overhead during the training process. The augmented dataset along with the relevant code are available as part of the supplementary material.