From Sight to Insight: Unleashing Eye-Tracking in Weakly Supervised Video Salient Object Detection

TL;DR

This paper introduces a weakly supervised video salient object detection method leveraging eye-tracking fixation data, employing novel modules for feature guidance and contrastive learning, achieving superior results on benchmark datasets.

Contribution

It proposes a new weakly supervised framework that integrates fixation data with position and semantic embedding, and introduces contrastive learning modules for improved spatiotemporal modeling.

Findings

Outperforms existing methods on five benchmark datasets

Effective utilization of eye-tracking data enhances detection accuracy

Novel modules improve feature learning and contrastive modeling

Abstract

The eye-tracking video saliency prediction (VSP) task and video salient object detection (VSOD) task both focus on the most attractive objects in video and show the result in the form of predictive heatmaps and pixel-level saliency masks, respectively. In practical applications, eye tracker annotations are more readily obtainable and align closely with the authentic visual patterns of human eyes. Therefore, this paper aims to introduce fixation information to assist the detection of video salient objects under weak supervision. On the one hand, we ponder how to better explore and utilize the information provided by fixation, and then propose a Position and Semantic Embedding (PSE) module to provide location and semantic guidance during the feature learning process. On the other hand, we achieve spatiotemporal feature modeling under weak supervision from the aspects of feature selection and feature contrast. A Semantics and Locality Query (SLQ) Competitor with semantic and locality constraints is designed to effectively select the most matching and accurate object query for spatiotemporal modeling. In addition, an Intra-Inter Mixed Contrastive (IIMC) model improves the spatiotemporal modeling capabilities under weak supervision by forming an intra-video and inter-video contrastive learning paradigm. Experimental results on five popular VSOD benchmarks indicate that our model outperforms other competitors on various evaluation metrics.