MDS-ViTNet: Improving saliency prediction for Eye-Tracking with Vision Transformer

TL;DR

This paper introduces MDS-ViTNet, a novel vision transformer-based network for improved saliency prediction in eye-tracking, achieving state-of-the-art results and applicable across various fields.

Contribution

The paper presents a new encoder-decoder architecture using Vision Transformer and dual decoders for enhanced saliency prediction, surpassing previous methods.

Findings

Achieves state-of-the-art performance on multiple benchmarks.

Utilizes a novel multi-decoder approach for saliency map generation.

Demonstrates effectiveness of Vision Transformer in eye-tracking applications.

Abstract

In this paper, we present a novel methodology we call MDS-ViTNet (Multi Decoder Saliency by Vision Transformer Network) for enhancing visual saliency prediction or eye-tracking. This approach holds significant potential for diverse fields, including marketing, medicine, robotics, and retail. We propose a network architecture that leverages the Vision Transformer, moving beyond the conventional ImageNet backbone. The framework adopts an encoder-decoder structure, with the encoder utilizing a Swin transformer to efficiently embed most important features. This process involves a Transfer Learning method, wherein layers from the Vision Transformer are converted by the Encoder Transformer and seamlessly integrated into a CNN Decoder. This methodology ensures minimal information loss from the original input image. The decoder employs a multi-decoding technique, utilizing dual decoders to generate two distinct attention maps. These maps are subsequently combined into a singular output via an additional CNN model. Our trained model MDS-ViTNet achieves state-of-the-art results across several benchmarks. Committed to fostering further collaboration, we intend to make our code, models, and datasets accessible to the public.