Spherical Vision Transformers for Audio-Visual Saliency Prediction in 360-Degree Videos

TL;DR

This paper introduces spherical vision transformer models that incorporate spatial audio cues to improve visual saliency prediction in 360-degree videos, supported by a new dataset and outperforming existing methods.

Contribution

It proposes two novel transformer-based models for 360-degree saliency prediction, integrating spherical geometry and audio cues, and provides a new dataset for this task.

Findings

Models outperform existing methods on benchmark datasets.

Inclusion of spatial audio cues improves saliency prediction accuracy.

The dataset enables further research in 360-degree audio-visual saliency.

Abstract

Omnidirectional videos (ODVs) are redefining viewer experiences in virtual reality (VR) by offering an unprecedented full field-of-view (FOV). This study extends the domain of saliency prediction to 360-degree environments, addressing the complexities of spherical distortion and the integration of spatial audio. Contextually, ODVs have transformed user experience by adding a spatial audio dimension that aligns sound direction with the viewer's perspective in spherical scenes. Motivated by the lack of comprehensive datasets for 360-degree audio-visual saliency prediction, our study curates YT360-EyeTracking, a new dataset of 81 ODVs, each observed under varying audio-visual conditions. Our goal is to explore how to utilize audio-visual cues to effectively predict visual saliency in 360-degree videos. Towards this aim, we propose two novel saliency prediction models: SalViT360, a vision-transformer-based framework for ODVs equipped with spherical geometry-aware spatio-temporal attention layers, and SalViT360-AV, which further incorporates transformer adapters conditioned on audio input. Our results on a number of benchmark datasets, including our YT360-EyeTracking, demonstrate that SalViT360 and SalViT360-AV significantly outperform existing methods in predicting viewer attention in 360-degree scenes. Interpreting these results, we suggest that integrating spatial audio cues in the model architecture is crucial for accurate saliency prediction in omnidirectional videos. Code and dataset will be available at https://cyberiada.github.io/SalViT360.

Tables

Table 1. Table II : Quantitative Evaluation on VR-EyeTracking and PVS-HMEM. Although the VR-EyeTracking dataset contains mono audio, it lacks spatial audio cues and is thus treated as a visual saliency benchmark. The PVS-HMEM dataset does not include sound. The models are grouped by category using background color: blue for 360 ∘ visual-only, orange for 2D audio-visual, and green for 360 ∘ audio-visual saliency models. Bold scores indicate the best performance, while underlined scores denote the second best.

	VR-EyeTracking				PVS-HMEM
Method	NSS$\uparrow$	KLD$\downarrow$	CC$\uparrow$	SIM$\uparrow$	NSS$\uparrow$	KLD$\downarrow$	CC$\uparrow$	SIM$\uparrow$
\rowcolorniceblue CP-360	0.624	15.338	0.165	0.240	0.576	4.738	0.162	0.198
\rowcolorniceblue ATSal	1.317	12.259	0.336	0.318	0.732	4.303	0.183	0.219
\rowcolorniceblue SalGAN360	1.753	10.845	0.370	0.355	1.513	4.394	0.314	0.291
\rowcolorniceblue MV-SalGAN360	1.818	8.713	0.382	0.357	1.546	4.112	0.316	0.295
\rowcolorniceblue PAVER	1.511	13.267	0.307	0.294	0.750	3.736	0.224	0.269
\rowcolorniceblue Djilali et al.	3.183	6.570	0.565	0.475	1.688	2.430	0.447	0.404
\rowcolorniceblue SalViT360	2.630	5.744	0.586	0.492	2.191	1.841	0.626	0.495
\rowcolorniceorange DAVE	1.821	12.199	0.331	0.304	1.644	4.252	0.309	0.273
\rowcolorniceorange STAViS	1.642	13.148	0.316	0.297	1.539	4.320	0.306	0.266
\rowcolorniceorange DiffSal	2.599	5.979	0.566	0.473	1.926	2.539	0.443	0.399
\rowcolorniceblue2 \rowcolornicegreen SalViT360-AV	2.821	5.334	0.599	0.511	2.191	1.841	0.626	0.495

Table 2. Table III : Evaluation on YT360-EyeTracking to Assess the Impact of Spatial Audio Integration. We compare visual-only, 2D audio-visual, 360 ∘ audio-only, and 360 ∘ audio-visual saliency models. Categories are color-coded: blue for 360 ∘ visual, orange for 2D audio-visual, green for 360 ∘ audio-visual, and gray for the 360 ∘ audio-only models.

Method	NSS$\uparrow$	KLD$\downarrow$	CC$\uparrow$	SIM$\uparrow$
\rowcolorniceblue CP-360	1.041	22.356	0.120	0.118
\rowcolorniceblue ATSal	1.711	13.948	0.242	0.244
\rowcolorniceblue SalGAN360	1.415	14.412	0.236	0.239
\rowcolorniceblue MV-SalGAN360	1.433	14.381	0.244	0.246
\rowcolorniceblue PAVER	1.093	14.830	0.254	0.226
\rowcolorniceblue Djilali et al.	1.738	11.085	0.369	0.326
\rowcolorniceblue SalViT360	2.346	9.861	0.484	0.373
\rowcolorniceorange DAVE	2.290	8.858	0.462	0.363
\rowcolorniceorange STAViS	1.535	12.039	0.343	0.294
\rowcolorniceorange DiffSal	2.293	8.458	0.477	0.383
\rowcolorniceblue2 SSSL	1.212	13.398	0.214	0.244
\rowcolornicegreen AVS360	2.170	7.801	0.457	0.374
\rowcolornicegreen SVGC-AVA	2.191	8.860	0.488	0.399
\rowcolornicegreenSalViT360-AV (w/ raw FOA)	2.443	8.539	0.508	0.406
\rowcolornicegreenSalViT360-AV (proposed)	2.449	8.341	0.512	0.407

Table 3. Table IV : Cross-Dataset Generalization Results on 360AV-HM, AVS-ODV, and SVGC-AVA. All datasets include ambisonic spatial audio. The models are grouped by category using background color: blue for 360 ∘ visual saliency, orange for 2D audio-visual saliency, gray for the 360 ∘ audio-only saliency, green for 360 ∘ audio-visual saliency prediction.

	360AV-HM					AVS-ODV					SVGC-AVA
Method	NSS$\uparrow$	KLD$\downarrow$	CC$\uparrow$	SIM$\uparrow$		NSS$\uparrow$	KLD$\downarrow$	CC$\uparrow$	SIM$\uparrow$		NSS$\uparrow$	KLD$\downarrow$	CC$\uparrow$	SIM$\uparrow$
\rowcolorniceblue ATSal	1.322	14.141	0.156	0.121		1.430	13.159	0.233	0.229		1.510	11.886	0.314	0.311
\rowcolorniceblue PAVER	1.019	19.136	0.176	0.115		1.619	13.252	0.251	0.241		1.641	11.302	0.422	0.396
\rowcolorniceblue Djilali et al.	2.050	16.618	0.308	0.209		2.014	10.167	0.344	0.300		2.020	10.239	0.511	0.419
\rowcolorniceblue SalViT360	2.285	15.879	0.349	0.246		2.192	8.626	0.390	0.321		2.179	9.320	0.557	0.434
\rowcolorniceorange DAVE	2.178	14.693	0.344	0.256		2.050	12.065	0.341	0.279		1.997	10.451	0.446	0.401
\rowcolorniceorange STAViS	1.479	17.934	0.237	0.162		2.012	12.195	0.340	0.277		1.991	10.543	0.443	0.400
\rowcolorniceorange DiffSal	2.460	13.991	0.376	0.271		2.368	11.866	0.398	0.317		2.276	8.274	0.541	0.408
\rowcolorniceblue2 SSSL	1.181	14.192	0.203	0.184		1.019	13.225	0.198	0.184		1.190	11.469	0.211	0.187
\rowcolornicegreen AVS360	2.501	13.581	0.380	0.290		2.386	8.130	0.394	0.332		2.320	9.245	0.559	0.433
\rowcolornicegreen SVGC-AVA	2.448	14.012	0.367	0.266		2.387	8.379	0.394	0.334		2.371	9.241	0.560	0.434
\rowcolornicegreen SalViT360-AV	2.473	13.830	0.379	0.278		2.389	8.481	0.397	0.332		2.342	9.251	0.564	0.440

Table 4. Table V : Ablation Study of SalViT360 Components on VR-EyeTracking. We evaluate the impact of core architectural components. Replacing 1D positional embeddings with spherical ones improves performance. Introducing our VSTA module yields further gains, and the VAC module—with masking—achieves the best results, demonstrating the benefit of spatial priors and temporal context.

Method	# params	NSS$\uparrow$	KLD$\downarrow$	CC$\uparrow$	SIM$\uparrow$
VSA (w/ 1D Pos. Emb.)	18.76M	2.518	6.445	0.560	0.472
+ Spherical Pos. Emb.	19.26M	2.575	6.221	0.563	0.475
VSTA (w/ Sph.  Pos. Emb.)	25.56M	2.664	6.174	0.570	0.479
+ VAC (w/o mask)	25.56M	2.624	6.011	0.576	0.490
+ VAC (w/ mask)	25.56M	2.630	5.744	0.586	0.492

Table 5. Table VI : Comparison of spatio-temporal modeling strategies. We compare our VSTA mechanism against a 2+1D CNN and offline EMA aggregation. VSTA outperforms both alternatives across all metrics, confirming the advantage of end-to-end learnable spatio-temporal attention in 360 ∘ video saliency prediction.

Method	# params	NSS$\uparrow$	KLD$\downarrow$	CC$\uparrow$	SIM$\uparrow$
VSA $+$ 2+1D-CNN Enc.	17.31M	2.568	5.915	0.568	0.477
VSA $+$ Offline EMA	19.27M	2.591	6.018	0.566	0.477
VSTA	25.56M	2.664	6.174	0.570	0.479

Table 6. Table VII : Effect of audio modality in ground truth saliency. We evaluate SalViT360 using saliency maps collected under mute, mono, and ambisonics audio. Performance improves with richer audio conditions, highlighting how spatial audio influences human attention—even when the model is trained without audio.

	360AV-HM				YT360-EyeTracking
Audio	NSS$\uparrow$	KLD$\downarrow$	CC$\uparrow$	SIM$\uparrow$	NSS$\uparrow$	KLD$\downarrow$	CC$\uparrow$	SIM$\uparrow$
Mute	1.961	16.819	0.289	0.225	2.115	9.798	0.461	0.370
Mono	2.230	16.356	0.329	0.238	2.315	9.993	0.478	0.371
Ambisonics	2.285	15.879	0.349	0.246	2.346	9.861	0.484	0.373

Table 7. Table A.1 : Quantitative comparison for space-only attention, the proposed Viewport Spatio-Temporal Attention and existing Joint Spatio-Temporal attention for omnidirectional video saliency prediction on the validation split of VR-EyeTracking dataset. Due to memory limitations, JSTA model could not be trained on our GPU.

Attention	# params	GFLOPs	NSS$\uparrow$	KLD$\downarrow$	CC$\uparrow$	SIM$\uparrow$
None	11.81M	0.00	2.306	7.718	0.497	0.434
VSA	30.78M	57.08	2.575	6.221	0.563	0.475
VSTA	37.07M	63.30	2.664	6.174	0.570	0.479
JSTA	30.78M	77.57	n/a	n/a	n/a	n/a

Table 8. Table A.2 : Quantitative comparison for the proposed VAC Loss with and without mask, on the validation split of the VR-EyeTracking dataset.

Model	NSS$\uparrow$	KLD$\downarrow$	CC$\uparrow$	SIM$\uparrow$
VSTA $+$ VAC (w/o mask)	2.624	6.011	0.576	0.490
VSTA $+$ VAC (w/ mask)	2.630	5.744	0.586	0.492

Table 9. Table B.1 : Performance comparison of three pre-trained audio models as the audio backbone of SalViT360-AV, on VR-EyeTracking, 360AV-HM, YT360-EyeTracking datasets.

	VR-EyeTracking xu2018gaze				360AV-HM 9301766				YT360-EyeTracking
Method	NSS$\uparrow$	KLD$\downarrow$	CC$\uparrow$	SIM$\uparrow$	NSS$\uparrow$	KLD$\downarrow$	CC$\uparrow$	SIM$\uparrow$	NSS$\uparrow$	KLD$\downarrow$	CC$\uparrow$	SIM$\uparrow$
AVSA morgado2020learning	n/a	n/a	n/a	n/a	2.499	12.916	0.381	0.284	2.501	7.768	0.516	0.417
EZ-VSL morgado2020learning	2.817	5.342	0.598	0.509	2.472	13.833	0.380	0.277	2.449	8.288	0.511	0.407
PaSST koutini22passt	2.821	5.334	0.599	0.511	2.473	13.830	0.379	0.278	2.449	8.341	0.512	0.407

Table 10. Table D.1 : Comparison with state-of-the-art saliency models for saliency-guided omnidirectional video quality assessment on VQA-ODV  Li:2018:BGV:3240508.3240581 dataset, as a downstream task.

Weight	PSNR			WS-PSNR Sun2017WeightedtoSphericallyUniformQE
	PCC$\uparrow$	SRCC$\uparrow$	RMSE$\downarrow$	PCC$\uparrow$	SRCC$\uparrow$	RMSE$\downarrow$
None	0.650	6.664	7.502	0.671	0.686	7.233
PAVER yun2022panoramic	0.661	0.667	7.481	0.679	0.691	6.914
Djilali et. al. Djilali_2021_ICCV	0.648	0.721	7.336	0.684	0.721	6.829
SalViT360 (ours)	0.688	0.733	7.295	0.689	0.737	6.673
HM (Supervised)	0.764	0.759	6.601	0.759	0.756	6.612

Equations

$$\text{VSTA}({\mathbf{z}}_{(t,f)}^{(l)})$$

$$\text{VTA}({\mathbf{z}}_{(t,f)}^{(l)})$$

$$\text{VSA}({\mathbf{z}}_{(t,f)}^{(l)})$$

$${\mathcal{L}}_{\text{VAC}}(\hat{\mathbf{y}},\bar{\mathbf{y}})=$$

$${\mathcal{L}}_{\text{KLD}}^{\text{weighted}}(\hat{\mathbf{y}},\bar{\mathbf{y}})=$$

$${\mathcal{L}}_{\text{CC}}^{\text{weighted}}(\hat{\mathbf{y}},\bar{\mathbf{y}})=$$

$${\hat{z}}_{\text{av, t}}=\text{ReLU}(\text{LN}({\bar{z}}_{\text{av, t}})\cdot {\mathbf{W}}_{down})\cdot {\mathbf{W}}_{up},$$

$${\mathbf{z}}_{\text{av, t}}=\text{MLP}({\bar{z}}_{\text{av, t}})+s\cdot {\hat{z}}_{\text{av, t}}+{\bar{z}}_{\text{av, t}}$$

$$F=(\sqrt{2}W+X)\cdot 2.$$

$$\mathcal{L}(P,Q_s,Q_f)$$

$$f((\theta ,\varphi ),t)={\sum }_{n=0}^N{\sum }_{m=-n}^n{Y}_n^m(\theta ,\varphi ){\alpha }_n^m(t),$$

$${\mathbf{\alpha }}_N(t)={\sum }_{i=1}^k{y}_N(({\theta }_i,{\varphi }_i))s_i(t).$$

$${\mathbf{\alpha }}_N^{\prime }(t)=\mathbf{R}\cdot {\mathbf{\alpha }}_N(t).$$

$$\begin{array}{rl}{\text{PSNR}}_{}=& 10{\log }_{10}\left(\frac{Y_{max}^2\cdot {\sum }_{p\in \mathbb{P}}{w}_{sal}(p)}{{\sum }_{p\in \mathbb{P}}(Y(p)-Y^{\prime }(p){)}^2\cdot w_{sal}(p)}\right)\end{array}$$

$$\begin{array}{rl}{\text{WS-PSNR}}_{}=& 10{\log }_{10}\left(\frac{Y_{max}^2\cdot {\sum }_{p\in \mathbb{P}}{w}_{sal}(p)\cos {\theta }_p}{{\sum }_{p\in \mathbb{P}}(Y(p)-Y^{\prime }(p){)}^2\cdot w_{sal}(p)\cos {\theta }_p}\right)\end{array}$$

Full text

1 Introduction

The rapid proliferation of virtual reality (VR) and multimedia streaming platforms has led to an unprecedented surge in the popularity and usage of omnidirectional or 360∘ videos (ODVs). These videos offer viewers a fully immersive experience by providing a complete field-of-view (FOV), fundamentally reshaping the way users interact with visual content. Accurately predicting viewer attention, termed visual saliency prediction, is increasingly critical in optimizing various multimedia tasks such as video compression 7962230 ; mishra2021multi ; 8692638 ; zhu2018spatiotemporal , perceptually-driven super-resolution hou2022multi , quality assessment guan2017visual ; yang2019sgdnet ; zhu2021saliency ; qiu2021blind ; jabbari2023st360iq , and adaptive streaming techniques like foveated rendering in VR environments patney2016towards . The success of these applications hinges on a precise understanding of where users naturally fixate their attention during video viewing.

Predicting visual saliency in ODVs, however, presents unique challenges not encountered with traditional 2D video content. The inherent spherical geometry of ODVs introduces significant spatial distortions when conventional projection methods, such as equirectangular projection (ERP) or cubemaps, are employed. ERP, while computationally simple, severely distorts content near the poles, and cubemaps introduce discontinuities at the edges, both negatively impacting model prediction accuracy. To overcome these challenges, in our recent work, we proposed SalViT360 cokelek2023spherical , a vision transformer (ViT) dosovitskiy2020image based model explicitly tailored for omnidirectional videos. SalViT360 uniquely addressed spherical distortions using locally undistorted tangent images generated through gnomonic projection eder2020tangent , and incorporated spherical geometry-aware spatio-temporal transformer attention. This model established state-of-the-art performance for visual-only saliency prediction in ODVs. Though, a critical limitation remained: the absence of audio modality integration, despite its significant role in guiding viewer attention in immersive scenarios.

Indeed, human perception naturally and continuously integrates auditory and visual stimuli. Numerous studies confirm that audio cues significantly modulate visual attention, guiding eye movements and enhancing viewer immersion tsiami2020stavis ; Tavakoli2019 ; min2020multimodal . In ODVs, spatial audio provides directional auditory cues aligned with visual elements, profoundly influencing where viewers focus their attention. Despite its recognized importance, existing computational models and datasets rarely consider audio-visual interactions comprehensively. To illustrate and emphasize the impact of spatial audio, we introduce an illustrative example in Fig. 1, which demonstrates how integrating spatial audio significantly alters visual attention patterns in omnidirectional contexts.

Motivated by the significant yet under-explored influence of spatial audio on visual attention, we present a novel, large-scale dataset named YT360-EyeTracking. This dataset is explicitly curated for audio-visual saliency prediction in ODVs, offering extensive annotations across various audio conditions, including mute, mono, and spatial (ambisonic) audio. YT360-EyeTracking addresses existing limitations by providing a robust foundation to systematically investigate how different audio modalities affect viewer gaze behavior in omnidirectional video scenarios.

In addition, we introduce SalViT360-AV, a novel audio-visual saliency prediction framework tailored specifically for 360∘ content. SalViT360-AV significantly expands upon our earlier SalViT360 model by incorporating a parameter-efficient transformer-adapter architecture inspired by AdaptFormer chen2022adaptformer . This enables effective integration of spatial audio cues into the transformer-based visual saliency prediction pipeline. By capturing complex interactions between audio and visual stimuli, SalViT360-AV achieves substantially improved prediction accuracy, setting a new state-of-the-art benchmark for audio-visual saliency prediction in omnidirectional contexts.

In summary, our contributions to the domain of audio-visual saliency prediction for 360∘ videos are as follows:

• •

We introduce YT360-EyeTracking, the first large-scale dataset explicitly designed for audio-visual saliency prediction in ODVs. It uniquely enables the systematic analysis of audio-visual interactions under varying audio modalities (mute, mono, ambisonics).

• •

We propose SalViT360-AV, a novel audio-visual saliency prediction framework that innovatively extends our previous visual-only model, SalViT360, by incorporating spatial audio through transformer adapters. This approach efficiently captures intricate audio-visual interactions, significantly improving model accuracy.

• •

We conduct comprehensive experiments across four benchmark datasets, including our newly introduced YT360-EyeTracking dataset, demonstrating that SalViT360-AV achieves state-of-the-art performance in predicting viewer attention, effectively highlighting the critical role of spatial audio integration in saliency prediction tasks.

2 Background and Related Work

Saliency prediction models aim to predict the eye fixations of humans exploring visual stimuli by producing saliency maps resembling ground-truth maps. The ground-truth annotations are collected from subjects with eye-tracking hardware and represented as fixation and saliency maps. Fixation maps contain the actual eye gaze data of humans. Fixations are binary for each pixel in the scene, corresponding to whether an observer has fixated on that location or not. Saliency maps are heatmaps generated by convolving the fixation maps with a Gaussian kernel to create a continuous measure of saliency or likelihood that a location attracts human attention. In this section, we provide an overview of 360∘ video saliency prediction models and datasets.

2.1 Overview of Saliency Prediction Models

Saliency prediction models can be divided into two categories based on the scene characteristic, namely, static and dynamic saliency prediction. Static saliency considers input in the image level and ignores temporal information between frames. Conversely, dynamic saliency includes both spatial and temporal cues in the videos. For $360^{\circ}$ saliency prediction, previous works primarily focused on addressing the representation of $360^{\circ}$ scenes, with each method trying to balance representative power and computational complexity. Nevertheless, most works ignore the audio modality, which contains essential stimuli that drive the human visual attention system. First, we review the $360^{\circ}$ static and dynamic saliency literature and give an overview of the audio-visual saliency models for 2D and $360^{\circ}$ domains.

360∘ Static Saliency Prediction. Chao et al.chao2018salgan360 introduced SalGAN360, utilizing cubemap projection to reduce ERP distortion. They fine-tuned the SalGAN modelPan_2017_SalGAN for cubemap faces, then reprojected the outcomes to ERP for final predictions. While this method lessens local distortion, it faces edge discontinuities due to CNNs’ inability to recognize adjacency across cube faces, leading to prediction errors at edges. To address this, the authors suggest employing multiple cubemap sets with horizontal and vertical shifts, resulting in $81$ projections per image, which significantly increases computational demand. In the follow-up work MV-SalGAN360 9122430 , the authors extended SalGAN360 with multi-view fusion. Unlike the single-scale cubemap projection, MV-SalGAN360 uses three scales. The first scale corresponds to the cubemap of six faces with $90^{\circ}$ FOV each. The second scale consists of five faces, each with an FOV of $120^{\circ}$. The last scale is the original ERP, with $180^{\circ}\times 360^{\circ}$ FOV. While the multi-view representation is important for local-global scene encoding, MV-SalGAN360 requires training a separate set of parameters for each level, which is computationally expensive.

Dahou et al. dahou2021atsal proposed ATSal, a two-stream model to compute global and local saliency in $360^{\circ}$ videos. They use global prediction as a rough attention estimate, and the local stream on cube faces predicts local saliency. The local predictions are projected back to ERP and linearly combined with the global map for the final saliency prediction. Zhang et al. Zhang_2018_ECCV developed a spherical U-Net for saliency prediction, applying modified kernels directly on ERP in spherical coordinates. These convolution kernels adapt to their location on the sphere, with dynamic receptive fields and a parameter-sharing approach for translation equivariance. However, computing a unique spherical crown for each convolution increases the model’s runtime complexity. Djilali et al. Djilali_2021_ICCV used a self-supervised pre-training based on learning the association between several different views of the same scene and trained a supervised decoder for $360^{\circ}$ saliency prediction as a downstream task. Their framework maximizes the mutual information between different views on the spherical image, and the decoder is trained with these embeddings. Although their approach can model the global relationship between viewports, it ignores the crucial temporal dimension for video understanding.

Zhu et al.zhu2018spic suggested a framework for generating separate head and eye saliency maps for static panoramas using some hand-crafted features and graphical models. Their follow-up zhu2020tmm extended this pipeline by combining visual equilibrium, uncertainty, and object cues to refine head and eye saliency predictions. A deep reinforcement learning agent was later introduced in zhu2020acmtmcca , encoding viewport content (CNN features, spherical coordinates, visited maps) and employing a free-energy-based reward to predict head movement trajectories in static 360∘ images.

360∘ Dynamic Saliency Prediction. Cheng et al. cheng2018cube propose cube-padding to address discontinuities at cube face boundaries, facilitating information transfer across faces through overlapping regions. Particularly for CNN models, the information exchange increases with deeper layers. The model also utilizes ConvLSTMs to capture temporal features. Qiao et al. qiao2020viewport found that eye fixation distribution varies with viewport locations and treated $360^{\circ}$ video saliency prediction as a multi-task challenge, aiming to predict global saliency on the sphere and local saliency within viewports. This insight into fixation biases based on viewport locations inspired us to incorporate spherical position information into our models. Zhu  et al. zhu2022ieeetcsvt fuse RGB and motion cues over high-clarity viewports with a graph-based approach to predict saliency, demonstrating that combining appearance and dynamic information at select viewports improves temporal consistency. Yun et al. yun2022panoramic developed PAVER, utilizing deformable convolutions to correct spherical distortion, assigning specific offsets per convolution. This setup allows for spatial and temporal self-attention in vision transformers using undistorted patch features.

Several recent works attempt to reduce spherical distortion through projections or modified kernels while performing dense prediction tasks. For instance, Eder et al. eder2020tangent introduced tangent image representations using gnomonic projection to convert spherical images into multiple overlapping patches, each tangent to an icosahedron face. Their method, which also inspired our approach, addresses distortion at a local level but lacks explicit modeling of global interactions across tangent viewports. Similarly, Cheng et al. cheng2018cube utilize cube projections but do not incorporate mechanisms to capture spatial dependencies across cube faces.

While such techniques do facilitate localized predictions, they fall short in capturing global saliency cues critical for comprehensive scene understanding. In particular, Zhu et al. zhu2022ieeetcsvt and SalViT360 do process the whole field-of-view by integrating multiple partial views, but their fusion strategies differ: Zhu et al. rely on late or local fusion of viewport-specific outputs, whereas SalViT360 explicitly models long-range dependencies across all viewports at each layer. Panoramic Vision Transformer qiao2020viewport does perform spatio-temporal aggregation via deformable convolutions, but its windowed attention remains restricted to local regions. In contrast, our method enhances the tangent image approach by integrating spatial and temporal attention across viewports, allowing the model to reason globally while maintaining distortion-aware local representations. We note, however, that Yun et al. yun2022panoramic perform spatio-temporal modeling through spatially deformable convolutions.

In this work, we propose using tangent images to process undistorted local viewports and develop a transformer-based model, SalViT360, to learn their global and spatio-temporal association for 360∘ video saliency prediction. Our transformer module is inspired by TimeSformer gberta_2021_ICML , which is developed for 2D videos. TimeSformer proposes Divided Space-Time Attention, which computes spatio-temporal attention in two stages to optimize for model complexity. We present Viewport Spatio-Temporal Attention, which extends the Divided Space-Time Attention idea to $360^{\circ}$ videos.

2D Audio-Visual Saliency. Min et al. 7457921 proposed one of the earliest audio-visual saliency prediction methods, emphasizing the role of audio in guiding gaze. Their method computes separate spatial, temporal, and audio attention maps where the audio attention is generated through canonical correlation analysis between motion patterns and audio features—to produce an integrated audio-visual saliency map. Subjective eye-tracking experiments confirmed that this fusion improves prediction quality over visual-only models. Tavakoli et al. Tavakoli2019 proposed DAVE, an audio-visual encoder framework for 2D video saliency prediction. DAVE comprises a two-stream encoder architecture for audio and visual modalities. The video clips and audio mel-spectrograms are encoded by 3D ResNets in each stream, concatenated in the feature space, and then decoded with a 2D CNN to obtain the saliency predictions. Tsiami et al. tsiami2020stavis introduced STAViS, a spatio-temporal audio-visual saliency model that also adopts a two-stream CNN-based encoder. They further proposed a bilinear audio localization module that fuses the audio and visual features before decoding them into saliency maps.

Min et al. min2020multimodal presented the MMS model for fusing audio-visual cues under the assumption of high audio-visual correspondence. Their framework emphasizes the motion-consistent audio sources and performs late fusion with existing saliency maps, though it is less effective when visual and auditory cues diverge. Chen et al. chen2021audiovisual proposed an alternative architecture for two-stream feature extraction, semantic interaction, and their fusion for auditory and visual inputs. Semantic interaction brings the audio-visual features from lower and higher levels to the same dimensionality, demonstrating superior performance than fusing the features from only the final layers.

Zhu et al. Dandan2023 propose LAVS, which extracts visual features using a VGG-16 backbone with ConvLSTM and audio features via separable convolutions on MFCC, CFCC and CQT features. Cross-modal alignment is achieved through deep canonical correlation analysis, and the resulting audio map is fused with the visual map, yielding competitive accuracy with minimal computational overhead. More recently, Zhu  et al. 10422835 proposed MTCAM, a Transformer-based weakly supervised model for audio-visual saliency prediction. MTCAM jointly models audio and visual cues using separate encoders and a unified transformer-based fusion mechanism. Xiong et al. Xiong_2024_CVPR introduced DiffSal, a diffusion-based audio-visual saliency model that unifies saliency learning with generative modeling. DiffSal jointly learns audio and visual representations through a multi-stage diffusion process and enables fine-grained and high-resolution prediction with improved temporal coherence.

360∘ Audio-Visual Saliency. Spatial audio has been increasingly recognized as a critical cue for saliency prediction in omnidirectional videos (ODVs), particularly due to its alignment with immersive VR experiences. Several recent works have explored this direction.

Chao et al. chao2020towards proposed a two-stream pipeline where ERP frames are projected to padded cube maps and processed via a 3D ResNet, while mono audio mel-spectrograms are encoded separately. The resulting features are fused and enhanced with an audio energy map (AEM) that estimates the direction of arrival based on first-order ambisonics. Cokelek et al. cokelek2021leveraging introduced an unsupervised, audio-only method called MCSR, which computes directional audio saliency via mel-cepstrum features across six primary directions, projecting this as a bias onto ERP-based maps. More recently, Zhu  et al. 10109890 proposed AVPS, a unified saliency prediction model integrating both spatial audio localization and content-level audio features. Their approach combines improved group-equivariant CNNs with E3D-LSTMs to capture visual dynamics and computes spatial audio localization via AEM, further complemented by SoundNet-based attribute-aware audio embeddings. In another work, Zhu  et al. 11003418 presented OmniAVS, a fully convolutional framework that introduces a learnable spatial audio map (SAM) based on ambisonic energy. OmniAVS combines this SAM with visual features using an attention-based fusion module that models temporal and directional saliency consistency. Yang et al. yang2023svgc introduced SVGC-AVA, a graph-based model leveraging spherical vector-based graph convolution and audio-visual attention fusion. Their model encodes spherical geometry and spatial audio jointly to predict fixations.

It is worth noting that the aforementioned recent works are concurrent with ours and some of them have not yet released code or pretrained models. As such, we are unable to provide direct quantitative comparisons. Nevertheless, we include all these works here to present a comprehensive overview of the current research landscape in 360∘ audio-visual saliency prediction. Our SalViT360-AV model advances this research line by integrating spatial audio via transformer adapters conditioned on ambisonic input, allowing for dynamic, fine-grained cross-modal fusion within an end-to-end trainable architecture. Compared to previous efforts, it captures deeper semantic and spatial correlations, leading to superior or competitive performance on a large number of benchmark datasets.

2.2 360∘ Video Saliency Datasets

Several datasets have been proposed for visual saliency prediction in omnidirectional videos (ODVs), capturing varying levels of realism, scale, and audio modality.

Salient360 dataset david2018dataset is one of the earliest datasets to provide head movement (HM) and eye fixation (EM) annotations over ERP or cubemap projections. While it includes diverse content, it remain relatively small in size.

PVS-HMEM dataset xu2018predicting includes head movement and eye-tracking data collected from 76 $360^{\circ}$ video sequences from 58 subjects. The videos vary between $10-80$ secs, with a minimum resolution of 2K and frame rate between $24-60$fps. They includes indoor, outdoor, movie, and sports categories. The data collection is done in a muted modality.

VR-EyeTracking dataset xu2018gaze consists of eye tracking data collected for 208 videos (134 train, 74 test) lasting between 20-60 seconds. The videos have a minimum resolution of 4K in ERP format with frame rates between 24-60 fps. There are 20 female and 25 male subjects aged between 20-24 years. At least 31 subjects view each video, with each subject watching around 35 videos. Videos are picked under various scene categories, including indoors, outdoors, sports, games, documentation, short movies, and music shows. Subjects have viewed the videos with mono audio modality, as in traditional 2D videos. Mono modality delivers sound sources equally from all directions. Hence, the viewers cannot be guided by directional sound sources. In contrast, $360^{\circ}$ videos can be delivered with spatial audio, where the sound sources are precisely located, and the perceived sound changes with head movement (roll, pitch, yaw).

SVGC-AVA dataset yang2023svgc is a recent benchmark proposed to evaluate audio-visual saliency in ODVs. It comprises 57 360∘ videos recorded with spatial audio, annotated with eye-tracking data from 63 subjects in free-viewing conditions, supporting both ERP and cubemap formats. The scenes include complex audio-visual events such as overlapping sounds and directionally distinct audio sources, providing a challenging benchmark for multimodal attention modeling.

360AV-HM dataset chao2020audio consists of 15 ODVs with spatial audio lasting 25 seconds. Each video has a resolution of $3840\times 1920$ in ERP format, with frame rates between $24-60$fps. The videos are collected from YouTube under three categories, Conversation, Music, and Environment. The videos are viewed under mute, mono, and spatial audio settings by three subject groups, where each subject has only viewed the videos with one modality. Each video was viewed by fifteen subjects, and there are a total of 16 female and 29 male subjects aged between $21-40$ years.

AVS-ODV 11003418 dataset, proposed concurrently with our work, includes 162 ODV clips spanning human activities, natural scenes, vehicles categories and provides saliency annotations under three audio conditions (mute, mono, spatial). The clips are sourced from YouTube and feature real-world audio-visual correlations. The dataset is recorded with 15 participants using a VR headset and includes a thorough psychophysical analysis of how different audio configurations influence gaze behavior.

Together, these datasets illustrate the progression of 360∘ saliency research from smaller-scale, visual-only datasets to larger-scale benchmarks integrating multimodal annotations, particularly spatial audio. As detailed in Table I, our YT360-EyeTracking dataset, as compared to the recently introduced SVGC-AVAyang2023svgc and AVS-ODV11003418 , employs a significantly larger number of subjects. Additionally, YT360-EyeTracking explicitly explores the influence of audio modality by systematically providing annotations under three audio conditions (mute, mono, ambisonics), closely matching the concurrent AVS-ODV dataset in experimental conditions. But it also ensures a balanced gender representation among participants, featuring 53 female and 49 male subjects. These comparative strengths make our dataset particularly suitable for training deep-learning-based multimodal models such as SalViT360-AV, facilitating detailed analyses of audio’s role in guiding visual attention in omnidirectional video scenarios.

3 YT360-EyeTracking Dataset

Overview. We picked 81 videos from YouTube-360 morgado2020learning by carefully considering nine audio-visual semantic categories and selecting nine videos per category, viewed under three audio conditions (mute, mono, spatial audio) and two color conditions (colored, grayscale111Including grayscale video allows isolating luminance-driven attention from chromatic effects, offering a clearer view of the role of color in visual saliency. Psychophysical studies have shown that color can influence gaze allocation and scene interpretation in meaningful ways nuthmann16eye ; frey08what , making grayscale conditions a valuable complementary modality for exploring cross-modal and cross-condition saliency patterns. See ozsoy2024msc for related preliminary psychophysical analyses.). The audio semantic categories are (1) speech, (2) music, (3) vehicle, and visual categories are (1) indoors, (2) outdoors-natural, and (3) outdoors-human-made. Sample frames from each audio and scene category are given in Fig. 2. The distribution of samples for each audio and scene category is given in detail in Fig. 3. Each video in our dataset contains dynamic scenes rather than static images and does not have cutscenes. The video clips last 30 secs, and have a resolution of 3840$\times$1920 pixels in ERP format, with frame rates between $24-30$fps. Each video is watched by at least 15 observers under specific audio-visual conditions, with each observer viewing the video only once. YouTube video IDs, segment start timestamps, raw fixation data, as well as data preparation and post-processing scripts, will be made available to the public.

3.1 Stimuli

Videos. All clips within the YT-360 dataset were extracted as 10-second segments from longer videos, and there are multiple consecutive clips from the same videos. After re-downloading the original videos and trimming the reference sequences to 30 seconds, we obtained our new stimuli for ODV saliency. Moreover, we chose to exclusively incorporate high-quality videos, with a minimum resolution of $3840\times 1920$, and spatial audio modality. To facilitate the data filtering process while also keeping valuable semantic segmentation information in our new dataset, we deliberately chose the subset of the original sequences in YT-360.

Data Filtering and Labeling. We employed learning-based models chen2020vggsound ; gemmeke2017audio ; zhou2018places360 for initial video annotation, and selected the top 300 videos based on their annotation scores for a more detailed evaluation. For each of these videos, we manually labeled both the audio and scene categories. To streamline this process, we developed a website equipped with an intuitive interface specifically designed for video annotation. The annotation interface prompted researchers to identify the predominant type of audio in the video. The options provided were: vehicle sound, musical instrument, human speech, human vocalizations, other, and noise. For the environmental context, the interface asked whether the video was recorded indoors or outdoors, with further distinctions made between natural and human-made outdoor environments. We also incorporated several control questions to improve the overall stimuli quality. These questions aimed to identify issues such as black screens, sudden scene changes, and the origin of the audio (whether it was captured live or added in post-production). Lastly, we evaluated the ability to localize sound sources within the videos and the presence of text overlays, which helped us eliminate any erroneous content from our dataset. Following the initial annotation and filtering process, we are left with videos divided into three audio (speech, music, vehicle) and three visual (indoors, outdoors-human-made, outdoors-natural) categories.

To determine the number of stimuli for the dataset, a priori power analysis was conducted using G*Power version 3.1 faul2009gpower . Results indicated that to achieve $95\%$ power for detecting a medium effect ($\mathit{f}=0.25$), at a significance criterion of $\alpha=.05$, the required sample size was $\mathit{N}=63$ for a mixed model ANOVA (3$\times$3 between factors of sound and scene types, 3$\times$2 within factors of sound and color modalities) corresponding to at least $\mathit{n}=7$ videos per audio-visual category. Consequently, we chose nine representative videos for each audio-scene combination, resulting in a curated set of 81 videos, ensuring a diverse and representative dataset.

Pre-processing. Videos only available in higher resolutions were downsized to $3840\times 1920$ pixels via FFmpeg tomar2006converting_ffmpeg using bicubic interpolation to ensure consistency. In addition to the original first-order ambisonics, we altered all audio files to mono and mute versions by removing the audio channels. Moreover, we converted videos to grayscale by keeping only the Y (luma) component of the YUV format. As a result of these augmentations, we obtained a total of 486 distinct stimuli from 81 clips.

3.2 Eye-Tracking Data Collection

As illustrated in Fig. 4, our study employed a free-viewing task structured into three trials, with a 5-min. intermission between each to minimize participant fatigue. Each session began with eye calibration to ensure accuracy in tracking. Participants were then presented with a sequence of 27 stimuli in each trial, comprising a 30-second video clip accompanied by corresponding audio. To mitigate any carry-over effects, each stimulus was followed by a 10-second interval displaying a black screen. Overall, each video was viewed by each participant under one of six audio-visual conditions. Fig. 5 illustrates the variation in fixation density maps across these different audio modalities.

Before commencing the trials, we provided participants with guidelines to standardize the viewing experience. They were instructed to stand and view the 360° videos, allowing rotation but maintaining their position. Participants were informed about the lengths of trials, video clips, and intervening black screen periods. Although the specifics of the video content were not detailed, they were told to expect various presentations, including grayscale, colored, mute, and audiovisual formats, with an assurance of no distressing content to reduce surprises.

To examine the influence of audio and visual variables, a crossed experimental design was utilized. This design ensured that each participant experienced every video clip under specified audio and color conditions, with the sequence of these presentations randomized to control for order effects. Consistency in the viewing experience was maintained by standardizing the initial viewing angle, resetting the participant’s yaw rotation to 0 degrees at the onset of each stimulus presentation.

Apparatus. We used the HTC Vive Pro-Eye222https://vive.com/sea/product/vive-pro-eye/overview/ headset with a Tobii Eye Tracker333https://tobii.com/products/integration/xr-headsets/ for precise eye movement tracking. The virtual viewing environment was engineered in Unity3D444https://unity.com/. To facilitate free movement and rotation without cable entanglement, the headset cable was connected to a ceiling-mounted extensible and adjustable mechanism.

Participants. The study engaged 102 participants (53 females and 49 males), aged 18 to 33, predominantly affiliated with Boğaziçi University, Istanbul, TR. All had normal or corrected-to-normal vision, with corrections via contact lenses or glasses where necessary. Participant anonymity and data confidentiality were upheld through the use of randomly assigned IDs. The research protocol was approved by the Boğaziçi University Ethics Coordinating Committee555https://bogazici.edu.tr/en_US/Content/About_BU/Governance/Councils_Boards_and_Committees/Ethics_Committees, ensuring ethical compliance. Informed consent was obtained from all participants, who were briefed on the study’s procedures and their right to withdraw at any time.

3.3 Post-processing

Initially, the experiment records were synchronized with the corresponding tracking data, providing gaze and head direction details for each stimulus per participant. Fixations were extracted using the I-DT algorithm, as described in agtzidis2019eq_idt . Fixation parameters included a maximum dispersion of 1.5 visual angles and a minimum duration of 0.1 seconds. These fixation characteristics were then saved in comma-separated value text files. The chosen thresholds considered both the device’s accuracy range of 0.5-1.1 degrees and established guidelines for I-DT thresholds david2023salient360 ; schuetz2022vive_accuracy ; blignaut2009dispersion_thresh ; salvucci2000fixation_extraction .

4 Methodology

In this section, we first describe SalViT360, our vision-transformer-based 360∘ video saliency model. Then, we present SalViT360-AV, the unified audio-visual saliency prediction pipeline for 360∘ videos. SalViT360-AV is a two-stream architecture for audio and visual modalities. Both streams are pre-trained and frozen, and we introduce transformer adapters conditioned on audio features for parameter-efficient fine-tuning of the video stream.

4.1 Spherical Vision Transformers for 360∘ Video Saliency Prediction

In Fig. 6, we present the overview of SalViT360. We start with gnomonic projection eder2020tangent to obtain tangent images for each frame in the input video clip. The tangent images are passed to an encoder-transformer-decoder architecture. The image encoder is used to extract local features for each tangent viewport and reduce the input dimension for the subsequent self-attention stage. We map the pixel-wise angular coordinates to produce the proposed spherical geometry-aware position embeddings $\mathcal{F}(\phi,\theta)$ for the 360∘ transformer, enabling enriched spatial representations. The transformer utilizes our proposed Viewport Spatio-Temporal Attention (VSTA) to capture inter and intra-frame information across tangent viewports in a temporal window of $T=8$ frames. The transformed embeddings are then fed into a 2D CNN-based decoder, which predicts saliency on the tangent images. We then apply inverse gnomonic projection on the tangent predictions to obtain the final saliency maps in ERP. We propose an unsupervised consistency-based Viewport Augmentation Consistency Loss to mitigate the discrepancies after inverse gnomonic projection. The learnable parameters of the network are in tangent space, allowing us to leverage large-scale pre-trained 2D models (e.g., ResNet-18) for feature extraction while the rest of the network is trained from scratch.

Gnomonic Projection and Encoder. We project the input ERP clip $x\in\mathbb{R}^{F\times 3\times{H}\times{W}}$ to a set of tangent clips $\{x_{\textit{t}}\in\mathbb{R}^{F\times 3\times p\times p}\}_{t=1}^{T}$, with $F$, $3$, $H$, and $W$ indicating the number of frames, channel dimension (RGB), height, and width of the video, respectively. These tangent images have a patch size of $p\times p=224\times 224$ pixels. We select $T=18$ tangent images per frame and a $80^{\circ}$ field-of-view based on li2022omnifusion . After downsampling and flattening the encoder features, we obtain tangent feature vectors $\{{z_{\textit{t}}\in\mathbb{R}^{F\times d}}\}_{t=1}^{T}$. In a parallel stream, we map the angular coordinates $(\phi,\theta)$ for each pixel of the tangent viewports to the feature dimension $d=512$ using an FC layer and element-wise add these embeddings with encoder features to obtain the proposed spherical geometry-aware tangent embeddings $\left\{\mathbf{z}_{\textit{t}}\in\mathbb{R}^{F\times d}\right\}_{t=1}^{T}$ that are used in the transformer.

Viewport Spatio-Temporal Attention. While the pre-trained encoder extracts rich spatial features for each tangent image locally, aggregating the global context in the full FOV is essential for 360∘ scene understanding. We propose a self-attention mechanism on tangent viewport features to achieve this. However, since incorporating the temporal dimension of the videos increases the number of tokens, thus the computational complexity, inspired by the divided space-time attention mechanism in TimeSformer gberta_2021_ICML , we approximate spatio-temporal attention with two stages: we apply temporal attention (1) among the same tangent viewports from consecutive $F$ frames, then, spatial attention (2) among $T$ tangent viewports in the same frame. This reduces the overall self-attention complexity from $F^{2}\times{T^{2}}$ to $F^{2}+T^{2}$, effectively capturing the required global context for 360∘ video analysis. Our Viewport Spatio-Temporal Attention for 360∘ videos illustrated in Fig. 7 is defined as:

[TABLE]

where $\mathbf{z}^{(l)}_{(t,f)}\in\mathbb{R}^{d}$ denote the tangent features of viewport $t$ in frame $f$ at $l$-th transformer block, $\mathbf{q},\mathbf{k},\mathbf{v}$ are the query, key, and value projections of $\mathbf{z}$, and SM is the softmax operator. Attention heads and the scale multiplication of dot-product attention are not given for presentation clarity.

Decoder. The decoder comprises four upsample layers followed by $3\times 3$ convolutions and normalization layers. For a set of tangent clips, it takes the skip connection of encoder and transformer features $\{\mathbf{\hat{z}}_{\textit{t}}\in\mathbb{R}^{512\times 7\times 7}\}_{t=1}^{T}$ of the last frame as input and outputs saliency prediction $\{\mathbf{\hat{y}}_{\textit{t}}\in\mathbb{R}^{p^{\prime}\times p^{\prime}}\}_{t=1}^{T}$ on tangent planes. The final ERP saliency maps are obtained by passing the tangent predictions to inverse gnomonic projection. While we aggregate global information among tangent images through the transformer, each tangent plane is predicted separately in the decoder. This causes discrepancies in the overlapping regions on ERP. Prior work on omnidirectional depth estimation li2022omnifusion proposes iterative refining to compensate for these artifacts, requiring the whole model to run twice per sample during training and inference. To address this, we propose Viewport Augmentation Consistency, an unsupervised loss strategy as a regularizer that requires zero parameters and has no time overhead. It is trained on multiple tangent scales in parallel and requires only one tangent set for inference.

Viewport Augmentation Consistency. Our model learns saliency distribution with a supervised loss but faces discrepancies in ERP saliency maps due to separate predictions for each tangent plane. To address this, we propose an unsupervised Viewport Augmentation Consistency (VAC) loss to enhance prediction consistency across tangent projections. Specifically, we generate the second tangent set by applying different configurations, such as horizontally shifting the tangent planes on the sphere, using a larger FOV for the same viewports, and varying the number of tangent images at different viewports. We provide a detailed comparison of these approaches in the appendix. VAC does not require any additional memory or time overhead since it uses the shared parameters of the whole model, and the forward pass is done in parallel. Furthermore, since the ERP predictions from the original $P$ and augmented $P^{\prime}$ tangent sets are expected to be consistent, only one tangent set is sufficient for testing. The VAC loss is defined as:

[TABLE]

where $\hat{\mathbf{y}},\ \bar{\mathbf{y}}$ are the saliency predictions from original and augmented viewports, and ${w}$ is an optional weight matrix obtained from gnomonic projection to weigh the overlapping pixels of gnomonic projection on ERP predictions. Details for viewport augmentation approaches and the weighting operation are provided in the appendix.

4.2 Spherical Vision-Transformer Adapters for 360∘ Audio-Visual Saliency

In our prior work, SalViT360, we successfully leveraged spatio-temporal features for understanding $360^{\circ}$ scenes. However, SalViT360 solely relied on visual stimuli, overlooking the influential role of audio cues. Motivated by the strong impact of audio in guiding human visual attention, as discussed in Section 2, we have enhanced SalViT360 with spatial audio encoding capabilities, leading to the development of SalViT360-AV. As illustrated in Fig. 8, SalViT360-AV integrates two streams for jointly processing audio and video: the video branch (top) uses the original SalViT360 pipeline, while the audio branch (bottom) can flexibly incorporate any pre-trained audio backbone. It is important to note that when ambisonics are not available, the spatial audio stream in SalViT360-AV can be conveniently substituted with mono audio.

Audio Backbone. The audio stream receives spatial audio waveforms $x_{\text{aud}}\in\mathbb{R}^{4\times N}$ encoded as first-order ambisonics in 4-channel B-format containing audio streams $(W,Y,Z,X)$. To simulate what a subject would hear while gazing in a specific direction, we rotate the ambisonics for each tangent viewport, defined by angular coordinates $\{(\theta^{t},\phi^{t})\}_{t=1}^{T}$, producing mono directional waveforms $\{x_{t}\}_{t=1}^{T}$. This is achieved by applying a spherical harmonics rotation transformation to the ambisonics channels kronlachner2014spatial ; morgado2018self . This approach enables the use of any pre-trained audio model such as PaSST koutini22passt , EZ-VSL mo2022localizing or AVSA morgado2020learning for feature extraction. The resulting audio features $\mathbf{x}^{t}_{\text{aud}}\in\mathbb{R}^{\hat{d}}$ are then aligned with their corresponding visual tokens for each viewport.

In our experiments, we evaluated all three backbone models, and empirically selected PaSST as the main audio encoder. PaSST stands out with its self-attention mechanism on mel-spectrogram patches and its unique training approach, involving ’patchout’, a dropout-like mechanism. Its training leverages DeiT’s pre-trained weights from ImageNet, further adapted on AudioSet for enhanced audio classification capabilities. The model processes input mel-spectrograms, extracted using a window of 25ms and a hop length of 10ms, to produce feature vectors $\mathbf{x}^{t}_{\text{aud}}\in\mathbb{R}^{\hat{d}}$ for audio clips. The duration for these audio clips is set to 4 seconds, based on empirical observations.

During the development of our framework, we considered several alternatives to this ambisonics decoding strategy. One baseline fed the raw B-format channels $(W,Y,Z,X)$, as either waveforms or mel-spectrograms, directly into the model. However, this configuration consistently underperformed compared to our viewport-based decoding. The core advantage of our adopted strategy is perceptual alignment: saliency in $360^{\circ}$ scenes often relies on spatially localized audio cues, and decoding ambisonics into view-aligned signals exposes these cues in a geometry that directly matches the tangent visual representations. This alignment enables more effective audio–visual fusion, as features share a common spatial frame of reference.

We also experimented with learning a mapping from four global channels to eighteen spatial bins through an additional trainable module such as a cross-attention mapper. While this partially addressed the alignment issue, it increased both model size and inference time, without matching the efficiency or predictive accuracy of the viewport-based strategy. As a result, we selected the current design, which ensures a direct and efficient one-to-one correspondence between audio and visual streams and allows flexible integration of state-of-the-art audio backbones. Further implementation details and a comparative analysis of backbone choices are provided in Appendix B.

Adapter Architecture. As shown in Fig. 9, SalViT360-AV uses a parallel transformer adapter structure inspired by AdaptFormer chen2022adaptformer , with a frozen Viewport Spatio-Temporal Attention (VSTA) block from SalViT360 and a lightweight bottleneck adapter for audio-visual fusion. The model processes audio tokens $\{{z}_{\text{aud}}\in\mathbb{R}^{\hat{d}}\}_{t=1}^{T}$ alongside visual tokens $\{\mathbf{z}^{\textit{(l)}}_{\text{(t, f)}}\in\mathbb{R}^{d}\}_{(t=1,f=1)}^{(T,F)}$. This dual-stream setup involves a stack of down-projection and up-projection layers, interconnected with ReLU non-linearity, functioning as a bottleneck with significantly lower dimensions than the original embeddings. Specifically, this stream is designed as a bottleneck module with a dimension of $k$, as the projections are performed by parameters $\mathbf{W}_{down}\in\mathbb{R}^{{(d+\hat{d})}\times k}$, and $\mathbf{W}_{up}\in\mathbb{R}^{k\times\hat{d}}$, where $k\ll d,\hat{d}$. The concatenated audio-visual embeddings $\{\bar{z}_{\text{av, t}}\in\mathbb{R}^{d+\hat{d}}\}_{t=1}^{T}$ for each tangent viewport undergo a transformation to generate audio-conditioning features $\hat{z}_{\text{av, t}}$ as:

[TABLE]

which are then fused with the visual features $\bar{z}_{\text{av, t}}$ to enhance the saliency prediction capability of the model:

[TABLE]

Our adapter-based design ensures that when no audio input is provided such as in purely visual 360∘ saliency datasets, the audio branch transmits no signal to the adapter layers. In this case, SalViT360-AV behaves identically to SalViT360, and the model effectively reduces to single-modal visual saliency prediction without any loss of performance or the need for re-training. This flexibility allows our framework to generalize seamlessly across both single-modal (visual-only) and multi-modal (audio-visual) saliency prediction tasks.

5 Experiments

Our experimental evaluation was structured in a two-fold approach. In the first part, focusing on vision-only settings, we trained the SalViT360 model using the VR-EyeTracking dataset xu2018gaze . This dataset comprises 134 videos for training and 74 videos for testing. For validation, we randomly selected a subset of 30 videos from the training set and assessed the performance of SalViT360 on the test set. Additionally, to evaluate the model’s generalization ability, we conducted cross-dataset evaluations using the PVS-HMEM xu2018predicting , 360AV-HM chao2020audio datasets, and the test split of our YT360-EyeTracking dataset. These datasets include 76, 21, and 27 videos, respectively, and each were viewed by 58, 15, and 15 subjects. In the second part of our experimentation, we focused on audio-visual tuning of the SalViT360-AV model. We began by utilizing the pre-trained weights of SalViT360 from the VR-EyeTracking dataset, which we kept frozen throughout this phase. The training of SalViT360-AV was conducted using the training split of the YT360-EyeTracking dataset. We then evaluated SalViT360-AV’s performance on various datasets, including the test set of VR-EyeTracking, the YT360-EyeTracking test split, and the 360AV-HM dataset.

Data Pre-processing. The audio waveforms are resampled to a frequency of 32.0 kHz to align with the PaSST backbone. The duration of audio clips is empirically chosen to be a maximum of 4 seconds. This choice is based on discussions in cokelek2021leveraging , which suggest that shorter clips tend to capture short-term audio saliency, while longer clips may encompass more than local saliency. A detailed comparative analysis of clip durations can be found in the Appendix.

After rotating the 4-channel ambisonics for each viewport, we decode them to extract a single-channel directional waveform $F$ for mono audio backbones (like PaSST and EZ-VSL), using the formula:

[TABLE]

Evaluation Metrics and Loss Functions. We evaluate the performance of the models using four commonly used metrics, namely, Normalized Scanpath Saliency (NSS), KL-Divergence (KLD), Pearson’s Correlation Coefficient (CC), and Similarity Metric (SIM). We compute the training objective as a weighted differentiable combination of KLD, CC, and Selective-MSE (MSE on normalized saliency maps at only eye-fixation points salfbnet ) as given below:

[TABLE]

where $P,\ {Q}_{s},\ {Q}_{f}$ are the predicted saliency, ground truth density and fixation maps, respectively, and $\alpha=0.05$.

Architecture and Optimization Details. We trained SalViT360 for 100K iterations using AdamW loshchilov2017decoupled with momentum parameters $(\beta_{1},\ \beta_{2})=(0.9,\ 0.999)$, a weight decay of $1e-2$ and an initial learning rate of $1e-5$. A cosine scheduler was applied, decaying the lr to $2e-6$ over the course of training. The batch size was set to $16$, and all experiments were conducted on a single Tesla V100 GPU with 32 GB of memory. Subsequently, we performed parameter-efficient fine-tuning of SalViT360-AV for an additional 20,000 steps using the same optimizer with a fixed learning rate of 2e-6.

5.1 Empirical Evaluation Across Visual and Audio-Visual Saliency Benchmarks

We conduct an extensive evaluation of our proposed SalViT360 and SalViT360-AV models, systematically examining their performance across a diverse set of benchmarks. Our analysis is structured into three main parts: (1) evaluation on datasets without spatial audio, which primarily assess visual saliency models; (2) assessment on our proposed YT360-EyeTracking dataset to evaluate the benefits of integrating spatial audio; and (3) cross-dataset generalization on unseen spatial audio-visual datasets.

Evaluation on Datasets without Spatial Audio. We first evaluate the effectiveness of our visual transformer model SalViT360 using the widely adopted VR-EyeTracking dataset xu2018gaze , which, despite containing mono audio, lacks spatial audio annotations and thus serves primarily as a visual-only benchmark. VR-EyeTracking provides an extensive test scenario, comprising 134 training and 74 testing videos. To comprehensively assess the model’s generalization beyond training data, we further evaluate its performance on the PVS-HMEM dataset xu2018predicting , which features purely visual annotations without any audio content. Quantitative results provided in Table 5.1 clearly demonstrate that SalViT360 achieves top-tier performance, frequently surpassing existing state-of-the-art visual-only saliency models. In particular, our transformer-based architecture shows substantial advantages in capturing complex visual contexts and accurately predicting viewer fixation patterns.

Regarding the audio-visual evaluation on these datasets, we note that 360∘ audio-visual saliency models, apart from our SalViT360-AV, could not be directly evaluated on the PVS-HMEM dataset, as it entirely lacks audio data. Moreover, they were incompatible with the VR-EyeTracking dataset, as they require spatial audio to process. On the other hand, our SalViT360-AV model can effectively process mono audio waveforms or even no audio (see our discussion at the end of Sec. 4.2); thus, we can provide results on both datasets. As expected, in the presence mono audio, SalViT360-AV demonstrate improved predictive accuracy compared to existing 2D audio-visual models on the VR-EyeTracking dataset. As shown in Fig. 10, SalViT360 and SalViT360-AV generate saliency maps that accurately localize human gaze across the provided panoramic scenes. Our models reliably focus on the most visually salient regions, even in cases where attention is narrowly concentrated. In contrast, previous visual-only 360∘ models as well as 2D audio-visual models often produce saliency maps that are either overly blurred or misplace attention, highlighting irrelevant parts of the scene. Our models are less prone to center or equator bias and more reliably highlight peripheral yet visually relevant regions, yielding predictions that are much closer to the ground truth fixations.