Self-view Grounding Given a Narrated 360{\deg} Video

TL;DR

This paper introduces a novel visual grounding model that automatically predicts the normal field of view in narrated 360-degree videos by integrating video content and subtitles, enhancing user guidance without human supervision.

Contribution

The proposed Visual Grounding Model (VGM) efficiently combines CNN and RNN with attention mechanisms to accurately ground NFoVs in 360-degree videos using subtitles, without requiring manual annotations.

Findings

Achieved state-of-the-art NFoV-grounding performance on a new dataset.

Effectively integrates video features and subtitles for accurate NFoV prediction.

Introduced a reverse sentence training strategy to improve model robustness.

Abstract

Narrated 360{\deg} videos are typically provided in many touring scenarios to mimic real-world experience. However, previous work has shown that smart assistance (i.e., providing visual guidance) can significantly help users to follow the Normal Field of View (NFoV) corresponding to the narrative. In this project, we aim at automatically grounding the NFoVs of a 360{\deg} video given subtitles of the narrative (referred to as "NFoV-grounding"). We propose a novel Visual Grounding Model (VGM) to implicitly and efficiently predict the NFoVs given the video content and subtitles. Specifically, at each frame, we efficiently encode the panorama into feature map of candidate NFoVs using a Convolutional Neural Network (CNN) and the subtitles to the same hidden space using an RNN with Gated Recurrent Units (GRU). Then, we apply soft-attention on candidate NFoVs to trigger sentence decoder aiming to minimize the reconstruct loss between the generated and given sentence. Finally, we obtain the NFoV as the candidate NFoV with the maximum attention without any human supervision. To train VGM more robustly, we also generate a reverse sentence conditioning on one minus the soft-attention such that the attention focuses on candidate NFoVs less relevant to the given sentence. The negative log reconstruction loss of the reverse sentence (referred to as "irrelevant loss") is jointly minimized to encourage the reverse sentence to be different from the given sentence. To evaluate our method, we collect the first narrated 360{\deg} videos dataset and achieve state-of-the-art NFoV-grounding performance.