Subtitle-based Viewport Prediction for 360-degree Virtual Tourism Video

TL;DR

This paper introduces a novel viewport prediction model for 360-degree virtual tourism videos that incorporates subtitle-based navigation cues alongside head movement and saliency, significantly enhancing prediction accuracy.

Contribution

It is the first to utilize subtitle navigation information for viewport prediction, improving accuracy over existing methods that only consider head movement and saliency.

Findings

The proposed model outperforms baseline methods in prediction accuracy.

Inclusion of subtitle navigation cues improves viewport prediction.

Experimental results validate the effectiveness of the subtitle-based approach.

Abstract

360-degree streaming videos can provide a rich immersive experiences to the users. However, it requires an extremely high bandwidth network. One of the common solutions for saving bandwidth consumption is to stream only a portion of video covered by the user's viewport. To do that, the user's viewpoint prediction is indispensable. In existing viewport prediction methods, they mainly concentrate on the user's head movement trajectory and video saliency. None of them consider navigation information contained in the video, which can turn the attention of the user to specific regions in the video with high probability. Such information can be included in video subtitles, especially the one in 360-degree virtual tourism videos. This fact reveals the potential contribution of video subtitles to viewport prediction. Therefore, in this paper, a subtitle-based viewport prediction model for 360-degree virtual tourism videos is proposed. This model leverages the navigation information in the video subtitles in addition to head movement trajectory and video saliency, to improve the prediction accuracy. The experimental results demonstrate that the proposed model outperforms baseline methods which only use head movement trajectory and video saliency for viewport prediction.