A Multi-user Oriented Live Free-viewpoint Video Streaming System Based On View Interpolation

TL;DR

This paper presents a real-time multi-user free-viewpoint video streaming system that uses CNN-based view interpolation to synthesize virtual views efficiently, enabling immersive experiences with low latency.

Contribution

Introduces a CNN-based view interpolation algorithm and an end-to-end multi-user streaming system that reduces client load and supports real-time immersive free-viewpoint video.

Findings

System achieves real-time performance with multiple users

Provides high visual quality and low latency

Efficiently utilizes a single edge server for multi-user streaming

Abstract

As an important application form of immersive multimedia services, free-viewpoint video(FVV) enables users with great immersive experience by strong interaction. However, the computational complexity of virtual view synthesis algorithms poses a significant challenge to the real-time performance of an FVV system. Furthermore, the individuality of user interaction makes it difficult to serve multiple users simultaneously for a system with conventional architecture. In this paper, we novelly introduce a CNN-based view interpolation algorithm to synthesis dense virtual views in real time. Based on this, we also build an end-to-end live free-viewpoint system with a multi-user oriented streaming strategy. Our system can utilize a single edge server to serve multiple users at the same time without having to bring a large view synthesis load on the client side. We analyze the whole system and show that our approaches give the user a pleasant immersive experience, in terms of both visual quality and latency.