FVV Live: A real-time free-viewpoint video system with consumer electronics hardware

TL;DR

FVV Live is a low-cost, real-time free-viewpoint video system using consumer electronics, enabling immersive broadcasting and videoconferencing with high-quality virtual view synthesis and minimal latency.

Contribution

The paper introduces a novel end-to-end free-viewpoint video system that leverages off-the-shelf hardware for real-time, high-quality virtual view synthesis with flexible camera configurations.

Findings

Achieves low motion-to-photon and end-to-end delays for seamless navigation.

Subjectively rated as high quality and preferred over existing DIBR methods.

Supports arbitrary numbers of cameras and capture servers, demonstrated with nine stereo cameras.

Abstract

FVV Live is a novel end-to-end free-viewpoint video system, designed for low cost and real-time operation, based on off-the-shelf components. The system has been designed to yield high-quality free-viewpoint video using consumer-grade cameras and hardware, which enables low deployment costs and easy installation for immersive event-broadcasting or videoconferencing. The paper describes the architecture of the system, including acquisition and encoding of multiview plus depth data in several capture servers and virtual view synthesis on an edge server. All the blocks of the system have been designed to overcome the limitations imposed by hardware and network, which impact directly on the accuracy of depth data and thus on the quality of virtual view synthesis. The design of FVV Live allows for an arbitrary number of cameras and capture servers, and the results presented in this paper correspond to an implementation with nine stereo-based depth cameras. FVV Live presents low motion-to-photon and end-to-end delays, which enables seamless free-viewpoint navigation and bilateral immersive communications. Moreover, the visual quality of FVV Live has been assessed through subjective assessment with satisfactory results, and additional comparative tests show that it is preferred over state-of-the-art DIBR alternatives.