LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video

TL;DR

LiveStre4m is a real-time feed-forward model for novel view synthesis from unposed multi-view video, enabling stable, temporally consistent streaming without camera calibration, suitable for practical live applications.

Contribution

It introduces a novel feed-forward approach with a multi-view transformer, diffusion interpolation, and pose prediction, enabling real-time, uncalibrated, unposed multi-view video streaming.

Findings

Achieves 0.07s per-frame reconstruction at 1024x768 resolution.

Operates with as few as two synchronized unposed input streams.

Outperforms optimization-based methods in runtime by orders of magnitude.

Abstract

Live-streaming Novel View Synthesis (NVS) from unposed multi-view video remains an open challenge in a wide range of applications. Existing methods for dynamic scene representation typically require ground-truth camera parameters and involve lengthy optimizations ($\approx 2.67$s), which makes them unsuitable for live streaming scenarios. To address this issue, we propose a novel viewpoint video live-streaming method (LiveStre4m), a feed-forward model for real-time NVS from unposed sparse multi-view inputs. LiveStre4m introduces a multi-view vision transformer for keyframe 3D scene reconstruction coupled with a diffusion-transformer interpolation module that ensures temporal consistency and stable streaming. In addition, a Camera Pose Predictor module is proposed to efficiently estimate both poses and intrinsics directly from RGB images, removing the reliance on known camera calibration information. Our approach enables temporally consistent novel-view video streaming in real-time using as few as two synchronized unposed input streams. LiveStre4m attains an average reconstruction time of $ 0.07$s per-frame at $ 1024 \times 768$ resolution, outperforming the optimization-based dynamic scene representation methods by orders of magnitude in runtime. These results demonstrate that LiveStre4m makes real-time NVS streaming feasible in practical settings, marking a substantial step toward deployable live novel-view synthesis systems. Code available at: https://github.com/pedro-quesado/LiveStre4m