iButter: Neural Interactive Bullet Time Generator for Human Free-viewpoint Rendering

TL;DR

This paper introduces iButter, a neural system enabling real-time, interactive, and photo-realistic human free-viewpoint video generation with bullet-time effects, suitable for immersive VR/AR applications.

Contribution

iButter is the first neural framework to provide real-time, interactive bullet-time design and rendering for human videos without per-scene training, using a hybrid training set and trajectory-aware refinement.

Findings

Achieves real-time preview and design of bullet-time effects.

Produces photo-realistic human free-viewpoint videos.

Supports flexible, interactive visual effect creation.

Abstract

Generating ``bullet-time'' effects of human free-viewpoint videos is critical for immersive visual effects and VR/AR experience. Recent neural advances still lack the controllable and interactive bullet-time design ability for human free-viewpoint rendering, especially under the real-time, dynamic and general setting for our trajectory-aware task. To fill this gap, in this paper we propose a neural interactive bullet-time generator (iButter) for photo-realistic human free-viewpoint rendering from dense RGB streams, which enables flexible and interactive design for human bullet-time visual effects. Our iButter approach consists of a real-time preview and design stage as well as a trajectory-aware refinement stage. During preview, we propose an interactive bullet-time design approach by extending the NeRF rendering to a real-time and dynamic setting and getting rid of the tedious per-scene training. To this end, our bullet-time design stage utilizes a hybrid training set, light-weight network design and an efficient silhouette-based sampling strategy. During refinement, we introduce an efficient trajectory-aware scheme within 20 minutes, which jointly encodes the spatial, temporal consistency and semantic cues along the designed trajectory, achieving photo-realistic bullet-time viewing experience of human activities. Extensive experiments demonstrate the effectiveness of our approach for convenient interactive bullet-time design and photo-realistic human free-viewpoint video generation.