VistaFlow: Photorealistic Volumetric Reconstruction with Dynamic Resolution Management via Q-Learning

TL;DR

VistaFlow is a scalable 3D reconstruction method that synthesizes photorealistic views from 2D photos, using Q-learning to dynamically manage resolution and achieve high performance on low-end hardware.

Contribution

It introduces QuiQ, a novel Q-learning based controller for dynamic resolution management, and employs PlenOctree for efficient rendering without NeRFs.

Findings

Achieves 1080p rendering at over 100 fps on consumer hardware.

Runs efficiently on integrated CPU graphics, including mobile devices.

Outperforms state-of-the-art view synthesis methods.

Abstract

We introduce VistaFlow, a scalable three-dimensional imaging technique capable of reconstructing fully interactive 3D volumetric images from a set of 2D photographs. Our model synthesizes novel viewpoints through a differentiable rendering system capable of dynamic resolution management on photorealistic 3D scenes. We achieve this through the introduction of QuiQ, a novel intermediate video controller trained through Q-learning to maintain a consistently high framerate by adjusting render resolution with millisecond precision. Notably, VistaFlow runs natively on integrated CPU graphics, making it viable for mobile and entry-level devices while still delivering high-performance rendering. VistaFlow bypasses Neural Radiance Fields (NeRFs), using the PlenOctree data structure to render complex light interactions such as reflection and subsurface scattering with minimal hardware requirements. Our model is capable of outperforming state-of-the-art methods with novel view synthesis at a resolution of 1080p at over 100 frames per second on consumer hardware. By tailoring render quality to the capabilities of each device, VistaFlow has the potential to improve the efficiency and accessibility of photorealistic 3D scene rendering across a wide spectrum of hardware, from high-end workstations to inexpensive microcontrollers.