NeARportation: A Remote Real-time Neural Rendering Framework

TL;DR

NeARportation is a neural rendering framework that enables real-time, high-fidelity photo-realistic object appearance streaming for augmented virtuality, balancing quality and performance through server-client communication.

Contribution

It introduces a novel framework combining neural rendering and server-client communication to achieve real-time, photo-realistic appearance streaming in augmented virtuality environments.

Findings

Achieved 35-40 fps rendering on stereo displays

Displayed high-fidelity appearance with realistic quality

Successfully integrated neural rendering with real-time streaming

Abstract

While the presentation of photo-realistic appearance plays a major role in immersion in an augmented virtuality environment, displaying the photo-realistic appearance of real objects remains a challenging problem. Recent developments in photogrammetry have facilitated the incorporation of real objects into virtual space. However, photo-realistic photogrammetry requires a dedicated measurement environment, and there is a trade-off between measurement cost and quality. Furthermore, even with photo-realistic appearance measurements, there is a trade-off between rendering quality and framerate. There is no framework that could resolve these trade-offs and easily provide a photo-realistic appearance in real-time. Our NeARportation framework combines server-client bidirectional communication and neural rendering to resolve these trade-offs. Neural rendering on the server receives the client's head posture and generates a novel-view image with realistic appearance reproduction, which is streamed onto the client's display. By applying our framework to a stereoscopic display, we confirmed that it could display a high-fidelity appearance on full-HD stereo videos at 35-40 frames-per-second (fps), according to the user's head motion.