MetaSapiens: Real-Time Neural Rendering with Efficiency-Aware Pruning and Accelerated Foveated Rendering

TL;DR

MetaSapiens is a novel point-based neural rendering system that achieves real-time, high-quality rendering on mobile devices by combining efficiency-aware pruning, foveated rendering, and specialized acceleration techniques.

Contribution

It introduces the first real-time neural rendering system for mobile devices, integrating pruning, foveated rendering, and acceleration to enhance speed and maintain visual quality.

Findings

Achieves an order of magnitude speedup over existing models.

Maintains subjective visual quality confirmed by user study.

Enables real-time neural rendering on mobile devices.

Abstract

Point-Based Neural Rendering (PBNR) is emerging as a promising class of rendering techniques, which are permeating all aspects of society, driven by a growing demand for real-time, photorealistic rendering in AR/VR and digital twins. Achieving real-time PBNR on mobile devices is challenging. This paper proposes MetaSapiens, a PBNR system that for the first time delivers real-time neural rendering on mobile devices while maintaining human visual quality. MetaSapiens combines three techniques. First, we present an efficiency-aware pruning technique to optimize rendering speed. Second, we introduce a Foveated Rendering (FR) method for PBNR, leveraging humans' low visual acuity in peripheral regions to relax rendering quality and improve rendering speed. Finally, we propose an accelerator design for FR, addressing the load imbalance issue in (FR-based) PBNR. Our evaluation shows that our system achieves an order of magnitude speedup over existing PBNR models without sacrificing subjective visual quality, as confirmed by a user study. The code and demo are available at: https://horizon-lab.org/metasapiens/.