Deep Billboards towards Lossless Real2Sim in Virtual Reality
Naruya Kondo, So Kuroki, Ryosuke Hyakuta, Yutaka Matsuo, Shixiang, Shane Gu, Yoichi Ochiai

TL;DR
Deep Billboards use neural networks to model 3D objects implicitly, enabling real-time, high-resolution, lossless rendering of diverse objects in virtual reality, significantly advancing the realism and interactivity of VR environments.
Contribution
The paper introduces Deep Billboards, a neural rendering technique that models 3D objects implicitly for lossless, real-time VR rendering, bridging the real-to-simulation gap.
Findings
Enables real-time high-resolution rendering of complex objects in VR.
Supports diverse object types including hairy and dynamic objects.
Allows quick capture and immersive experience of objects in VR.
Abstract
An aspirational goal for virtual reality (VR) is to bring in a rich diversity of real world objects losslessly. Existing VR applications often convert objects into explicit 3D models with meshes or point clouds, which allow fast interactive rendering but also severely limit its quality and the types of supported objects, fundamentally upper-bounding the "realism" of VR. Inspired by the classic "billboards" technique in gaming, we develop Deep Billboards that model 3D objects implicitly using neural networks, where only 2D image is rendered at a time based on the user's viewing direction. Our system, connecting a commercial VR headset with a server running neural rendering, allows real-time high-resolution simulation of detailed rigid objects, hairy objects, actuated dynamic objects and more in an interactive VR world, drastically narrowing the existing real-to-simulation (real2sim) gap.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
