TL;DR
This paper presents MatryODShka, a real-time method converting stereo 360° imagery into a layered multi-sphere format for accurate 6DoF VR rendering, improving viewer comfort by handling disocclusions and motion parallax.
Contribution
It introduces a novel multi-sphere image representation that learns depth and disocclusions simultaneously for 6DoF VR video synthesis.
Findings
Enables real-time 6DoF rendering from stereo 360° images.
Improves VR comfort by accurately modeling motion parallax.
Supports dynamic scenes with correct disocclusion handling.
Abstract
We introduce a method to convert stereo 360{\deg} (omnidirectional stereo) imagery into a layered, multi-sphere image representation for six degree-of-freedom (6DoF) rendering. Stereo 360{\deg} imagery can be captured from multi-camera systems for virtual reality (VR), but lacks motion parallax and correct-in-all-directions disparity cues. Together, these can quickly lead to VR sickness when viewing content. One solution is to try and generate a format suitable for 6DoF rendering, such as by estimating depth. However, this raises questions as to how to handle disoccluded regions in dynamic scenes. Our approach is to simultaneously learn depth and disocclusions via a multi-sphere image representation, which can be rendered with correct 6DoF disparity and motion parallax in VR. This significantly improves comfort for the viewer, and can be inferred and rendered in real time on modern GPU…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
