Enhancing Perception and Immersion in Pre-Captured Environments through Learning-Based Eye Height Adaptation

TL;DR

This paper investigates how adapting eye height in pre-captured environments affects perception and immersion, proposing a learning-based view synthesis method that improves visual realism and user experience.

Contribution

It introduces a novel multitask learning approach for synthesizing views at different eye heights in omnidirectional images, enhancing perception and immersion in virtual environments.

Findings

Eye height significantly influences distance perception and immersion.

The proposed method outperforms state-of-the-art view synthesis techniques.

User studies confirm improved perception and immersion with the new approach.

Abstract

Pre-captured immersive environments using omnidirectional cameras provide a wide range of virtual reality applications. Previous research has shown that manipulating the eye height in egocentric virtual environments can significantly affect distance perception and immersion. However, the influence of eye height in pre-captured real environments has received less attention due to the difficulty of altering the perspective after finishing the capture process. To explore this influence, we first propose a pilot study that captures real environments with multiple eye heights and asks participants to judge the egocentric distances and immersion. If a significant influence is confirmed, an effective image-based approach to adapt pre-captured real-world environments to the user's eye height would be desirable. Motivated by the study, we propose a learning-based approach for synthesizing novel views for omnidirectional images with altered eye heights. This approach employs a multitask architecture that learns depth and semantic segmentation in two formats, and generates high-quality depth and semantic segmentation to facilitate the inpainting stage. With the improved omnidirectional-aware layered depth image, our approach synthesizes natural and realistic visuals for eye height adaptation. Quantitative and qualitative evaluation shows favorable results against state-of-the-art methods, and an extensive user study verifies improved perception and immersion for pre-captured real-world environments.