Beyond Being Real: A Sensorimotor Control Perspective on Interactions in Virtual Reality

TL;DR

This paper surveys and categorizes VR interactions, especially beyond-real techniques that remap or alter user perception, discussing their transformations, challenges, and future research directions from a sensorimotor control perspective.

Contribution

It provides a comprehensive categorization of VR interactions, focusing on beyond-real techniques and their implications for sensorimotor control and future VR design.

Findings

Beyond-real techniques enable novel VR interactions.

Open challenges include plausibility and individual adaptation.

Future research should address control and long-term use.

Abstract

We can create Virtual Reality (VR) interactions that have no equivalent in the real world by remapping spacetime or altering users' body representation, such as stretching the user's virtual arm for manipulation of distant objects or scaling up the user's avatar to enable rapid locomotion. Prior research has leveraged such approaches, what we call beyond-real techniques, to make interactions in VR more practical, efficient, ergonomic, and accessible. We present a survey categorizing prior movement-based VR interaction literature as reality-based, illusory, or beyond-real interactions. We survey relevant conferences (CHI, IEEE VR, VRST, UIST, and DIS) while focusing on selection, manipulation, locomotion, and navigation in VR. For beyond-real interactions, we describe the transformations that have been used by prior works to create novel remappings. We discuss open research questions through the lens of the human sensorimotor control system and highlight challenges that need to be addressed for effective utilization of beyond-real interactions in future VR applications, including plausibility, control, long-term adaptation, and individual differences.