Towards spatial computing: recent advances in multimodal natural interaction for XR headsets

TL;DR

This paper reviews recent advances in multimodal natural interaction techniques for XR headsets, emphasizing AI-driven methods and analyzing research from 2022 to 2024 to guide future spatial computing developments.

Contribution

It provides a structured classification and analysis of recent multimodal natural interaction research for XR, highlighting new paradigms and future research directions.

Findings

AI and LLMs have enabled new interaction paradigms.

Multimodal interaction techniques are diverse across applications.

Identified challenges and future directions for natural XR interactions.

Abstract

With the widespread adoption of Extended Reality (XR) headsets, spatial computing technologies are gaining increasing attention. Spatial computing enables interaction with virtual elements through natural input methods such as eye tracking, hand gestures, and voice commands, thus placing natural human-computer interaction at its core. While previous surveys have reviewed conventional XR interaction techniques, recent advancements in natural interaction, particularly driven by artificial intelligence (AI) and large language models (LLMs), have introduced new paradigms and technologies. In this paper, we review research on multimodal natural interaction for wearable XR, focusing on papers published between 2022 and 2024 in six top venues: ACM CHI, UIST, IMWUT (Ubicomp), IEEE VR, ISMAR, and TVCG. We classify and analyze these studies based on application scenarios, operation types, and interaction modalities. This analysis provides a structured framework for understanding how researchers are designing advanced natural interaction techniques in XR. Based on these findings, we discuss the challenges in natural interaction techniques and suggest potential directions for future research. This review provides valuable insights for researchers aiming to design natural and efficient interaction systems for XR, ultimately contributing to the advancement of spatial computing.