sMoRe: Enhancing Object Manipulation and Organization in Mixed Reality Spaces with LLMs and Generative AI

TL;DR

sMoRe is a novel mixed reality system that leverages large language models and generative AI to facilitate intuitive creation, placement, and management of virtual objects, significantly improving user interaction and spatial organization.

Contribution

This work introduces sMoRe, integrating LLMs and generative AI for real-time virtual object creation and spatial management in MR environments, a novel combination not previously explored.

Findings

Enhanced user understanding of MR spaces

Improved interaction efficiency with virtual objects

Positive user feedback on system usability

Abstract

In mixed reality (MR) environments, understanding space and creating virtual objects is crucial to providing an intuitive and rich user experience. This paper introduces sMoRe (Spatial Mapping and Object Rendering Environment), an MR application that combines Generative AI (GenAI) with large language models (LLMs) to assist users in creating, placing, and managing virtual objects within physical spaces. sMoRe allows users to use voice or typed text commands to create and place virtual objects using GenAI while specifying spatial constraints. The system leverages LLMs to interpret users' commands, analyze the current scene, and identify optimal locations. Additionally, sMoRe integrates text-to-3D generative AI to dynamically create 3D objects based on users' descriptions. Our user study demonstrates the effectiveness of sMoRe in enhancing user comprehension, interaction, and organization of the MR environment.