Mixing Modalities of 3D Sketching and Speech for Interactive Model Retrieval in Virtual Reality

TL;DR

This paper presents a multimodal interface combining 3D sketching and speech for interactive 3D model retrieval in virtual reality, demonstrating improved search strategies through user studies.

Contribution

It introduces a new multimodal retrieval system that integrates sketch and speech in VR, and evaluates its effectiveness with user experiments.

Findings

Hybrid search strategies outperform single modality approaches.

Combining sketch and speech enhances retrieval accuracy and user experience.

User studies confirm the advantages of multimodal interaction in VR retrieval.

Abstract

Sketch and speech are intuitive interaction methods that convey complementary information and have been independently used for 3D model retrieval in virtual environments. While sketch has been shown to be an effective retrieval method, not all collections are easily navigable using this modality alone. We design a new challenging database for sketch comprised of 3D chairs where each of the components (arms, legs, seat, back) are independently colored. To overcome this, we implement a multimodal interface for querying 3D model databases within a virtual environment. We base the sketch on the state-of-the-art for 3D Sketch Retrieval, and use a Wizard-of-Oz style experiment to process the voice input. In this way, we avoid the complexities of natural language processing which frequently requires fine-tuning to be robust. We conduct two user studies and show that hybrid search strategies emerge from the combination of interactions, fostering the advantages provided by both modalities.