Prop-Chromeleon: Adaptive Haptic Props in Mixed Reality through Generative Artificial Intelligence

TL;DR

Prop-Chromeleon is an AI-driven MR system that creates adaptive haptic props from everyday objects, enhancing realism and immersion through dynamic virtual content generation based on user prompts.

Contribution

It introduces a novel AI pipeline that generates and anchors virtual assets to physical objects in MR, enabling realistic and adaptable haptic interactions.

Findings

Prop-Chromeleon improves perceived realism and immersion in MR.

Shape-aware generation supports believable haptic interaction.

User study shows increased enjoyment with Prop-Chromeleon.

Abstract

Mixed Reality (MR) aims to blend digital and physical worlds, but the absence of haptic feedback often breaks visual-tactile consistency. We introduce Prop-Chromeleon, a MR system based on generative artificial intelligence (AI) that dynamically transforms everyday objects into adaptive passive haptic props through user-provided text prompts. Our AI pipeline performs generation and anchoring of virtual assets that align with the shape of physical props, allowing us to study how virtual content generation behaves under geometric and prompt-based constraints. We evaluate Prop-Chromeleon's effectiveness through a generation study using varied object shapes and user prompts, combining quantitative shape similarity metrics with qualitative prompt fidelity analysis. Our user study further showcases Prop-Chromeleon's improvements in perceived realism, immersion, and enjoyment compared to static baselines. These results show that shape-aware generation can support both believable haptic interaction and creative engagement in MR.