3Description: An Intuitive Human-AI Collaborative 3D Modeling Approach

TL;DR

3Description is a web-based human-AI collaborative system that enables non-professionals to create and modify 3D models using verbal and gesture inputs, enhancing accessibility and user engagement in 3D design.

Contribution

It introduces an intuitive, multimodal human-AI collaborative approach for 3D modeling that combines NLP and computer vision technologies for broader accessibility.

Findings

Successful user testing demonstrating ease of use

Enhanced engagement in 3D co-creation

Integration of NLP and computer vision in a web platform

Abstract

This paper presents 3Description, an experimental human-AI collaborative approach for intuitive 3D modeling. 3Description aims to address accessibility and usability challenges in traditional 3D modeling by enabling non-professional individuals to co-create 3D models using verbal and gesture descriptions. Through a combination of qualitative research, product analysis, and user testing, 3Description integrates AI technologies such as Natural Language Processing and Computer Vision, powered by OpenAI and MediaPipe. Recognizing the web has wide cross-platform capabilities, 3Description is web-based, allowing users to describe the desired model and subsequently adjust its components using verbal and gestural inputs. In the era of AI and emerging media, 3Description not only contributes to a more inclusive and user-friendly design process, empowering more people to participate in the construction of the future 3D world, but also strives to increase human engagement in co-creation with AI, thereby avoiding undue surrender to technology and preserving human creativity.