Generative AI Framework for 3D Object Generation in Augmented Reality

TL;DR

This paper introduces a comprehensive framework that combines generative AI and augmented reality to enable real-time, user-friendly creation of 3D objects from various inputs, broadening accessibility and application scope.

Contribution

It presents a novel integrated system utilizing advanced AI models and multimodal inputs for real-time 3D object generation in AR, making the technology more accessible and versatile.

Findings

Effective conversion of images and speech into 3D models

Enhanced accuracy in object detection and model generation

Successful real-time integration of 3D objects into AR environments

Abstract

This thesis presents a framework that integrates state-of-the-art generative AI models for real-time creation of three-dimensional (3D) objects in augmented reality (AR) environments. The primary goal is to convert diverse inputs, such as images and speech, into accurate 3D models, enhancing user interaction and immersion. Key components include advanced object detection algorithms, user-friendly interaction techniques, and robust AI models like Shap-E for 3D generation. Leveraging Vision Language Models (VLMs) and Large Language Models (LLMs), the system captures spatial details from images and processes textual information to generate comprehensive 3D objects, seamlessly integrating virtual objects into real-world environments. The framework demonstrates applications across industries such as gaming, education, retail, and interior design. It allows players to create personalized in-game assets, customers to see products in their environments before purchase, and designers to convert real-world objects into 3D models for real-time visualization. A significant contribution is democratizing 3D model creation, making advanced AI tools accessible to a broader audience, fostering creativity and innovation. The framework addresses challenges like handling multilingual inputs, diverse visual data, and complex environments, improving object detection and model generation accuracy, as well as loading 3D models in AR space in real-time. In conclusion, this thesis integrates generative AI and AR for efficient 3D model generation, enhancing accessibility and paving the way for innovative applications and improved user interactions in AR environments.