Sketch2Prototype: Rapid Conceptual Design Exploration and Prototyping with Generative AI

TL;DR

Sketch2Prototype leverages generative AI to convert hand-drawn sketches into diverse 2D images and 3D prototypes, enhancing early-stage design exploration with rapid, multimodal outputs.

Contribution

The framework introduces a novel sketch-to-text-to-image-to-3D pipeline that outperforms direct sketch-to-3D methods in generating diverse, manufacturable models.

Findings

Text modality improves 3D model diversity and manufacturability.

Current image-to-3D techniques have notable limitations.

Text feedback facilitates iterative design augmentation.

Abstract

Sketch2Prototype is an AI-based framework that transforms a hand-drawn sketch into a diverse set of 2D images and 3D prototypes through sketch-to-text, text-to-image, and image-to-3D stages. This framework, shown across various sketches, rapidly generates text, image, and 3D modalities for enhanced early-stage design exploration. We show that using text as an intermediate modality outperforms direct sketch-to-3D baselines for generating diverse and manufacturable 3D models. We find limitations in current image-to-3D techniques, while noting the value of the text modality for user-feedback and iterative design augmentation.