Predictive Prototyping: Evaluating Design Concepts with ChatGPT

TL;DR

This paper explores using GPT-4 with retrieval-augmented generation to predict design prototype outcomes, enabling faster, cost-effective evaluation of concepts before physical prototyping.

Contribution

Introduces a GPT-based RAG method for predicting design metrics, demonstrating improved accuracy over human estimates and practical prototype development.

Findings

GPT-RAG outperforms humans in cost and performance prediction.

GPT-RAG-informed prototypes outperform baseline and optimized designs.

Repeated querying enhances prediction accuracy.

Abstract

The design-build-test cycle is essential for innovation, but physical prototyping is often slow and expensive. Although physics-based simulation and strategic prototyping can reduce cost, meaningful evaluation is frequently constrained until an integrated prototype is built. This paper investigates whether a generative pretrained transformer (GPT) can predict information typically obtained through prototyping, including cost, performance, and perceived usability. We introduce a retrieval-augmented generation (RAG) method to emulate design feedback using OpenAI GPT-4o, grounded in prototyping data scraped from Instructables.com to increase access to relevant precedent. Two studies are reported. First, a controlled experiment compares GPT-RAG and human designers, who receive design sketches and predict cost, performance, and usability; predictions are evaluated against ground-truth results from physical prototypes. Second, we report an applied demonstration in which a physical prototype is produced from GPT-RAG recommendations and compared with a commercial baseline and a topology-optimized design. Results show that GPT-RAG provides more accurate cost and performance estimates than individual or crowd human estimates, while yielding comparable usability insights; the GPT-RAG-informed prototype also outperforms both comparison prototypes. Repeated querying with response averaging significantly improves accuracy, suggesting that LLMs can emulate crowd aggregation effects consistent with the law of large numbers.