ChatGPT as an inventor: Eliciting the strengths and weaknesses of current large language models against humans in engineering design

TL;DR

This paper evaluates ChatGPT 4.0's capabilities in engineering design by comparing its performance to graduate students in a prototyping hackathon, highlighting its strengths in idea generation and limitations in consistency and decision-making.

Contribution

It provides a novel assessment of ChatGPT's practical application in engineering design tasks and offers specific recommendations for its effective integration.

Findings

ChatGPT performed well in concept generation and prototyping.

It finished second among six teams in a design competition.

The model faced challenges with maintaining focus and clarity in communication.

Abstract

This study compares the design practices and performance of ChatGPT 4.0, a large language model (LLM), against graduate engineering students in a 48-hour prototyping hackathon, based on a dataset comprising more than 100 prototypes. The LLM participated by instructing two participants who executed its instructions and provided objective feedback, generated ideas autonomously and made all design decisions without human intervention. The LLM exhibited similar prototyping practices to human participants and finished second among six teams, successfully designing and providing building instructions for functional prototypes. The LLM's concept generation capabilities were particularly strong. However, the LLM prematurely abandoned promising concepts when facing minor difficulties, added unnecessary complexity to designs, and experienced design fixation. Communication between the LLM and participants was challenging due to vague or unclear descriptions, and the LLM had difficulty maintaining continuity and relevance in answers. Based on these findings, six recommendations for implementing an LLM like ChatGPT in the design process are proposed, including leveraging it for ideation, ensuring human oversight for key decisions, implementing iterative feedback loops, prompting it to consider alternatives, and assigning specific and manageable tasks at a subsystem level.