Putting GPT-3's Creativity to the (Alternative Uses) Test

TL;DR

This paper evaluates GPT-3's ability to generate creative responses on the Alternative Uses Test, comparing its performance to humans and discussing implications for AI and human creativity.

Contribution

It introduces a systematic assessment of GPT-3's creativity using established human creativity metrics and automated semantic distance measures.

Findings

Humans outperform GPT-3 in creativity tasks

GPT-3 shows potential to match human creativity in the future

Automated measures correlate with expert ratings of creativity

Abstract

AI large language models have (co-)produced amazing written works from newspaper articles to novels and poetry. These works meet the standards of the standard definition of creativity: being original and useful, and sometimes even the additional element of surprise. But can a large language model designed to predict the next text fragment provide creative, out-of-the-box, responses that still solve the problem at hand? We put Open AI's generative natural language model, GPT-3, to the test. Can it provide creative solutions to one of the most commonly used tests in creativity research? We assessed GPT-3's creativity on Guilford's Alternative Uses Test and compared its performance to previously collected human responses on expert ratings of originality, usefulness and surprise of responses, flexibility of each set of ideas as well as an automated method to measure creativity based on the semantic distance between a response and the AUT object in question. Our results show that -- on the whole -- humans currently outperform GPT-3 when it comes to creative output. But, we believe it is only a matter of time before GPT-3 catches up on this particular task. We discuss what this work reveals about human and AI creativity, creativity testing and our definition of creativity.