Benchmarking Language Model Creativity: A Case Study on Code Generation

TL;DR

This paper introduces a framework to quantify creativity in large language models through a new prompting technique and a metric, revealing that even advanced models like GPT-4 lack human-like creativity in code generation.

Contribution

The paper presents DENIAL PROMPTING and NEOGAUGE, novel methods for enhancing and measuring creativity in LLMs, applied to coding tasks with a new dataset for future benchmarking.

Findings

GPT-4 still lags behind human creativity in coding tasks.

Advanced reasoning strategies do not significantly boost LLM creativity.

The NEOCODER dataset enables future reproducibility and benchmarking.

Abstract

As LLMs become increasingly prevalent, it is interesting to consider how ``creative'' these models can be. From cognitive science, creativity consists of at least two key characteristics: \emph{convergent} thinking (purposefulness to achieve a given goal) and \emph{divergent} thinking (adaptability to explore new environments or constraints) \citep{runco2003critical}. In this work, we introduce a framework for quantifying LLM creativity that incorporates the two design ingredients: (1) We introduce DENIAL PROMPTING which pushes LLMs to develop more creative solutions to a given problem by incrementally imposing new constraints on the previous solution, compelling LLMs to adopt new strategies. (2) We define NEOGAUGE, a metric that quantifies both convergent and divergent thinking in the generated creative responses by LLMs. We test the proposed framework on Codeforces problems, which serve as both a natural dataset for coding tasks and a collection of prior human solutions. We quantify NEOGAUGE for various proprietary and open-source models and find that even the most creative model, GPT-4, still falls short of demonstrating human-like creativity. We also experiment with advanced reasoning strategies (MCTS, self-correction, etc.) and observe no significant improvement in creativity. As a by-product of our analysis, we release NEOCODER dataset for reproducing our results on future models.