Beyond Divergent Creativity: A Human-Based Evaluation of Creativity in Large Language Models

TL;DR

This paper critiques existing LLM creativity assessments, introduces a new human-grounded evaluation method called CDAT that better captures the balance of novelty and appropriateness, and analyzes how model training influences creative output.

Contribution

It proposes the Conditional Divergent Association Task (CDAT), a novel evaluation method grounded in human creativity theory, to better assess LLMs' creative capabilities.

Findings

Existing DAT scores are misleading, often lower than non-creative baselines.

Smaller models tend to produce more creative outputs under CDAT.

Advanced models prioritize appropriateness over novelty, shifting along the creativity frontier.

Abstract

Large language models (LLMs) are increasingly used in verbal creative tasks. However, previous assessments of the creative capabilities of LLMs remain weakly grounded in human creativity theory and are thus hard to interpret. The widely used Divergent Association Task (DAT) focuses on novelty, ignoring appropriateness, a core component of creativity. We evaluate a range of state-of-the-art LLMs on DAT and show that their scores on the task are lower than those of two baselines that do not possess any creative abilities, undermining its validity for model evaluation. Grounded in human creativity theory, which defines creativity as the combination of novelty and appropriateness, we introduce Conditional Divergent Association Task (CDAT). CDAT evaluates novelty conditional on contextual appropriateness, separating noise from creativity better than DAT, while remaining simple and objective. Under CDAT, smaller model families often show the most creativity, whereas advanced families favor appropriateness at lower novelty. We hypothesize that training and alignment likely shift models along this frontier, making outputs more appropriate but less creative. We release the dataset and code.