CreativityPrism: A Holistic Evaluation Framework for Large Language Model Creativity

TL;DR

CreativityPrism is a comprehensive, scalable framework for evaluating large language model creativity across multiple domains and dimensions, addressing limitations of existing fragmented and human-dependent methods.

Contribution

It introduces a holistic evaluation framework that consolidates diverse creative tasks and employs reliable automatic judges validated against human annotations.

Findings

Proprietary LLMs outperform open-source models in creative writing and reasoning.

No significant advantage of proprietary models in divergent thinking.

High performance in one creative dimension rarely correlates with others.

Abstract

Creativity is often seen as a hallmark of human intelligence. While large language models (LLMs) are increasingly perceived as generating creative text, there is still no holistic and scalable framework to evaluate their creativity across diverse scenarios. Existing methods of LLM creativity evaluation either heavily rely on humans, limiting speed and scalability, or are fragmented across different domains and different definitions of creativity. To address this gap, we propose CREATIVITYPRISM, an evaluation analysis framework that consolidates eight tasks from three domains, divergent thinking, creative writing, and logical reasoning, into a taxonomy of creativity that emphasizes three dimensions: quality, novelty, and diversity of LLM generations. The framework is designed to be scalable with reliable automatic evaluation judges that have been validated against human annotations. We evaluate 17 state-of-the-art (SoTA) proprietary and open-sourced LLMs on CREATIVITYPRISM and find that while proprietary LLMs dominate creative writing and logical reasoning tasks by a 15% lead over open-sourced ones, they offer no significant advantage in divergent thinking, a domain much less explored in existing post-training regimes. Our analysis also shows that high performance in one creative dimension or domain rarely generalizes to others; specifically, novelty metrics often show weak or negative correlations with other metrics. This fragmentation confirms that a holistic, multi-dimensional framework like CREATIVITYPRISM is essential for meaningful assessment of LLM creativity.