Rethinking Creativity Evaluation: A Critical Analysis of Existing Creativity Evaluations

TL;DR

This paper critically analyzes four existing creativity evaluation metrics across diverse domains, revealing their limitations and inconsistencies, and emphasizes the need for more reliable, human-aligned assessment methods.

Contribution

It provides a comprehensive comparison of current creativity metrics across multiple domains, highlighting their shortcomings and proposing the necessity for improved evaluation frameworks.

Findings

Metrics often fail to generalize across domains.

Different metrics frequently produce conflicting judgments.

Perplexity reflects fluency, not creativity.

Abstract

We examine, analyze, and compare four representative creativity measures--perplexity, LLM-as-a-Judge, the Creativity Index (CI; measuring n-gram overlap with web corpora), and syntactic templates (detecting repetition of common part-of-speech patterns)--across the diverse creative domains, such as creative writing, unconventional problem-solving, and research ideation. For each domain, we compile datasets with human-aligned creative and uncreative examples and evaluate each metric's ability to discriminate between the two sets. Our analyses reveal limited consistency both across domains and metrics, as metrics that distinguish creativity in one domain fail in others (e.g., CI correctly distinguishes in creative writing but fails in problem-solving), and different metrics often disagree on the same data points (e.g., CI suggests one set to be more creative, while perplexity indicates the other set to be more creative.) We highlight key limitations, such as perplexity reflecting fluency rather than novelty; LLM-as-a-Judge producing inconsistent judgments under minor prompt variations and exhibiting bias towards particular labels; CI primarily measuring lexical diversity, with high sensitivity to implementation choices; and syntactic templates being ineffective in settings dominated by formulaic language. Our findings underscore the need for more robust, generalizable evaluation frameworks that better align with human judgments of creativity.