Evaluation Framework for AI Creativity: A Case Study Based on Story Generation

TL;DR

This paper introduces a structured evaluation framework for AI story generation that captures subjective creativity aspects through multiple components, validated by crowdsourced human judgments, revealing hierarchical and stage-dependent evaluation dynamics.

Contribution

The paper presents a novel multi-component evaluation framework for AI creativity, incorporating controlled experiments and human studies to better assess subjective creative qualities.

Findings

Creativity is evaluated hierarchically rather than cumulatively.

Reflective evaluation significantly alters ratings and agreement.

The framework reveals creativity dimensions obscured by reference-based metrics.

Abstract

Evaluating creative text generation remains a challenge because existing reference-based metrics fail to capture the subjective nature of creativity. We propose a structured evaluation framework for AI story generation comprising four components (Novelty, Value, Adherence, and Resonance) and eleven sub-components. Using controlled story generation via ``Spike Prompting'' and a crowdsourced study of 115 readers, we examine how different creative components shape both immediate and reflective human creativity judgments. Our findings show that creativity is evaluated hierarchically rather than cumulatively, with different dimensions becoming salient at different stages of judgment, and that reflective evaluation substantially alters both ratings and inter-rater agreement. Together, these results support the effectiveness of our framework in revealing dimensions of creativity that are obscured by reference-based evaluation.