Beyond Productivity: Rethinking the Impact of Creativity Support Tools

TL;DR

This paper reviews evaluation metrics for Creativity Support Tools, emphasizing the need for holistic, user-centered measures including well-being and self-reflection, especially in the context of generative AI advancements.

Contribution

It identifies current evaluation practices, highlights gaps in measures, and advocates for broader, validated assessment approaches in CST research.

Findings

Current metrics focus on user interaction and output quality.

Underexplored measures include self-reflection and well-being.

Call for validated, comprehensive evaluation methods for generative AI in CSTs.

Abstract

Creativity Support Tools (CSTs) are widely used across diverse creative domains, with generative AI recently increasing the abilities of CSTs. To better understand how the success of CSTs is determined in the literature, we conducted a review of outcome measures used in CST evaluations. Drawing from (n=173) CST evaluations in the ACM Digital Library, we identified the metrics commonly employed to assess user interactions with CSTs. Our findings reveal prevailing trends in current evaluation practices, while exposing underexplored measures that could broaden the scope of future research. Based on these results, we argue for a more holistic approach to evaluating CSTs, encouraging the HCI community to consider not only user experience and the quality of the generated output, but also user-centric aspects such as self-reflection and well-being as critical dimensions of assessment. We also highlight a need for validated measures specifically suited to the evaluation of generative AI in CSTs.