Beyond Semantic Similarity: Open Challenges for Embedding-Based Creative Process Analysis Across AI Design Tools

TL;DR

This paper discusses the limitations of current embedding-based analysis in creative AI tools, highlighting open challenges and proposing context-aware interventions to better capture creative dynamics across different domains.

Contribution

It identifies key open challenges in embedding-based creative process analysis and proposes using large language models for context-aware, session-specific analysis.

Findings

Fixed embedding similarity can misrepresent creative pivots.

Open challenges include aligning similarity with creative significance.

Proposes context-aware interventions with large language models.

Abstract

AI-based creativity support tools (CSTs) are evaluated through domain-specific metrics, limiting cross-domain comparison of creative processes. Embedding-based protocol analysis offers a potential domain-agnostic analytical layer. However, we argue that fixed embedding similarity can misrepresent creative dynamics: it may not detect creative pivots that occur within superficially similar language, treating shifts in the problem being addressed as continued elaboration. We identify three open challenges stemming from this gap: aligning similarity measures with creative significance, segmenting and representing multimodal design traces, and evaluating agentic systems where embedding-based metrics enter the generation loop and shape agent behavior. We propose context-aware interventions using large language models as a direction for making trace analysis sensitive to session-specific creative dynamics.