A Taxonomy of Human--MLLM Interaction in Early-Stage Sketch-Based Design Ideation

TL;DR

This study explores how designers interact with multimodal large language models during early-stage sketch-based design, revealing dynamic role shifts and collaboration modes that inform future AI-supported design tools.

Contribution

It introduces a taxonomy of interaction modes between humans and AI in sketch-based design, based on empirical analysis of user behavior and collaboration patterns.

Findings

Designers frequently shift between human-led and AI-led roles.

Interaction modes are often interwoven rather than static.

Empirical insights into dynamic human-AI collaboration in design.

Abstract

As multimodal large language models (MLLMs) are increasingly integrated into early-stage design tools, it is important to understand how designers collaborate with AI during ideation. In a user study with 12 participants, we analysed sketch-based design interactions with an MLLM-powered system using automatically recorded interaction logs and post-task interviews. Based on how creative responsibility was allocated between humans and the AI, we predefined four interaction modes: Human-Only, Human-Lead, AI-Lead, and Co-Evolution, and analysed how these modes manifested during sketch-based design ideation. Our results show that designers rarely rely on a single mode; instead, human-led and AI-led roles are frequently interwoven and shift across ideation instances. These findings provide an empirical basis for future work to investigate why designers shift roles with AI and how interactive systems can better support such dynamic collaboration.