A Taxonomy of Human--MLLM Interaction in Early-Stage Sketch-Based Design Ideation
Weiyan Shi, Kenny Tsu Wei Choo

TL;DR
This study explores how designers interact with multimodal large language models during early-stage sketch-based design, revealing dynamic role shifts and collaboration modes that inform future AI-supported design tools.
Contribution
It introduces a taxonomy of interaction modes between humans and AI in sketch-based design, based on empirical analysis of user behavior and collaboration patterns.
Findings
Designers frequently shift between human-led and AI-led roles.
Interaction modes are often interwoven rather than static.
Empirical insights into dynamic human-AI collaboration in design.
Abstract
As multimodal large language models (MLLMs) are increasingly integrated into early-stage design tools, it is important to understand how designers collaborate with AI during ideation. In a user study with 12 participants, we analysed sketch-based design interactions with an MLLM-powered system using automatically recorded interaction logs and post-task interviews. Based on how creative responsibility was allocated between humans and the AI, we predefined four interaction modes: Human-Only, Human-Lead, AI-Lead, and Co-Evolution, and analysed how these modes manifested during sketch-based design ideation. Our results show that designers rarely rely on a single mode; instead, human-led and AI-led roles are frequently interwoven and shift across ideation instances. These findings provide an empirical basis for future work to investigate why designers shift roles with AI and how interactive…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDesign Education and Practice · AI in Service Interactions · Innovative Human-Technology Interaction
