TaskLens: Generating Task-Conditioned Scaffolded Interfaces for Learning Professional Creative Software

TL;DR

TaskLens leverages large language models to automatically generate task-specific, scaffolded user interfaces for professional creative software, enhancing learning and productivity for both beginners and experts.

Contribution

Introduces a novel LLM-based method for automatically creating task-conditioned scaffolded interfaces from natural language descriptions.

Findings

Beginners experienced reduced perceived task load.

Improved task performance with embedded workflow guidance.

Enhanced domain concept learning during tasks.

Abstract

Professional creative software has steep learning curves for novices due to complex interfaces, limited guidance, and unfamiliar terminology. To support educators and tool creators in addressing learner challenges, we introduce TaskLens, an LLM-based method that automatically generates task-conditioned scaffolded UIs from natural language task descriptions. Our method uses LLMs to identify workflow stages and domain concepts, select task-relevant tools, generate implementation code, and execute the code to produce scaffolded interfaces. The interfaces surface relevant tools, organize them by workflow stage, link them to domain concepts, and progressively disclose advanced features. We evaluate TaskLens by deploying two LLM-generated scaffolded interfaces in Blender, a professional 3D modeling software. A user study with beginners (n=32) showed that our scaffolded interfaces significantly reduced perceived task load, improved task performance through embedded workflow guidance, and increased domain concept learning in Blender during task execution. A second study with experts (n=8) showed improved task efficiency and potential to create personalized UIs for productivity and creativity.