Drawing Your Programs: Exploring the Applications of Visual-Prompting with GenAI for Teaching and Assessment

TL;DR

This paper explores the use of visual prompts, like diagrams, in human-AI collaborative programming, showing that multimodal prompts can enhance teaching and assessment in computing education.

Contribution

It introduces the concept of visual and spatial prompts for GenAI in programming, demonstrating their effectiveness in code generation and potential for educational assessment.

Findings

GPT-4.1 successfully generates code from student-created diagrams.

Visual prompts outperform text prompts in certain programming tasks.

Multimodal prompting offers new avenues for teaching and evaluating programming skills.

Abstract

When designing a program, both novice programmers and seasoned developers alike often sketch out -- or, perhaps more famously, whiteboard -- their ideas. Yet despite the introduction of natively multimodal Generative AI models, work on Human-GenAI collaborative coding has remained overwhelmingly focused on textual prompts -- largely ignoring the visual and spatial representations that programmers naturally use to reason about and communicate their designs. In this proposal and position paper, we argue and provide tentative evidence that this text-centric focus overlooks other forms of prompting GenAI models, such as problem decomposition diagrams functioning as prompts for code generation in their own right enabling new types of programming activities and assessments. To support this position, we present findings from a large introductory Python programming course, where students constructed decomposition diagrams that were used to prompt GPT-4.1 for code generation. We demonstrate that current models are very successful in their ability to generate code from student-constructed diagrams. We conclude by exploring the implications of embracing multimodal prompting for computing education, particularly in the context of assessment.