An Exploratory Study of ML Sketches and Visual Code Assistants
Lu\'is F. Gomes, Vincent J. Hellendoorn, Jonathan Aldrich, Rui Abreu

TL;DR
This study investigates how visual sketches in IDEs can be used to generate code for ML workflows, demonstrating that even brief sketches can produce useful code outlines and highlighting potential applications in education and prototyping.
Contribution
The paper presents a prototype Visual Code Assistant that converts developer sketches into Python code using an LLM, exploring its effectiveness and developer perceptions.
Findings
Sketches with diagrams, lists, and numbered points are common in ML workflows.
Brief sketches can generate useful code outlines with high quality.
Positive developer feedback on the tool's usefulness and potential applications.
Abstract
This paper explores the integration of Visual Code Assistants in Integrated Development Environments (IDEs). In Software Engineering, whiteboard sketching is often the initial step before coding, serving as a crucial collaboration tool for developers. Previous studies have investigated patterns in SE sketches and how they are used in practice, yet methods for directly using these sketches for code generation remain limited. The emergence of visually-equipped large language models presents an opportunity to bridge this gap, which is the focus of our research. In this paper, we built a first prototype of a Visual Code Assistant to get user feedback regarding in-IDE sketch-to-code tools. We conduct an experiment with 19 data scientists, most of whom regularly sketch as part of their job. We investigate developers' mental models by analyzing patterns commonly observed in their sketches when…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsOnline Learning and Analytics · Teaching and Learning Programming · Intelligent Tutoring Systems and Adaptive Learning
MethodsFocus
