The importance of visual modelling languages in generative software engineering

TL;DR

This paper explores the role of multimodal GPTs, like GPT-4, in software engineering, emphasizing the significance of visual modeling languages in leveraging AI for complex tasks.

Contribution

It introduces the novel investigation of multimodal GPTs in software engineering, focusing on the use of diagrams and natural language prompts for enhanced task execution.

Findings

Identifies new use cases for multimodal GPTs in software engineering.

Highlights the importance of visual modeling languages in AI-driven software tasks.

Suggests potential for improved software engineering workflows with multimodal AI.

Abstract

Multimodal GPTs represent a watershed in the interplay between Software Engineering and Generative Artificial Intelligence. GPT-4 accepts image and text inputs, rather than simply natural language. We investigate relevant use cases stemming from these enhanced capabilities of GPT-4. To the best of our knowledge, no other work has investigated similar use cases involving Software Engineering tasks carried out via multimodal GPTs prompted with a mix of diagrams and natural language.