From Image to UML: First Results of Image Based UML Diagram Generation Using LLMs

TL;DR

This paper investigates using large language models to automatically convert hand-drawn UML diagrams into formal models, aiming to streamline the transition from informal sketches to machine-readable representations in software engineering.

Contribution

It presents an initial exploration of LLMs for image-based UML diagram generation, demonstrating potential and identifying current limitations for future improvements.

Findings

LLMs can generate UML models from images with promising accuracy

Current limitations require human oversight to ensure correctness

Approach can be integrated into model-driven engineering pipelines

Abstract

In software engineering processes, systems are first specified using a modeling language such as UML. These initial designs are often collaboratively created, many times in meetings where different domain experts use whiteboards, paper or other types of quick supports to create drawings and blueprints that then will need to be formalized. These proper, machine-readable, models are key to ensure models can be part of automated processes (e.g. input of a low-code generation pipeline, a model-based testing system, ...). But going from hand-drawn diagrams to actual models is a time-consuming process that sometimes ends up with such drawings just added as informal images to the software documentation, reducing their value a lot. To avoid this tedious task, we explore the usage of Large Language Models (LLM) to generate the formal representation of (UML) models from a given drawing. More specifically, we have evaluated the capabilities of different LLMs to convert images of UML class diagrams into the actual models represented in the images. While the results are good enough to use such an approach as part of a model-driven engineering pipeline we also highlight some of their current limitations and the need to keep the human in the loop to overcome those limitations.