Assessing UML Diagrams by GPT: Implications for Education

TL;DR

This study explores the potential of GPT to evaluate UML diagrams in software engineering education, proposing evaluation criteria and testing GPT's grading performance against human experts.

Contribution

The paper introduces a set of evaluation criteria for UML diagrams and assesses GPT's effectiveness in grading them, highlighting its current capabilities and limitations.

Findings

GPT can evaluate UML diagrams but cannot fully replace human experts.

There are five key discrepancies between GPT and human assessments.

GPT shows strengths and weaknesses depending on diagram type and evaluation criteria.

Abstract

In software engineering (SE) research and practice, UML is well known as an essential modeling methodology for requirements analysis and software modeling in both academia and industry. In particular, fundamental knowledge of UML modeling and practice in creating high-quality UML diagrams are included in SE-relevant courses in the undergraduate programs of many universities. This leads to a time-consuming and labor-intensive task for educators to review and grade a large number of UML diagrams created by the students. Recent advances in generative AI techniques, such as GPT, have paved new ways to automate many SE tasks. However, current research or tools seldom explore the capabilities of GPT in evaluating the quality of UML diagrams. This paper aims to investigate the feasibility and performance of GPT in assessing the quality of UML use case diagrams, class diagrams, and sequence diagrams. First, 11 evaluation criteria with grading details were proposed for these UML diagrams. Next, a series of experiments was designed and conducted on 40 students' UML modeling reports to explore the performance of GPT in evaluating and grading these UML diagrams. The research findings reveal that GPT can complete this assessment task, but it cannot replace human experts yet. Meanwhile, there are five evaluation discrepancies between GPT and human experts. These discrepancies vary in the use of different evaluation criteria in different types of UML diagrams, presenting GPT's strengths and weaknesses in this automatic evaluation task.