A System Model Generation Benchmark from Natural Language Requirements

TL;DR

This paper introduces SysMBench, a benchmark dataset with 151 scenarios for evaluating large language models' ability to generate system models from natural language requirements, highlighting current limitations.

Contribution

The paper presents SysMBench and SysMEval, the first benchmark and evaluation metric for assessing LLMs in system model generation from natural language.

Findings

LLMs perform poorly on the benchmark, with BLEU scores up to 4%.

SysMEval achieves a maximum F1 score of 62%.

The benchmark and evaluation framework are publicly released for future research.

Abstract

System models, a critical artifact in software development, provide a formal abstraction of both the structural and behavioral aspects of software systems, which can facilitate the early requirements analysis and architecture design. However, developing system models remains challenging due to the specific syntax of model description languages and the relative scarcity of public model examples. While large language models (LLMs) have shown promise in generating code with programming languages and could potentially aid in system model development, no benchmarks currently exist for evaluating their ability to generate system models with specific description languages. We present SysMBench, which comprises 151 human-curated scenarios spanning a wide range of popular domains and varying difficulty levels. Each scenario mainly comprises a natural language requirements description, a system model expressed in a specific model description language, and a visualized system model diagram. The requirements description is fed as user input to the LLM, the system model with description language is used to verify if the generated system model conforms to the requirements, and the visualized diagram serves to support manual validation. We introduce SysMEval, a semantic-aware evaluation metric to evaluate the quality of generated system models. We evaluate 17 popular LLMs on this task with three traditional metrics and SysMEval, from directly prompting to three commonly used enhancement strategies. Our in-depth evaluation shows that LLMs perform poorly on SysMBench, with the highest BLEU of 4% and SysMEval-F1 of 62%. We release the SysMBench and its evaluation framework to enable future research on LLM-based system model generation.