From Text to DSL: Evaluating Grammar-Based Model Generation Using Open LLMs

TL;DR

This paper evaluates the ability of open-source LLMs of various sizes to generate DSL-conformant models from natural language without fine-tuning, focusing on syntactic validity, semantic completeness, and consistency.

Contribution

It extends prior work by generating both UI and data models from scratch, demonstrating smaller open-source LLMs can effectively produce grammar-conformant models for MDE tasks.

Findings

Several compact models approach or match larger models in quality.

Smaller open-source LLMs are feasible for grammar-conformant DSL generation.

Evaluation combines automatic parsing and expert feedback across 39 LLMs.

Abstract

Large Language Models (LLMs) have shown increasing potential in automating model-driven software engineering tasks, particularly in generating models conforming to Domain Specific Languages (DSLs) from natural language. While most existing approaches rely on large proprietary models, their high cost and limited deployability hinder broader adoption. In this paper, we evaluate whether open-source LLMs of varying sizes (0.5B to 32B parameters) can generate DSL-conformant models using only few-shot prompting, without any fine-tuning. Our evaluation focuses on key model-driven engineering (MDE) requirements, including syntactic validity, semantic completeness, and inter-model reference consistency. We extend our prior work by moving from generating user interface models (referred to as "UI models" in this paper) over fixed, predefined data schemas ("data models") to generating both the UI and data models entirely from scratch. This shift serves two purposes: first, it highlights the LLM's ability to infer domain-specific relationships and maintain consistency across multiple interconnected models; second, it allows us to generalize earlier findings by testing DSL generation across models of different natures and structural roles. Our structured evaluation combines automatic parsing and expert feedback across 39 LLMs, revealing that several compact models (e.g., \texttt{gemma3:12b}, \texttt{mistral:7b-instruct}) approach or match the quality of much larger models. These findings demonstrate the feasibility of using smaller, open-source LLMs for grammar-conformant DSL generation in MDE workflows, offering a cost-effective and deployable alternative to closed LLMs.