Leveraging LLMs for Grammar Adaptation: A Study on Metamodel-Grammar Co-Evolution

TL;DR

This study explores using Large Language Models to automate grammar adaptation in model-driven engineering, showing high accuracy on small to medium grammars but limitations on large-scale ones.

Contribution

It introduces a novel LLM-based method for automatic grammar adaptation, outperforming rule-based approaches in complex scenarios and demonstrating practical effectiveness.

Findings

LLMs achieved 100% consistency on small test sets.

Rule-based methods scored 84.21% and 62.50%.

LLMs struggled with large-scale grammars, dropping below 90%.

Abstract

In model-driven engineering, metamodel evolution leads to the need to adapt corresponding grammars to maintain consistency, which typically requires tedious manual work. Existing rule-based methods can achieve partial automation but have limitations when handling complex grammar scenarios. This paper proposes a Large Language Model-based approach that automatically applies adaptations to new grammars after evolution by learning grammar adaptations from previous versions. We evaluated this approach on six real-world Xtext domain-specific languages, using four DSLs as a training set to develop prompting strategies, two DSLs as a test set for validation, and conducting a longitudinal case study on QVTo. The evaluation used three Large Language Models (Claude Sonnet 4.5, ChatGPT 5.1, Gemini 3) and measured grammar adaptation quality from three dimensions: grammar rule-level adaptation consistency, output similarity, and metamodel conformance. Results show that on the test set, all three LLMs achieved 100% adaptation consistency and output similarity, while the rule-based approach achieved only 84.21% on DOT and 62.50% on Xcore. In the QVTo longitudinal study, the LLM-based approach successfully reused learned adaptations across all three evolution steps without manual grammar editing, while the rule-based approach required manual adjustments in two of three transitions. However, on large-scale grammars (EAST-ADL, 297 rules), LLMs' adaptation consistency was far below 90%. This study demonstrates the advantages of LLM-based approaches in handling complex grammar scenarios, while revealing their limitations in large-scale grammar adaptation.