Accurate and Consistent Graph Model Generation from Text with Large Language Models

TL;DR

This paper introduces a new framework that leverages multiple outputs from large language models to generate more accurate, consistent, and constraint-satisfying graph models from natural language descriptions, addressing common issues like syntax violations and hallucinations.

Contribution

The paper presents an abstraction-concretization framework that improves graph model generation from text by aggregating multiple LLM outputs and refining them to satisfy constraints, enhancing quality and consistency.

Findings

Significant improvement in model correctness and consistency.

Effective reduction of syntax violations and hallucinations.

Framework outperforms baseline methods on multiple datasets.

Abstract

Graph model generation from natural language description is an important task with many applications in software engineering. With the rise of large language models (LLMs), there is a growing interest in using LLMs for graph model generation. Nevertheless, LLM-based graph model generation typically produces partially correct models that suffer from three main issues: (1) syntax violations: the generated model may not adhere to the syntax defined by its metamodel, (2) constraint inconsistencies: the structure of the model might not conform to some domain-specific constraints, and (3) inaccuracy: due to the inherent uncertainty in LLMs, the models can include inaccurate, hallucinated elements. While the first issue is often addressed through techniques such as constraint decoding or filtering, the latter two remain largely unaddressed. Motivated by recent self-consistency approaches in LLMs, we propose a novel abstraction-concretization framework that enhances the consistency and quality of generated graph models by considering multiple outputs from an LLM. Our approach first constructs a probabilistic partial model that aggregates all candidate outputs and then refines this partial model into the most appropriate concrete model that satisfies all constraints. We evaluate our framework on several popular open-source and closed-source LLMs using diverse datasets for model generation tasks. The results demonstrate that our approach significantly improves both the consistency and quality of the generated graph models.