LLM Assisted Coding with Metamorphic Specification Mutation Agent

TL;DR

This paper introduces CodeMetaAgent, an LLM-based framework that uses metamorphic relations to refine task specifications and generate more reliable, semantically consistent code, significantly improving accuracy and coverage.

Contribution

The paper presents a novel LLM agent leveraging metamorphic relations to systematically refine specifications and enhance code generation reliability, a departure from traditional post-validation use of MRs.

Findings

CodeMetaAgent improved code accuracy by up to 17%.

CodeMetaAgent achieved code coverage of up to 99.81%.

Framework demonstrated effectiveness across multiple datasets and models.

Abstract

Metamorphic Relations (MRs) serve as a foundational mechanism for generating semantically equivalent mutations. Software engineering has advanced significantly in recent years with the advent of Large Language Models (LLMs). However, the reliability of LLMs in software engineering is often compromised by ambiguities and inconsistencies due to improper user specification. To address this challenge, we present CodeMetaAgent (CMA), a metamorphic relation-driven LLM agent that systematically refines task specifications and generates semantically constrained test cases. Our proposed framework uses MRs with LLMs to improve generation consistency and reduce variability caused by specifications, unlike the traditional use of MRs as post validations. Our framework has been evaluated on the HumanEval-Pro, MBPP-Pro, and SWE-Bench_Lite datasets using the GPT-4o, Mistral Large, GPT-OSS, and Qwen3-Coder models. It improved code generation accuracy by up to 17% and achieved code coverage gains of up to 99.81%. These results show that metamorphic relations can be a simple but effective guide in assisting LLM-based software development.