Quality Assurance of LLM-generated Code: Addressing Non-Functional Quality Characteristics

TL;DR

This study investigates the non-functional quality characteristics of LLM-generated code, revealing gaps between research focus, industry priorities, and actual model behavior, emphasizing the need for integrated quality assurance.

Contribution

It provides a comprehensive multi-method analysis of non-functional qualities in LLM-generated code, highlighting misalignments and proposing the need for quality assurance mechanisms.

Findings

Research mainly emphasizes security, performance, and maintainability.

Practitioners prioritize maintainability and readability.

Optimizing non-functional qualities via prompts is unstable in practice.

Abstract

In recent years, large language models have been widely integrated into software engineering workflows, supporting tasks like code generation. While prior evaluations focus on functional correctness, there is still a limited understanding of the non-functional quality characteristics of generated code. Guided by the ISO/IEC 25010 quality model, this study adopts a multi-methods approach comprising three complementary elements: a literature review of 109 papers, two industry workshops with practitioners from multiple organizations, and an empirical analysis of patching real-world software issues using three LLMs. Motivated by insights from both the literature and practitioners, the empirical study examined the quality of generated patches regarding security, maintainability, and performance efficiency, which were identified as critical code-level quality attributes. Our results indicate that existing research primarily emphasizes security, performance efficiency, and maintainability, while other quality attributes are understudied. In contrast, practitioners prioritize maintainability and readability, warning that generated code may accelerate the accumulation of technical debt. The empirical evaluation demonstrates the instability of optimizing NFQCs through prompts in practical software engineering settings. Overall, our findings expose a misalignment between academic focus, industry priorities, and observed model behavior, highlighting the need to integrate quality assurance mechanisms into LLM code generation pipelines to ensure that future generated code not only passes tests but truly passes with quality.