Evaluation of LLM-Based Software Engineering Tools: Practices, Challenges, and Future Directions

TL;DR

This paper critically examines the challenges and limitations of evaluating LLM-based software engineering tools, emphasizing the need for robust, scalable, and trustworthy evaluation methods.

Contribution

It provides a comprehensive analysis of current evaluation practices, identifies key challenges, and proposes future directions for improving LLM evaluation in software engineering.

Findings

Current evaluation methods face issues with non-determinism and lack of ground truth.

Subjectivity and multi-dimensional quality complicate LLM assessment.

Fragmented practices hinder consistent and reliable evaluation.

Abstract

Large Language Models (LLMs) are increasingly embedded in software engineering (SE) tools, powering applications such as code generation, automated code review, and bug triage. As these LLM-based AI for Software Engineering (AI4SE) systems transition from experimental prototypes to widely deployed tools, the question of what it means to evaluate their behavior reliably has become both critical and unanswered. Unlike traditional SE or machine learning systems, LLM-based tools often produce open-ended, natural language outputs, admit multiple valid answers, and exhibit non-deterministic behavior across runs. These characteristics fundamentally challenge long-standing evaluation assumptions such as the existence of a single ground truth, deterministic outputs, and objective correctness. In this paper, we examine LLM evaluation as a general, task-dependent concept through the lens of SE tasks. We discuss why reliable evaluation is essential for trust, adoption, and meaningful assessment of LLM-based tools, summarize the current state of evaluation practices, and highlight their limitations in realistic AI4SE settings. We then identify key challenges facing current approaches, including the absence of stable ground truth, subjectivity and multi-dimensional quality, evaluation instability due to non-determinism, limitations of automated and model-based evaluation, and fragmentation of evaluation practices. Finally, we outline future directions aimed at advancing LLM evaluation toward more robust, scalable, and trustworthy methodologies, to stimulate discussion on principled evaluation practices that can keep pace with the growing role of LLMs in SE.