Assertion-Aware Test Code Summarization with Large Language Models

TL;DR

This paper investigates how different prompting strategies influence large language models' ability to generate concise, accurate summaries of Java test code, emphasizing assertion semantics and providing a new benchmark dataset.

Contribution

It introduces a benchmark of 91 real-world Java test cases, conducts an ablation study on prompt components, and evaluates multiple LLMs with various prompt configurations for test code summarization.

Findings

Assertion semantics improve summary quality and reduce input tokens.

Codex and Qwen-Coder outperform other models in aligning with human summaries.

DeepSeek underperforms despite lexical overlap.

Abstract

Unit tests often lack concise summaries that convey test intent, especially in auto-generated or poorly documented codebases. Large Language Models (LLMs) offer a promising solution, but their effectiveness depends heavily on how they are prompted. Unlike generic code summarization, test-code summarization poses distinct challenges because test methods validate expected behavior through assertions rather than implementing functionality. This paper presents a new benchmark of 91 real-world Java test cases paired with developer-written summaries and conducts a controlled ablation study to investigate how test code-related components-such as the method under test (MUT), assertion messages, and assertion semantics-affect the performance of LLM-generated test summaries. We evaluate four code LLMs (Codex, Codestral, DeepSeek, and Qwen-Coder) across seven prompt configurations using n-gram metrics (BLEU, ROUGE-L, METEOR), semantic similarity (BERTScore), and LLM-based evaluation. Results show that prompting with assertion semantics improves summary quality by an average of 0.10 points (2.3%) over full MUT context (4.45 vs. 4.35) while requiring fewer input tokens. Codex and Qwen-Coder achieve the highest alignment with human-written summaries, while DeepSeek underperforms despite high lexical overlap. The replication package is publicly available at https://doi.org/10. 5281/zenodo.17067550