Revisiting and Improving Retrieval-Augmented Deep Assertion Generation

TL;DR

This paper analyzes the limitations of existing retrieval-augmented assertion generation methods and introduces EditAS, a novel retrieve-and-edit approach that significantly improves accuracy and BLEU scores in unit test assertion generation.

Contribution

The paper provides an in-depth analysis of retrieval-augmented assertion generation and proposes EditAS, a more generalizable method that outperforms state-of-the-art approaches.

Findings

Retrieval success is the main factor in overall performance.

Integration struggles with semantic differences and token edits.

EditAS achieves significant improvements in accuracy and BLEU scores.

Abstract

Unit testing validates the correctness of the unit under test and has become an essential activity in software development process. A unit test consists of a test prefix that drives the unit under test into a particular state, and a test oracle (e.g., assertion), which specifies the behavior in that state. To reduce manual efforts in conducting unit testing, Yu et al. proposed an integrated approach (integration for short), combining information retrieval (IR) with a deep learning-based approach, to generate assertions for a unit test. Despite promising, there is still a knowledge gap as to why or where integration works or does not work. In this paper, we describe an in-depth analysis of the effectiveness of integration. Our analysis shows that: 1) The overall performance of integration is mainly due to its success in retrieving assertions. 2) integration struggles to understand the semantic differences between the retrieved focal-test (focal-test includes a test prefix and a unit under test) and the input focal-test; 3) integration is limited to specific types of edit operations and cannot handle token addition or deletion. To improve the effectiveness of assertion generation, this paper proposes a novel retrieve-and-edit approach named EditAS. Specifically, EditAS first retrieves a similar focal-test from a pre-defined corpus and treats its assertion as a prototype. Then, EditAS reuses the information in the prototype and edits the prototype automatically. EditAS is more generalizable than integration. We conduct experiments on two large-scale datasets and experimental results demonstrate that EditAS outperforms the state-of-the-art approaches, with an average improvement of 10.00%-87.48% and 3.30%-42.65% in accuracy and BLEU score, respectively.