Learning Deep Semantics for Test Completion

TL;DR

This paper introduces TeCo, a deep learning model that leverages code semantics to automatically complete test methods, significantly improving accuracy and functional correctness over syntax-only models.

Contribution

The paper presents a novel task of test completion and develops TeCo, a deep learning model that incorporates code semantics for improved test statement prediction.

Findings

TeCo achieves 18% exact-match accuracy, 29% higher than syntax-only baselines.

TeCo generates runnable code in 29% of cases, outperforming the 18% of the best baseline.

TeCo outperforms prior work on test oracle generation.

Abstract

Writing tests is a time-consuming yet essential task during software development. We propose to leverage recent advances in deep learning for text and code generation to assist developers in writing tests. We formalize the novel task of test completion to automatically complete the next statement in a test method based on the context of prior statements and the code under test. We develop TeCo -- a deep learning model using code semantics for test completion. The key insight underlying TeCo is that predicting the next statement in a test method requires reasoning about code execution, which is hard to do with only syntax-level data that existing code completion models use. TeCo extracts and uses six kinds of code semantics data, including the execution result of prior statements and the execution context of the test method. To provide a testbed for this new task, as well as to evaluate TeCo, we collect a corpus of 130,934 test methods from 1,270 open-source Java projects. Our results show that TeCo achieves an exact-match accuracy of 18, which is 29% higher than the best baseline using syntax-level data only. When measuring functional correctness of generated next statement, TeCo can generate runnable code in 29% of the cases compared to 18% obtained by the best baseline. Moreover, TeCo is significantly better than prior work on test oracle generation.