Do LLMs generate test oracles that capture the actual or the expected program behaviour?

TL;DR

This paper investigates whether Large Language Models generate test oracles that accurately reflect the expected program behavior, revealing they tend to capture actual behavior and are more effective at generating than classifying oracles.

Contribution

It provides an empirical analysis of LLM-generated test oracles, highlighting their limitations and strengths in capturing expected behavior compared to traditional methods.

Findings

LLMs tend to generate oracles capturing actual behavior rather than expected.

LLMs are better at generating than classifying test oracles.

LLM-generated oracles have higher fault detection potential.

Abstract

Software testing is an essential part of the software development cycle to improve the code quality. Typically, a unit test consists of a test prefix and a test oracle which captures the developer's intended behaviour. A known limitation of traditional test generation techniques (e.g. Randoop and Evosuite) is that they produce test oracles that capture the actual program behaviour rather than the expected one. Recent approaches leverage Large Language Models (LLMs), trained on an enormous amount of data, to generate developer-like code and test cases. We investigate whether the LLM-generated test oracles capture the actual or expected software behaviour. We thus, conduct a controlled experiment to answer this question, by studying LLMs performance on two tasks, namely, test oracle classification and generation. The study includes developer-written and automatically generated test cases and oracles for 24 open-source Java repositories, and different well tested prompts. Our findings show that LLM-based test generation approaches are also prone on generating oracles that capture the actual program behaviour rather than the expected one. Moreover, LLMs are better at generating test oracles rather than classifying the correct ones, and can generate better test oracles when the code contains meaningful test or variable names. Finally, LLM-generated test oracles have higher fault detection potential than the Evosuite ones.