Test Oracle Automation in the era of LLMs
Facundo Molina, Alessandra Gorla

TL;DR
This paper explores the potential and challenges of using Large Language Models to automate test oracles, emphasizing their role in improving software testing and highlighting associated risks.
Contribution
It initiates a discussion on leveraging LLMs for test oracle automation, addressing benefits, challenges, and threats in software testing.
Findings
LLMs show promise in automating test oracles.
Challenges include oracle correctness and data security concerns.
The paper highlights the need for further research in this area.
Abstract
The effectiveness of a test suite in detecting faults highly depends on the correctness and completeness of its test oracles. Large Language Models (LLMs) have already demonstrated remarkable proficiency in tackling diverse software testing tasks, such as automated test generation and program repair. This paper aims to enable discussions on the potential of using LLMs for test oracle automation, along with the challenges that may emerge during the generation of various types of oracles. Additionally, our aim is to initiate discussions on the primary threats that SE researchers must consider when employing LLMs for oracle automation, encompassing concerns regarding oracle deficiencies and data leakages.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Data Storage Technologies · Power Systems Fault Detection · Technology and Security Systems
