Isolating Noisy Labelled Test Cases in Human-in-the-Loop Oracle Learning

TL;DR

This paper presents ISONOISE, a method to identify and correct mislabelled test cases in human-in-the-loop oracle learning, improving the reliability of the training process with high accuracy and minimal relabelling effort.

Contribution

Introduces ISONOISE, a novel technique for isolating mislabelled test cases in human-in-the-loop oracle learning, enhancing test suite accuracy and learning effectiveness.

Findings

ISONOISE achieves over 67% accuracy in detecting mislabelled test cases.

The method requires only a small number of relabelling queries.

Experimental results show improved reliability in human-in-the-loop learning.

Abstract

Incorrectly labelled test cases can adversely affect the training process of human-in-the-loop oracle learning tech-niques. This paper introduces ISONOISE, a technique designed to identify such mislabelled test cases introduced during human-in-the-loop oracle learning. This technique can be applied to programs taking numeric inputs. Given a compromised automatic test oracle and its training test suite, ISONOISE first isolates thetest cases suspected of being mislabelled. This task is performed based on the level of disagreement of a test case with respect to the others. An intermediate automatic test oracle is trained based on the slightly disagreeing test cases. Based on the predictions of this intermediate oracle, the test cases suspected of being mislabelled are systematically presented for relabelling. When mislabelled test cases are found, the intermediate test oracle is updated. This process repeats until no mislabelled test case is found in relabelling. ISONOISE was evaluated within the human-in-the-loop oracle learning method used in LEARN2FIX. Experimental results demonstrate that ISONOISE can identify mislabelled test cases introduced by the human in LEARN2FIX with over 67% accuracy, while requiring only a small number of relabelling queries. These findings highlight the potential of ISONOISE to enhance the reliability of human-in-the-loop oracle learning.