A Review on Oracle Issues in Machine Learning

TL;DR

This paper surveys oracle issues in machine learning, highlighting challenges and solutions related to data quality, testing methods, and robustness improvements to enhance model reliability.

Contribution

It provides a comprehensive review of oracle problems in machine learning and discusses recent advancements and tools for addressing these issues.

Findings

Differential testing and metamorphic testing are key approaches.

Robustness improvements help mitigate oracle issues.

Tools and frameworks assist in dataset testing and issue discovery.

Abstract

Machine learning contrasts with traditional software development in that the oracle is the data, and the data is not always a correct representation of the problem that machine learning tries to model. We present a survey of the oracle issues found in machine learning and state-of-the-art solutions for dealing with these issues. These include lines of research for differential testing, metamorphic testing, and test coverage. We also review some recent improvements to robustness during modeling that reduce the impact of oracle issues, as well as tools and frameworks for assisting in testing and discovering issues specific to the dataset.