Data Synthesis for Testing Black-Box Machine Learning Models

TL;DR

This paper introduces an automated framework for synthesizing realistic test data to evaluate black-box machine learning models, aiming to improve testing coverage and trustworthiness.

Contribution

It presents a novel, model-agnostic data synthesis method for testing ML models with user-controllable, realistic data to enhance testing effectiveness.

Findings

Effective in increasing test coverage

Generates realistic, user-controllable data

Demonstrated success across multiple models

Abstract

The increasing usage of machine learning models raises the question of the reliability of these models. The current practice of testing with limited data is often insufficient. In this paper, we provide a framework for automated test data synthesis to test black-box ML/DL models. We address an important challenge of generating realistic user-controllable data with model agnostic coverage criteria to test a varied set of properties, essentially to increase trust in machine learning models. We experimentally demonstrate the effectiveness of our technique.