Unsupposable Test-data Generation for Machine-learned Software

TL;DR

This paper introduces a novel method called unsupposable test-data generation (UTG) using variational autoencoders to create data that challenges machine learning models, helping developers evaluate model robustness against unexpected inputs.

Contribution

The paper proposes UTG, a new approach leveraging VAEs to generate unsupposable data, enhancing model testing by producing data with rare or unforeseen features.

Findings

UTG successfully generated unsupposable data for MNIST and House Sales Price datasets.

Generated data revealed new features, aiding developers in creating more robust test cases.

Feasibility of UTG demonstrated through experimental results.

Abstract

As for software development by machine learning, a trained model is evaluated by using part of an existing dataset as test data. However, if data with characteristics that differ from the existing data is input, the model does not always behave as expected. Accordingly, to confirm the behavior of the model more strictly, it is necessary to create data that differs from the existing data and test the model with that different data. The data to be tested includes not only data that developers can suppose (supposable data) but also data they cannot suppose (unsupposable data). To confirm the behavior of the model strictly, it is important to create as much unsupposable data as possible. In this study, therefore, a method called "unsupposable test-data generation" (UTG)---for giving suggestions for unsupposable data to model developers and testers---is proposed. UTG uses a variational autoencoder (VAE) to generate unsupposable data. The unsupposable data is generated by acquiring latent values with low occurrence probability in the prior distribution of the VAE and inputting the acquired latent values into the decoder. If unsupposable data is included in the data generated by the decoder, the developer can recognize new unsupposable features by referring to the data. On the basis of those unsupposable features, the developer will be able to create other unsupposable data with the same features. The proposed UTG was applied to the MNIST dataset and the House Sales Price dataset. The results demonstrate the feasibility of UTG.