Effect of pseudo datasets for the classification-based engineering design

TL;DR

This study evaluates how pseudo datasets generated by surrogate models influence classification accuracy in engineering design, highlighting the effectiveness of certain classifiers and the impact of surrogate uncertainty.

Contribution

It investigates the mutual effects of pseudo datasets and surrogate modeling uncertainty on classification performance in engineering design.

Findings

Large pseudo datasets improve classification accuracy depending on the problem and algorithm.

Support vector machine, random forest, and neural network outperform Naive Bayes with pseudo data.

Random forest shows high robustness under surrogate uncertainty, while neural networks are more sensitive.

Abstract

Machine learning classification techniques have been used widely to recognize the feasible design domain and discover hidden patterns in engineering design. An accurate classification model needs a large dataset; however, generating a large dataset is costly for complex simulation-based problems. After training by a small dataset, surrogate models can generate a large pseudo dataset efficiently. Errors, however, may be introduced by surrogate modeling. This paper investigates the mutual effect of a large pseudo dataset and surrogate modeling uncertainty. Four widely used methods, i.e., Naive Bayes classifier, support vector machine, random forest regression, and artificial neural network for classification, are studied on four benchmark problems. Kriging is used as the basic surrogate model method. The results show that a large pseudo dataset improves the classification accuracy, which depends on both design problems and classification algorithms. Except for the Naive Bayes, the other three methods are recommended for using pseudo data to improve classification performance. Also, a wind turbine design problem is used to illustrate the effect of the pseudo dataset on feasible subspace recognition. The large pseudo dataset improves the recognized subspace bound greatly, which can be reproduced by classification models well except for the Naive Bayes classifier. Under the uncertainty of surrogate modeling, the random forest presents high robustness to recognize the feasible design domain bound, while the artificial neural network demonstrates a high sensitivity to uncertainty with the recognized bound deteriorated.