The Impact of Sampling and Rule Set Size on Generated Fuzzy Inference System Predictive Accuracy: Analysis of a Software Engineering Data Set

TL;DR

This paper empirically examines how sampling methods and rule set size influence the predictive accuracy of fuzzy inference systems in software engineering, highlighting the importance of multiple sample rule sets for better predictions.

Contribution

It provides new insights into the effects of sampling and rule set size on fuzzy inference system accuracy in software engineering applications.

Findings

Multiple sample rule sets improve accuracy

No consistent pattern with rule set size

Empirical analysis is sensitive to model-building choices

Abstract

Software project management makes extensive use of predictive modeling to estimate product size, defect proneness and development effort. Although uncertainty is acknowledged in these tasks, fuzzy inference systems, designed to cope well with uncertainty, have received only limited attention in the software engineering domain. In this study we empirically investigate the impact of two choices on the predictive accuracy of generated fuzzy inference systems when applied to a software engineering data set: sampling of observations for training and testing; and the size of the rule set generated using fuzzy c-means clustering. Over ten samples we found no consistent pattern of predictive performance given certain rule set size. We did find, however, that a rule set compiled from multiple samples generally resulted in more accurate predictions than single sample rule sets. More generally, the results provide further evidence of the sensitivity of empirical analysis outcomes to specific model-building decisions.