Beyond Accuracy: ROI-driven Data Analytics of Empirical Data

TL;DR

This paper emphasizes the importance of ROI in guiding data analytics decisions, proposing a framework validated through case studies on dependency extraction in Mozilla Firefox, demonstrating how ROI can optimize analysis efforts.

Contribution

It introduces a conceptual framework for ROI-driven data analytics and validates it with empirical studies, guiding when to stop analysis based on cost-benefit considerations.

Findings

Fine-tuned BERT outperforms Random Forest with over 40% training data.

Active Learning achieves higher accuracy with fewer iterations.

ROI analysis helps determine the optimal stopping point for data analysis.

Abstract

This vision paper demonstrates that it is crucial to consider Return-on-Investment (ROI) when performing Data Analytics. Decisions on "How much analytics is needed"? are hard to answer. ROI could guide for decision support on the What?, How?, and How Much? analytics for a given problem. Method: The proposed conceptual framework is validated through two empirical studies that focus on requirements dependencies extraction in the Mozilla Firefox project. The two case studies are (i) Evaluation of fine-tuned BERT against Naive Bayes and Random Forest machine learners for binary dependency classification and (ii) Active Learning against passive Learning (random sampling) for REQUIRES dependency extraction. For both the cases, their analysis investment (cost) is estimated, and the achievable benefit from DA is predicted, to determine a break-even point of the investigation. Results: For the first study, fine-tuned BERT performed superior to the Random Forest, provided that more than 40% of training data is available. For the second, Active Learning achieved higher F1 accuracy within fewer iterations and higher ROI compared to Baseline (Random sampling based RF classifier). In both the studies, estimate on, How much analysis likely would pay off for the invested efforts?, was indicated by the break-even point. Conclusions: Decisions for the depth and breadth of DA of empirical data should not be made solely based on the accuracy measures. Since ROI-driven Data Analytics provides a simple yet effective direction to discover when to stop further investigation while considering the cost and value of the various types of analysis, it helps to avoid over-analyzing empirical data.