How Low Can You Go? The Data-Light SE Challenge

TL;DR

This paper challenges the assumption that extensive datasets and complex optimizers are necessary in Software Engineering, demonstrating that simple, lightweight methods often achieve near-optimal results with minimal data.

Contribution

It introduces the data-light challenge, formalizes labeling, proposes lightweight baselines, and provides empirical results showing when simple methods suffice in SE tasks.

Findings

Simple methods achieve over 90% of best results with few labels

Lightweight approaches perform as well as complex optimizers in many cases

Few samples are sufficient for rapid, cost-effective SE guidance

Abstract

Much of Software Engineering (SE) research assumes that progress depends on massive datasets and CPU-intensive optimizers. Yet has this assumption been rigorously tested? The counter-evidence presented in this paper suggests otherwise. For over 100 optimization tasks from recent SE papers (including software configuration, performance tuning, product line engineering, project health forecasting, defect prediction, software testing, software process and cost estimation, and cross-domain generalization datasets), even with just a few dozen labels, very simple methods (e.g., diversity sampling, a minimal Bayesian learner, its distance-based non-parametric variant, or random probes) achieve over 90% of the best reported results. Furthermore, these simple methods perform just as well as more complex state-of-the-the-art optimizers like SMAC, TPE, DEHB etc. While some tasks would require better outcomes and more sampling, these results seen after a few dozen samples would suffice for many engineering needs (particularly when the goal is rapid and cost-efficient guidance rather than slow and exhaustive optimization). To say that another ways, at least some SE tasks are better served by lightweight approaches that demand fewer labels and far less computation. We hence propose the data-light challenge: when will a handful of labels suffice for SE tasks? To enable a large-scale investigation of this issue, we contribute (1) a mathematical formalization of labeling, (2) lightweight baseline algorithms, and (3) results on public-domain data showing the conditions under which lightweight methods excel or fail. For the purposes of open science, our scripts and data are online at https://github.com/KKGanguly/NEO .