Three Controlled Experiments in Software Engineering with the Two-Tier Programming Toolkit: Final Report

TL;DR

This paper reports on three controlled experiments demonstrating that the Two-Tier Programming Toolkit significantly improves productivity and accuracy in software engineering tasks for Java programmers, especially with minimal training.

Contribution

It provides empirical evidence of the benefits of the TTP Toolkit in enhancing programmer productivity and accuracy, with detailed experimental results.

Findings

76% productivity gain with minimal training

Almost tripled accuracy in quality-related tasks

No significant productivity gain for very short tasks

Abstract

Three controlled experiments testing the benefits that Java programmers gain from using the Two-Tier Programming Toolkit have recently been concluded. The first experiment offers statistically significant evidence (p-value: 0.02) that programmers who undertook only minimal (1-hour) training in using the current prototype exhibit 76% productivity gains in key tasks in software development and maintenance. The second experiment shows that the use of the TTP Toolkit is likely (p-value: 0.10) to almost triple the accuracy of programmers performing tasks associated with software quality. The third experiment shows that the TTP Toolkit does not offer significant productivity gains in performing very short (under 10 min.) tasks.