Which Combination of Test Metrics Can Predict Success of a Software Project? A Case Study in a Year-Long Project Course

TL;DR

This study analyzes how different combinations of automated and manual testing metrics influence the success of software projects, emphasizing the importance of comprehensive testing for meeting functional requirements in a year-long course.

Contribution

It provides empirical evidence on the impact of various testing types and coverage metrics on software success, using statistical analysis in an educational setting.

Findings

High code coverage across automated and manual testing significantly improves functional suitability.

Front-end unit testing and manual testing independently predict successful software delivery.

Low back-end testing is nearly significantly associated with postponement of user stories.

Abstract

Testing plays an important role in securing the success of a software development project. Prior studies have demonstrated beneficial effects of applying acceptance testing within a Behavioural-Driven Development method. In this research, we investigate whether we can quantify the effects various types of testing have on functional suitability, i.e. the software conformance to users' functional expectations. We explore which combination of software testing (automated and manual, including acceptance testing) should be applied to ensure the expected functional requirements are met, as well as whether the lack of testing during a development iteration causes a significant increase of effort spent fixing the project later on. To answer those questions, we collected and analysed data from a year-long software engineering project course. We combined manual observations and statistical methods, namely Linear Mixed-Effects Modelling, to evaluate the effects of coverage metrics as well as time effort on passed stories over 5 Scrum sprints. The results suggest that a combination of a high code coverage for all of automated unit, acceptance, and manual testing has a significant impact on functional suitability. Similarly, but to a lower extent, front-end unit testing and manual testing can predict the success of a software delivery when taken independently. We observed a close-to-significant effect between low back-end testing and deferral (i.e. postponement) of user stories.