Improving Testing Behavior by Gamifying IntelliJ

TL;DR

This paper presents IntelliGame, a gamified plugin for IntelliJ that motivates developers to improve testing behavior through rewards, leading to increased testing activity and better test coverage in a controlled experiment.

Contribution

The paper introduces IntelliGame, a novel gamified plugin for IntelliJ that directly incentivizes developers to enhance their testing practices within their IDE.

Findings

Developers using IntelliGame wrote more tests.

Higher code coverage and mutation scores were achieved.

Tested developers reached functionality earlier.

Abstract

Testing is an important aspect of software development, but unfortunately, it is often neglected. While test quality analyses such as code coverage or mutation analysis inform developers about the quality of their tests, such reports are viewed only sporadically during continuous integration or code review, if they are considered at all, and their impact on the developers' testing behavior therefore tends to be negligible. To actually influence developer behavior, it may rather be necessary to motivate developers directly within their programming environment, while they are coding. We introduce IntelliGame, a gamified plugin for the popular IntelliJ Java Integrated Development Environment, which rewards developers for positive testing behavior using a multi-level achievement system: A total of 27 different achievements, each with incremental levels, provide affirming feedback when developers exhibit commendable testing behavior, and provide an incentive to further continue and improve this behavior. A controlled experiment with 49 participants given a Java programming task reveals substantial differences in the testing behavior triggered by IntelliGame: Incentivized developers write more tests, achieve higher coverage and mutation scores, run their tests more often, and achieve functionality earlier.