Test Behaviors, Not Methods! Detecting Tests Obsessed by Methods

TL;DR

This paper introduces a new test smell called 'Test Obsessed by Method' that identifies tests covering multiple behaviors via different paths in a single method, using runtime analysis, and demonstrates its presence in Python's standard library tests.

Contribution

The paper proposes a novel runtime analysis-based test smell detection method for identifying tests that verify multiple behaviors within a single production method.

Findings

44 tests exhibited the smell across 11 test suites

Smelly tests verify a median of two behaviors each

Detected tests could be split into 118 distinct tests

Abstract

Best testing practices state that tests should verify a single functionality or behavior of the system. Tests that verify multiple behaviors are harder to understand, lack focus, and are more coupled to the production code. An attempt to identify this issue is the test smell \emph{Eager Test}, which aims to capture tests that verify too much functionality based on the number of production method calls. Unfortunately, prior research suggests that counting production method calls is an inaccurate measure, as these calls do not reliably serve as a proxy for functionality. We envision a complementary solution based on runtime analysis: we hypothesize that some tests that verify multiple behaviors will likely cover multiple paths of the same production methods. Thus, we propose a novel test smell named \emph{Test Obsessed by Method}, a test method that covers multiple paths of a single production method. We provide an initial empirical study to explore the presence of this smell in 2,054 tests provided by 12 test suites of the Python Standard Library. (1) We detect 44 \emph{Tests Obsessed by Methods} in 11 of the 12 test suites. (2) Each smelly test verifies a median of two behaviors of the production method. (3) The 44 smelly tests could be split into 118 novel tests. (4) 23% of the smelly tests have code comments recognizing that distinct behaviors are being tested. We conclude by discussing benefits, limitations, and further research.