Can We Trust Tests To Automate Dependency Updates? A Case Study of Java Projects

TL;DR

This study evaluates the reliability of tests in automating dependency updates in Java projects, revealing significant coverage gaps and proposing impact analysis to improve fault detection in dependency management.

Contribution

It provides a comprehensive analysis of test coverage and fault detection effectiveness in dependency updates, and introduces impact analysis as a complementary approach.

Findings

Tests cover only 58% of direct and 20% of transitive dependency calls.

Tests detect 47% of direct and 35% of indirect artificial faults.

Impact analysis uncovers up to 74% of faults, nearly doubling test effectiveness.

Abstract

Developers are increasingly using services such as Dependabot to automate dependency updates. However, recent research has shown that developers perceive such services as unreliable, as they heavily rely on test coverage to detect conflicts in updates. To understand the prevalence of tests exercising dependencies, we calculate the test coverage of direct and indirect uses of dependencies in 521 well-tested Java projects. We find that tests only cover 58% of direct and 20% of transitive dependency calls. By creating 1,122,420 artificial updates with simple faults covering all dependency usages in 262 projects, we measure the effectiveness of test suites in detecting semantic faults in dependencies; we find that tests can only detect 47% of direct and 35% of indirect artificial faults on average. To increase reliability, we investigate the use of change impact analysis as a means of reducing false negatives; on average, our tool can uncover 74% of injected faults in direct dependencies and 64% for transitive dependencies, nearly two times more than test suites. We then apply our tool in 22 real-world dependency updates, where it identifies three semantically conflicting cases and five cases of unused dependencies. Our findings indicate that the combination of static and dynamic analysis should be a requirement for future dependency updating systems.