Machines that test Software like Humans

TL;DR

This paper proposes a novel approach to automate software testing by enabling machines to mimic human visual perception and interaction, leveraging recent advances in computer vision to reduce scripting complexity and improve practical adoption.

Contribution

The paper introduces a computer vision-based method that allows machines to perform software testing like humans, addressing the complexity of current automation scripts and enhancing practical usability.

Findings

Machine can interpret GUI visually similar to humans.

Approach reduces the need for detailed implementation specifics.

Four use-cases demonstrate significant advancements in test automation.

Abstract

Automated software testing involves the execution of test scripts by a machine instead of being manually run. This significantly reduces the amount of manual time & effort needed and thus is of great interest to the software testing industry. There have been various tools developed to automate the testing of web applications (e.g. Selenium WebDriver); however, the practical adoption of test automation is still miniscule. This is due to the complexity of creating and maintaining automation scripts. The key problem with the existing methods is that the automation test scripts require certain implementation specifics of the Application Under Test (AUT) (e.g. the html code of a web element, or an image of a web element). On the other hand, if we look at the way manual testing is done, the tester interprets the textual test scripts and interacts with the AUT purely based on what he perceives visually through the GUI. In this paper, we present an approach to build a machine that can mimic human behavior for software testing using recent advances in Computer Vision. We also present four use-cases of how this approach can significantly advance the test automation space making test automation simple enough to be adopted practically.