Benchmarking Web-testing - Selenium versus Watir and the Choice of Programming Language and Browser

TL;DR

This study empirically compares web-testing tools Selenium and Watir across different programming languages and browsers, highlighting performance trade-offs to inform better tool and configuration choices.

Contribution

It provides a comprehensive benchmarking of Selenium and Watir with various language bindings and browsers, offering empirical data to guide test automation decisions.

Findings

Selenium with Python and Chrome was the fastest configuration.

Internet Explorer was the fastest browser but had poor stability.

Benchmarking tools before adoption is highly recommended.

Abstract

Context: Selenium is claimed to be the most popular software test automation tool. Past academic works have mainly neglected testing tools in favor of more methodological topics. Objective: We investigated the performance of web-testing tools, to provide empirical evidence supporting choices in software test tool selection and configuration. Method: We used 4*5 factorial design to study 20 different configurations for testing a web-store. We studied 5 programming language bindings (C#, Java, Python, and Ruby for Selenium, while Watir supports Ruby only) and 4 browsers (Google Chrome, Internet Explorer, Mozilla Firefox and Opera). Performance was measured with execution time, memory usage, length of the test scripts and stability of the tests. Results: Considering all measures the best configuration was Selenium with Python language binding for Chrome. Selenium with Python bindings was the best option for all browsers. The effect size of the difference between the slowest and fastest configuration was very high (Cohens d=41.5, 91% increase in execution time). Overall Internet Explorer was the fastest browser while having the worst results in the stability. Conclusions: We recommend benchmarking tools before adopting them. Weighting of factors, e.g. how much test stability is one willing to sacrifice for faster performance, affects the decision.