Benchmarking Web-testing - Selenium versus Watir and the Choice of Programming Language and Browser
Miikka Kuutila, Mika M\"antyl\"a, P\"aivi Raulamo-Jurvanen

TL;DR
This study empirically compares web-testing tools Selenium and Watir across different programming languages and browsers, highlighting performance trade-offs to inform better tool and configuration choices.
Contribution
It provides a comprehensive benchmarking of Selenium and Watir with various language bindings and browsers, offering empirical data to guide test automation decisions.
Findings
Selenium with Python and Chrome was the fastest configuration.
Internet Explorer was the fastest browser but had poor stability.
Benchmarking tools before adoption is highly recommended.
Abstract
Context: Selenium is claimed to be the most popular software test automation tool. Past academic works have mainly neglected testing tools in favor of more methodological topics. Objective: We investigated the performance of web-testing tools, to provide empirical evidence supporting choices in software test tool selection and configuration. Method: We used 4*5 factorial design to study 20 different configurations for testing a web-store. We studied 5 programming language bindings (C#, Java, Python, and Ruby for Selenium, while Watir supports Ruby only) and 4 browsers (Google Chrome, Internet Explorer, Mozilla Firefox and Opera). Performance was measured with execution time, memory usage, length of the test scripts and stability of the tests. Results: Considering all measures the best configuration was Selenium with Python language binding for Chrome. Selenium with Python bindings was…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Testing and Debugging Techniques · Software Engineering Research · Software System Performance and Reliability
