Exploring the Output of Software Testing Tools through a Visual Comparative Analysis

TL;DR

This study conducts a visual comparative analysis of 50 software testing tools to identify common interface elements, visualization patterns, and formatting trends in test outputs across CLI and GUI environments.

Contribution

It provides the first detailed analysis of shared interface elements and visualization patterns in testing tool outputs, informing future tool development.

Findings

Identified common interface elements across testing tools.

Analyzed how color and formatting are used in test result visualizations.

Revealed trends in output presentation for CLI and GUI testing tools.

Abstract

Software testing is a fundamental process of software development, and prior work has shown that visualizations of test results support testers' decision-making. However, Human-Computer Interaction research on software testing has yet to explore and understand the shared interface elements and patterns in visualization of testing outputs. To address this, we conducted a visual comparative analysis of the output of 50 software testing tools and harnesses (44 with CLI output, 6 with GUI output) across four popular programming languages. Our analysis reveals the common interface elements in software testing tools, how these tools display and visualize test results, as well as the specific make-up of the output. Our findings provide insight on how visual testing output is formatted and how colour is used across both CLI and GUI environments, identifying trends that can be applied by developers of testing tools.