Towards the Systematic Testing of Virtual Reality Programs (extended version)

TL;DR

This paper emphasizes the importance of systematic testing in VR software development, highlighting current deficiencies, high fault-proneness, and the need for better testing tools to improve reliability and quality.

Contribution

It provides an analysis of VR open-source projects, revealing the lack of testing practices, high code smell incidence, and the impact on fault-proneness, advocating for systematic testing approaches.

Findings

Low adoption of testing practices in VR projects

High incidence of code smells in VR software

Approximately 12.2% of classes are fault-prone

Abstract

Software testing is a critical activity to ensure that software complies with its specification. However, current software testing activities tend not to be completely effective when applied in specific software domains in Virtual Reality (VR) that has several new types of features such as images, sounds, videos, and differentiated interaction, which can become sources of new kinds of faults. This paper presents an overview of the main VR characteristics that can have an impact on verification, validation, and testing (VV&T). Furthermore, it analyzes some of the most successful VR open-source projects to draw a picture concerning the danger of the lack of software testing activities. We compared the current state of software testing practice in open-source VR projects and evaluate how the lack of testing can be damaging to the development of a product. We assessed the incidence of code smells and verified how such projects behave concerning the tendency to present faults. The results showed that the practice of software testing is not yet widespread in the development of VR applications. It was also found that there is a high incidence of code smells in VR projects. Regarding fault-proneness the results showed that about 12.2% of the classes analyzed in VR projects are fault-prone. Regarding the application of software testing techniques on VR projects, it was observed that only a small number of projects are concerned about developing test cases for VR projects, perhaps because we still do not have the necessary tools to help in this direction. Concerning smells, we concluded that there is a high incidence in VR projects, especially regarding implementing smells and this high incidence can have a significant influence on faults. Finally, the study related to fault proneness pointed out that the lack of software testing activity is a significant risk to the success of the projects.