Alternatives for Testing of Context-Aware Contemporary Software Systems in industrial settings: Results from a Rapid review

TL;DR

This paper reviews industry and research approaches to testing context-aware software systems, highlighting current solutions focus on test generation and modeling, but lack comprehensive lifecycle management and mainstream readiness.

Contribution

It provides a rapid review of existing research and industry practices, revealing gaps in managing context variation throughout the testing lifecycle of CACSS.

Findings

Research focuses on test suite generation and system behavior modeling.

Most solutions rely on increased computing resources for testing.

No evidence of lifecycle management approaches for context variation.

Abstract

Context: Context-aware contemporary software systems (CACSS) are mainstream. Furthermore, they present challenges for current engineering practices. These challenges are distinctively present when testing CACSS, as the variation of context deepens the limitations of available software testing practices and technologies. Objective: To understand how the industry deals with the variation of context when testing CACSS. Method: A Rapid Review was commissioned to uncover the necessary evidence to achieve the objectives. Results: Our results show that current research initiatives aim to generate or improve Test Suites that can deal with the variation of context and the sheer volume of test input possibilities. To achieve this, they mostly rely on modelling the systems' dynamic behavior and increasing computing resources to generate test inputs. We found no evidence of research results aiming at managing context variation through the testing lifecycle process. Conclusions: We discuss how the identified solutions are not ready for mainstream adoption. They are all domain-specific, and while the ideas and approaches can be reproduced in different settings, the technologies noon to be re-engineered and tailor to the specific CACSS.