Learning Non-robustness using Simulation-based Testing: a Network Traffic-shaping Case Study

TL;DR

This paper presents a simulation-based testing approach using regression trees to identify input ranges causing non-robust behaviors in network traffic-shaping systems, aiding in system robustness and quality-of-service improvements.

Contribution

It introduces a novel method combining simulation and regression models to accurately find non-robust inputs in network systems, validated through industrial collaboration.

Findings

Achieved 84% precision and 100% recall in identifying non-robust inputs.

Demonstrated no significant difference between simulated and hardware testbed results.

Provided insights into using non-robustness as a metric for system re-configuration.

Abstract

An input to a system reveals a non-robust behaviour when, by making a small change in the input, the output of the system changes from acceptable (passing) to unacceptable (failing) or vice versa. Identifying inputs that lead to non-robust behaviours is important for many types of systems, e.g., cyber-physical and network systems, whose inputs are prone to perturbations. In this paper, we propose an approach that combines simulation-based testing with regression tree models to generate value ranges for inputs in response to which a system is likely to exhibit non-robust behaviours. We apply our approach to a network traffic-shaping system (NTSS) -- a novel case study from the network domain. In this case study, developed and conducted in collaboration with a network solutions provider, RabbitRun Technologies, input ranges that lead to non-robustness are of interest as a way to identify and mitigate network quality-of-service issues. We demonstrate that our approach accurately characterizes non-robust test inputs of NTSS by achieving a precision of 84% and a recall of 100%, significantly outperforming a standard baseline. In addition, we show that there is no statistically significant difference between the results obtained from our simulated testbed and a hardware testbed with identical configurations. Finally we describe lessons learned from our industrial collaboration, offering insights about how simulation helps discover unknown and undocumented behaviours as well as a new perspective on using non-robustness as a measure for system re-configuration.