To What Extent Is Stress Testing of Android TV Applications Automated in Industrial Environments?

TL;DR

This study investigates the extent to which stress testing of Android TV applications can be automated in industrial settings, highlighting the importance of high-level GUI controls and resource setup for automation success.

Contribution

It provides the first empirical analysis of automation support in stress testing Android TV applications in real industrial environments.

Findings

87% of test specifications can be fully automated with support

71.4% of tests are reusable across hardware and OS upgrades

Automation support improves stress testing efficiency by 28-38%

Abstract

An Android-based smart Television (TV) must reliably run its applications in an embedded program environment under diverse hardware resource conditions. Owing to the diverse hardware components used to build numerous TV models, TV simulators are usually not high enough in fidelity to simulate various TV models, and thus are only regarded as unreliable alternatives when stress testing such applications. Therefore, even though stress testing on real TV sets is tedious, it is the de facto approach to ensure the reliability of these applications in the industry. In this paper, we study to what extent stress testing of smart TV applications can be fully automated in the industrial environments. To the best of our knowledge, no previous work has addressed this important question. We summarize the find-ings collected from 10 industrial test engineers to have tested 20 such TV applications in a real production environment. Our study shows that the industry required test automation supports on high-level GUI object controls and status checking, setup of resource conditions and the interplay between the two. With such supports, 87% of the industrial test specifications of one TV model can be fully automated and 71.4% of them were found to be fully reusable to test a subsequent TV model with major up-grades of hardware, operating system and application. It repre-sents a significant improvement with margins of 28% and 38%, respectively, compared to stress testing without such supports.