Hawkeye: Change-targeted Testing for Android Apps based on Deep Reinforcement Learning

TL;DR

Hawkeye is a deep reinforcement learning-based tool that prioritizes testing of code changes in Android apps, improving efficiency and effectiveness over existing methods, especially for large and complex applications.

Contribution

This paper introduces Hawkeye, a novel directed testing approach using deep reinforcement learning to focus on changed app components, outperforming existing tools in targeted GUI event generation.

Findings

Hawkeye reliably targets changed functions in testing.

Hawkeye performs comparably on small apps and better on large apps.

Industrial deployment shows Hawkeye is effective for smoke testing in real pipelines.

Abstract

Android Apps are frequently updated to keep up with changing user, hardware, and business demands. Ensuring the correctness of App updates through extensive testing is crucial to avoid potential bugs reaching the end user. Existing Android testing tools generate GUI events focussing on improving the test coverage of the entire App rather than prioritising updates and its impacted elements. Recent research has proposed change-focused testing but relies on random exploration to exercise the updates and impacted GUI elements that is ineffective and slow for large complex Apps with a huge input exploration space. We propose directed testing of App updates with Hawkeye that is able to prioritise executing GUI actions associated with code changes based on deep reinforcement learning from historical exploration data. Our empirical evaluation compares Hawkeye with state-of-the-art model-based and reinforcement learning-based testing tools FastBot2 and ARES using 10 popular open-source and 1 commercial App. We find that Hawkeye is able to generate GUI event sequences targeting changed functions more reliably than FastBot2 and ARES for the open source Apps and the large commercial App. Hawkeye achieves comparable performance on smaller open source Apps with a more tractable exploration space. The industrial deployment of Hawkeye in the development pipeline also shows that Hawkeye is ideal to perform smoke testing for merge requests of a complicated commercial App.