Applied Awareness: Test-Driven GUI Development using Computer Vision and Cryptography

TL;DR

This paper introduces a novel approach for GUI testing that interprets presentation in terms of backend communications, leveraging cryptographic awareness to enable test-driven development without initial implementations or golden images.

Contribution

It presents a practical, cryptography-inspired method for GUI testing that avoids platform-dependent dependencies and supports both online and offline validation.

Findings

Effective GUI testing without initial implementation

Validation with simple designer specifications

Potential for formal verification and AI interpretability

Abstract

Graphical user interface testing is significantly challenging, and automating it even more so. Test-driven development is impractical: it generally requires an initial implementation of the GUI to generate golden images or to construct interactive test scenarios, and subsequent maintenance is costly. While computer vision has been applied to several aspects of GUI testing, we demonstrate a novel and immediately applicable approach of interpreting GUI presentation in terms of backend communications, modeling "awareness" in the fashion employed by cryptographic proofs of security. This focus on backend communication circumvents deficiencies in typical testing methodologies that rely on platform-dependent UI affordances or accessibility features. Our interdisciplinary work is ready for off-the-shelf practice: we report self-contained, practical implementation with both online and offline validation, using simple designer specifications at the outset and specifically avoiding any requirements for a bootstrap implementation or golden images. In addition to practical implementation, ties to formal verification methods in cryptography are explored and explained, providing fertile perspectives on assurance in UI and interpretability in AI.