Artificial intelligence for context-aware visual change detection in software test automation
Milad Moradi, Ke Yan, David Colwell, Rhona Asgari

TL;DR
This paper introduces a graph-based, machine learning-driven method for context-aware visual change detection in software UI testing, outperforming traditional pixel and region comparison techniques by capturing structural and contextual UI relationships.
Contribution
It presents a novel approach combining UI control detection with graph modeling and recursive similarity computation for robust change detection.
Findings
Outperforms pixel-wise and region-based baselines in detecting UI changes
Effectively captures contextual and structural UI relationships
Reliable detection of simple and complex UI modifications
Abstract
Automated software testing is integral to the software development process, streamlining workflows and ensuring product reliability. Visual testing, particularly for user interface (UI) and user experience (UX) validation, plays a vital role in maintaining software quality. However, conventional techniques such as pixel-wise comparison and region-based visual change detection often fail to capture contextual similarities, subtle variations, and spatial relationships between UI elements. In this paper, we propose a novel graph-based approach for context-aware visual change detection in software test automation. Our method leverages a machine learning model (YOLOv5) to detect UI controls from software screenshots and constructs a graph that models their contextual and spatial relationships. This graph structure is then used to identify correspondences between UI elements across software…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAnomaly Detection Techniques and Applications · Industrial Vision Systems and Defect Detection · Software System Performance and Reliability
