Semantic GUI Scene Learning and Video Alignment for Detecting Duplicate Video-based Bug Reports

TL;DR

This paper presents JANUS, a novel approach using vision transformers and video alignment to improve duplicate detection in video-based GUI bug reports, significantly outperforming prior methods.

Contribution

JANUS introduces scene-learning with vision transformers and adaptive video frame alignment for more accurate duplicate detection in GUI bug videos.

Findings

Achieves 89.8% mRR and 84.7% mAP on benchmark

Outperforms prior work by around 9% in duplicate detection accuracy

Effectively captures subtle visual and textual patterns in GUI videos

Abstract

Video-based bug reports are increasingly being used to document bugs for programs centered around a graphical user interface (GUI). However, developing automated techniques to manage video-based reports is challenging as it requires identifying and understanding often nuanced visual patterns that capture key information about a reported bug. In this paper, we aim to overcome these challenges by advancing the bug report management task of duplicate detection for video-based reports. To this end, we introduce a new approach, called JANUS, that adapts the scene-learning capabilities of vision transformers to capture subtle visual and textual patterns that manifest on app UI screens - which is key to differentiating between similar screens for accurate duplicate report detection. JANUS also makes use of a video alignment technique capable of adaptive weighting of video frames to account for typical bug manifestation patterns. In a comprehensive evaluation on a benchmark containing 7,290 duplicate detection tasks derived from 270 video-based bug reports from 90 Android app bugs, the best configuration of our approach achieves an overall mRR/mAP of 89.8%/84.7%, and for the large majority of duplicate detection tasks, outperforms prior work by around 9% to a statistically significant degree. Finally, we qualitatively illustrate how the scene-learning capabilities provided by Janus benefits its performance.