GUI Element Detection Using SOTA YOLO Deep Learning Models

TL;DR

This paper evaluates the performance of the latest YOLO deep learning models specifically for detecting GUI elements, addressing domain-specific challenges like overlapping objects and limited classes.

Contribution

It provides a comparative analysis of recent YOLO models' effectiveness in GUI element detection, a domain-specific application of object detection.

Findings

YOLO models achieve high accuracy in GUI element detection

Performance varies depending on GUI element types

Deep learning models outperform traditional image processing methods

Abstract

Detection of Graphical User Interface (GUI) elements is a crucial task for automatic code generation from images and sketches, GUI testing, and GUI search. Recent studies have leveraged both old-fashioned and modern computer vision (CV) techniques. Oldfashioned methods utilize classic image processing algorithms (e.g. edge detection and contour detection) and modern methods use mature deep learning solutions for general object detection tasks. GUI element detection, however, is a domain-specific case of object detection, in which objects overlap more often, and are located very close to each other, plus the number of object classes is considerably lower, yet there are more objects in the images compared to natural images. Hence, the studies that have been carried out on comparing various object detection models, might not apply to GUI element detection. In this study, we evaluate the performance of the four most recent successful YOLO models for general object detection tasks on GUI element detection and investigate their accuracy performance in detecting various GUI elements.