MUIAnno: An Expert-Annotated Dataset and Evaluation Benchmark for Mobile UI Understanding

TL;DR

MUIAnno is a comprehensive, expert-annotated dataset of mobile UI screens designed to advance understanding and detection of interface elements in real-world applications, supported by benchmark experiments.

Contribution

The paper introduces MUIAnno, a high-quality, expert-annotated dataset of mobile UI screens from diverse apps, along with benchmark results for UI element detection.

Findings

MUIAnno contains detailed annotations of common UI components.

Benchmark experiments establish baseline UI element detection performance.

The dataset reflects real-world mobile application layouts and design patterns.

Abstract

Understanding mobile user interfaces is important for building intelligent systems such as automation tools, accessibility solutions, and UI-aware agents. However, progress in this area is still limited by the lack of high-quality datasets that reflect real-world mobile applications and include reliable annotations. In this work, we introduce MUIAnno, a publicly available expert-annotated dataset for mobile UI understanding, collected from a diverse set of applications across multiple categories available on the iTunes platform. Each app was manually explored to capture representative UI screens, resulting in a collection that reflects a wide range of layouts and design patterns found in practice. To ensure annotation quality, we developed a custom web-based tool that allows UI/UX experts to label interface elements through a simple drag-and-drop process and generate structured annotations in JSON format. MUIAnno includes detailed annotations of common UI components such as buttons, input fields, navigation elements, and other key interface elements. In addition to presenting the dataset, we also provide benchmark experiments for UI element detection along with baseline results, offering a starting point for future research. We believe MUIAnno can support further work in mobile UI understanding and help improve systems that rely on accurate interpretation of interface elements.