What do all these Buttons do? Statically Mining Android User Interfaces at Scale

TL;DR

FRONTMATTER is a static analysis tool that automatically extracts detailed UI models and behaviors from Android apps at scale, enabling large-scale empirical research on mobile interfaces.

Contribution

It introduces a novel static analysis approach for comprehensive UI and API extraction from thousands of Android apps, facilitating large-scale UI mining.

Findings

Enabled analysis of tens of thousands of apps

Identified UI and API usage patterns at scale

Facilitated research on accessibility and registration requirements

Abstract

We introduce FRONTMATTER: a tool to automatically mine both user interface models and behavior of Android apps at a large scale with high precision. Given an app, FRONTMATTER statically extracts all declared screens, the user interface elements, their textual and graphical features, as well as Android APIs invoked by interacting with them. Executed on tens of thousands of real-world apps, FRONTMATTER opens the door for comprehensive mining of mobile user interfaces, jumpstarting empirical research at a large scale, addressing questions such as "How many travel apps require registration?", "Which apps do not follow accessibility guidelines?", "Does the user interface correspond to the description?", and many more. FRONTMATTER and the mined dataset are available under an open-source license.