Searching Mobile App Screens via Text + Doodle

TL;DR

The paper introduces TpD, a novel system that combines interactive sketching and text queries to improve the accuracy and efficiency of searching mobile app screens from large repositories.

Contribution

TpD is the first system to enable iterative screen search by integrating sketch-based and keyword search methods, leveraging a large Android app screen dataset.

Findings

PSDoodle outperforms Swire in top-10 retrieval accuracy.

PSDoodle requires less query time than existing interactive solutions.

TpD effectively combines sketching and text for improved search results.

Abstract

Locating a specific mobile application screen from existing repositories is restricted to basic keyword searches, such as Google Image Search, or necessitates a complete query screen image, as in the case of Swire. However, interactive partial sketch-based solutions like PSDoodle have limitations, including inaccuracy and an inability to consider text appearing on the screen. A potentially effective solution involves implementing a system that provides interactive partial sketching functionality for efficiently structuring user interface elements. Additionally, the system should incorporate text queries to enhance its capabilities further. Our approach, TpD, represents the pioneering effort to enable an iterative search of screens by combining interactive sketching and keyword search techniques. TpD is built on a combination of the Rico repository of approximately 58k Android app screens and the PSDoodle. Our evaluation with third-party software developers showed that PSDoodle provided higher top-10 screen retrieval accuracy than state-of-the-art Swire and required less time to complete a query than other interactive solutions.