PSDoodle: Fast App Screen Search via Partial Screen Doodle

TL;DR

PSDoodle enables fast, interactive search of mobile app screens using partial sketches, significantly reducing search time while maintaining accuracy, thus improving the efficiency of app design retrieval.

Contribution

First system to enable interactive partial sketch-based search for app screens, combining multiple datasets for effective retrieval.

Findings

Achieved similar top-10 accuracy as state-of-the-art methods.

Reduced average search time by about 50%.

Validated with third-party software developers.

Abstract

Searching through existing repositories for a specific mobile app screen design is currently either slow or tedious. Such searches are either limited to basic keyword searches (Google Image Search) or require as input a complete query screen image (SWIRE). A promising alternative is interactive partial sketching, which is more structured than keyword search and faster than complete-screen queries. PSDoodle is the first system to allow interactive search of screens via interactive sketching. PSDoodle is built on top of a combination of the Rico repository of some 58k Android app screens, the Google QuickDraw dataset of icon-level doodles, and DoodleUINet, a curated corpus of some 10k app icon doodles collected from hundreds of individuals. In our evaluation with third-party software developers, PSDoodle provided similar top-10 screen retrieval accuracy as the state of the art from the SWIRE line of work, while cutting the average time required about in half.