Computational Approaches for App-to-App Retrieval and Design Consistency Check

TL;DR

This paper introduces scalable, open-source visual models trained on web-scale images for app-to-app UI retrieval and design consistency analysis, outperforming existing methods and enabling new applications.

Contribution

It presents a novel approach using large-scale visual models for zero-shot UI representation and app-to-app retrieval, addressing limitations of prior small dataset methods.

Findings

Improved retrieval accuracy over previous models

Enabled app-to-app retrieval and design consistency analysis

Demonstrated effectiveness on large-scale datasets

Abstract

Extracting semantic representations from mobile user interfaces (UI) and using the representations for designers' decision-making processes have shown the potential to be effective computational design support tools. Current approaches rely on machine learning models trained on small-sized mobile UI datasets to extract semantic vectors and use screenshot-to-screenshot comparison to retrieve similar-looking UIs given query screenshots. However, the usability of these methods is limited because they are often not open-sourced and have complex training pipelines for practitioners to follow, and are unable to perform screenshot set-to-set (i.e., app-to-app) retrieval. To this end, we (1) employ visual models trained with large web-scale images and test whether they could extract a UI representation in a zero-shot way and outperform existing specialized models, and (2) use mathematically founded methods to enable app-to-app retrieval and design consistency analysis. Our experiments show that our methods not only improve upon previous retrieval models but also enable multiple new applications.