VINS: Visual Search for Mobile User Interface Design

TL;DR

VINS is a visual search framework that enables retrieval of similar mobile UI designs based on structural and visual features, aiding designers in inspiration and comparison.

Contribution

Introduces VINS, a novel UI image retrieval system that incorporates UI structure and content for more accurate search results.

Findings

Achieves 76.39% mean Average Precision in UI detection

Develops a large-scale UI dataset with detailed view hierarchy

Demonstrates high performance in retrieving similar UI designs

Abstract

Searching for relative mobile user interface (UI) design examples can aid interface designers in gaining inspiration and comparing design alternatives. However, finding such design examples is challenging, especially as current search systems rely on only text-based queries and do not consider the UI structure and content into account. This paper introduces VINS, a visual search framework, that takes as input a UI image (wireframe, high-fidelity) and retrieves visually similar design examples. We first survey interface designers to better understand their example finding process. We then develop a large-scale UI dataset that provides an accurate specification of the interface's view hierarchy (i.e., all the UI components and their specific location). By utilizing this dataset, we propose an object-detection based image retrieval framework that models the UI context and hierarchical structure. The framework achieves a mean Average Precision of 76.39\% for the UI detection and high performance in querying similar UI designs.