GUI-ReRank: Enhancing GUI Retrieval with Multi-Modal LLM-based Reranking
Kristian Kolthoff, Felix Kretzer, Christian Bartelt, Alexander Maedche, Simone Paolo Ponzetto

TL;DR
GUI-ReRank introduces a multi-modal LLM-based reranking framework that significantly improves GUI retrieval accuracy and generalizability, enabling customizable, efficient GUI repository searches for interactive system development.
Contribution
The paper presents GUI-ReRank, a novel framework combining embedding retrieval with MLLM reranking, and offers a customizable pipeline for GUI repository annotation and embedding.
Findings
Outperforms state-of-the-art LTR models in retrieval accuracy
Demonstrates improved generalizability across GUI datasets
Provides a cost and efficiency analysis of MLLM reranking
Abstract
GUI prototyping is a fundamental component in the development of modern interactive systems, which are now ubiquitous across diverse application domains. GUI prototypes play a critical role in requirements elicitation by enabling stakeholders to visualize, assess, and refine system concepts collaboratively. Moreover, prototypes serve as effective tools for early testing, iterative evaluation, and validation of design ideas with both end users and development teams. Despite these advantages, the process of constructing GUI prototypes remains resource-intensive and time-consuming, frequently demanding substantial effort and expertise. Recent research has sought to alleviate this burden through NL-based GUI retrieval approaches, which typically rely on embedding-based retrieval or tailored ranking models for specific GUI repositories. However, these methods often suffer from limited…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
