LookBench: A Live and Holistic Open Benchmark for Fashion Image Retrieval

TL;DR

LookBench is a comprehensive, live benchmark for fashion image retrieval that includes real and AI-generated images, aiming to evaluate and advance models in realistic e-commerce scenarios.

Contribution

The paper introduces LookBench, a new challenging, live, and periodically updated benchmark for fashion image retrieval, with open-source tools and a leaderboard.

Findings

Many models achieve below 60% Recall@1 on LookBench.

The proprietary model outperforms others on LookBench.

Both the proprietary and open-source models set new state-of-the-art on Fashion200K.

Abstract

In this paper, we present LookBench (We use the term "look" to reflect retrieval that mirrors how people shop -- finding the exact item, a close substitute, or a visually consistent alternative.), a live, holistic and challenging benchmark for fashion image retrieval in real e-commerce settings. LookBench includes both recent product images sourced from live websites and AI-generated fashion images, reflecting contemporary trends and use cases. Each test sample is time-stamped and we intend to update the benchmark periodically, enabling contamination-aware evaluation aligned with declared training cutoffs. Grounded in our fine-grained attribute taxonomy, LookBench covers single-item and outfit-level retrieval across. Our experiments reveal that LookBench poses a significant challenge on strong baselines, with many models achieving below $60\%$ Recall@1. Our proprietary model achieves the best performance on LookBench, and we release an open-source counterpart that ranks second, with both models attaining state-of-the-art results on legacy Fashion200K evaluations. LookBench is designed to be updated semi-annually with new test samples and progressively harder task variants, providing a durable measure of progress. We publicly release our leaderboard, dataset, evaluation code, and trained models.