FORB: A Flat Object Retrieval Benchmark for Universal Image Embedding

TL;DR

This paper introduces FORB, a new benchmark dataset for flat object image retrieval, addressing the limitations of existing datasets by evaluating retrieval methods on diverse 2D flat objects and out-of-distribution domains.

Contribution

The paper presents a novel flat object retrieval benchmark (FORB) that expands evaluation beyond 3D landmarks to include diverse 2D flat objects, facilitating better assessment of image embedding quality.

Findings

Retrieval accuracy varies significantly across different methods.

Matching score margin provides additional insights into retrieval performance.

The benchmark reveals challenges and heterogeneity in flat object retrieval.

Abstract

Image retrieval is a fundamental task in computer vision. Despite recent advances in this field, many techniques have been evaluated on a limited number of domains, with a small number of instance categories. Notably, most existing works only consider domains like 3D landmarks, making it difficult to generalize the conclusions made by these works to other domains, e.g., logo and other 2D flat objects. To bridge this gap, we introduce a new dataset for benchmarking visual search methods on flat images with diverse patterns. Our flat object retrieval benchmark (FORB) supplements the commonly adopted 3D object domain, and more importantly, it serves as a testbed for assessing the image embedding quality on out-of-distribution domains. In this benchmark we investigate the retrieval accuracy of representative methods in terms of candidate ranks, as well as matching score margin, a viewpoint which is largely ignored by many works. Our experiments not only highlight the challenges and rich heterogeneity of FORB, but also reveal the hidden properties of different retrieval strategies. The proposed benchmark is a growing project and we expect to expand in both quantity and variety of objects. The dataset and supporting codes are available at https://github.com/pxiangwu/FORB/.