Revisiting Hotels-50K and Hotel-ID

TL;DR

This paper revisits the Hotels50K and Hotel-ID datasets by proposing new evaluation setups that better reflect real-world challenges like unseen hotel classes, revealing performance drops of state-of-the-art models.

Contribution

It introduces revised versions of two hotel recognition datasets with varied difficulty levels to improve real-world applicability and evaluation relevance.

Findings

Model performance decreases in more realistic unseen settings.

Model rankings change across different evaluation setups.

Revisited datasets better simulate real-world hotel recognition challenges.

Abstract

In this paper, we propose revisited versions for two recent hotel recognition datasets: Hotels50K and Hotel-ID. The revisited versions provide evaluation setups with different levels of difficulty to better align with the intended real-world application, i.e. countering human trafficking. Real-world scenarios involve hotels and locations that are not captured in the current data sets, therefore it is important to consider evaluation settings where classes are truly unseen. We test this setup using multiple state-of-the-art image retrieval models and show that as expected, the models' performances decrease as the evaluation gets closer to the real-world unseen settings. The rankings of the best performing models also change across the different evaluation settings, which further motivates using the proposed revisited datasets.