Hotels-50K: A Global Hotel Recognition Dataset

TL;DR

This paper introduces Hotels-50K, a large-scale dataset of over 1 million hotel room images from 50,000 hotels, aimed at improving hotel recognition for human trafficking investigations, and provides baseline methods for this challenging task.

Contribution

The creation of the Hotels-50K dataset, comprising diverse real-world hotel images, and the development of baseline recognition approaches tailored to this domain.

Findings

Dataset contains over 1 million images from 50,000 hotels.

Baseline models achieve initial performance benchmarks.

Data augmentation improves recognition accuracy.

Abstract

Recognizing a hotel from an image of a hotel room is important for human trafficking investigations. Images directly link victims to places and can help verify where victims have been trafficked, and where their traffickers might move them or others in the future. Recognizing the hotel from images is challenging because of low image quality, uncommon camera perspectives, large occlusions (often the victim), and the similarity of objects (e.g., furniture, art, bedding) across different hotel rooms. To support efforts towards this hotel recognition task, we have curated a dataset of over 1 million annotated hotel room images from 50,000 hotels. These images include professionally captured photographs from travel websites and crowd-sourced images from a mobile application, which are more similar to the types of images analyzed in real-world investigations. We present a baseline approach based on a standard network architecture and a collection of data-augmentation approaches tuned to this problem domain.

Tables

Table 1. Table 1: Retrieval results by hotel instance and by hotel chain, reported as top- K 𝐾 K accuracy.

	Instance
	K=1	10	100
Fixed-Object	0.8	0.9	1.3
Fixed-Scene	0.2	0.8	2.4
Ours	8.1	17.6	34.8
	Chain
	K=1	3	5
Fixed-Object	5.0	29.0	79.2
Fixed-Scene	7.2	34.2	78.7
Ours	42.5	56.4	62.8

Table 2. Table 2: Image retrieval comparison reported as top- K 𝐾 K accuracy.

Occlusion:	none			low			medium			high
	K=1	10	100	1	10	100	1	10	100	1	10	100
Fixed-Object	0.8	0.9	1.3	0.3	0.4	0.7	0.0	0.0	0.0	0.0	0.1	0.4
Fixed-Scene	0.2	0.8	2.4	0.1	0.5	1.9	0.1	0.4	1.5	0.0	0.1	1.0
Ours	8.1	17.6	34.8	7.1	16.4	33.1	5.9	14.1	29.9	4.2	10.5	24.0

Table 3. Table 3: Multi-class log loss for each method on the hotel instance classification task.

Occlusion:	none	low	medium	high
Fixed-Object	34.1	34.3	34.5	34.4
Fixed-Scene	33.8	33.9	34.1	34.2
Ours	23.8	24.0	25.4	27.2

Table 4. Table 4: Ablation study reported as top- K 𝐾 K hotel instance retrieval for our method and variants without data augmentation (-A) and without crowdsourced images (-I).

Occlusion:	none			medium
	K=1	10	100	1	10	100
Ours -A,-I	4.7	9.6	20.0	1.8	4.0	9.4
Ours -A	8.1	18.4	36.0	3.5	9.2	12.8
Ours	8.1	17.6	34.8	5.9	14.1	29.9

Full text

Introduction

In recent years, the number of images of victims of human trafficking available online has grown at an alarming rate (?; ?). Whether used for advertising or exchanged among criminal networks, these photographs can serve as visual evidence of where the victim was trafficked. Such images are often captured in hotel rooms. Identifying the hotels in these photographs to understand where a victim was (Figure 1), gives insight into trafficking operations, which is a a top priority for law enforcement (?).

Figure 2 shows a few example of law enforcement queries. Often the region of the images containing the victim is masked for privacy and legal reasons. Algorithms for recognition in this context must be robust to large occlusions, varying lighting conditions, and the unique perspectives of a hotel room.

This paper introduces the Hotels-50K dataset, which includes over 1 million images from 50,000 hotels around the world, designed to support efforts that address this challenging recognition task. Hotels-50K includes both professional photographs from travel websites and crowd-sourced images from a mobile application, which are more similar to the types of images analyzed in real-world investigations.

This domain poses unique challenges compared to generic scene and place recognition tasks. These recognition problems can be grouped based on the specificity of the categories (?):

Basic-level categories (e.g., ‘building’) 2. 2.

Specialized categories (e.g., ‘church’) 3. 3.

Exact instances (e.g., ‘the Notre-Dame’)

The second task (”What type of building is this?”) is often referred to as scene recognition and the third task (”What specific church is this?”) as place recognition. Scene recognition requires learning the shared properties of the examples in the specialized class, while place recognition requires learning the specific components and their configuration that correspond to a particular instance. Hotel recognition does not fit neatly into either task. It requires learning both the general, shared properties of all of the rooms in a particular hotel, such as its decor or star rating, or commonly used color profiles, as well as recognizing duplicated instances of furniture, art and bedding that may be used in different configurations throughout the hotel.

This paper has three main contributions. First, we propose and formulate the problem of hotel instance recognition. Second, we curate and share a data set and evaluation protocol for this problem at a scale that is relevant to international efforts to address trafficking. Third, we describe and test algorithms that include the data augmentation steps necessary to attack this problem as a reasonable baseline for comparisons.

Related Work

Hotels-50k is a large-scale dataset designed to support research in hotel recognition for images with the long term goal of supporting robust applications to aid in criminal investigations. In this section, we review related efforts towards (1) AI to combat human trafficking, (2) targeted large-scale image datasets, and (3) scene and place recognition.

AI to Combat Human Trafficking.

The Hotels-50K dataset and the problem of automatically recognizing hotel rooms fits within a larger set of efforts to apply machine learning, computer vision, and natural language processing to the domain of addressing human trafficking. These efforts largely focused on indexing online escort advertisements, based on locations and phone numbers in the advertisement text or imprinted on advertising images (?; ?; ?; ?). Additionally, there are larger-scale projects, such as Thorn111https://www.wearethorn.org/ that implement approaches including facial identification for identifying victims of child sex trafficking and sexual abuse.

Targeted Large-Scale Image Datasets

The computer vision community has a long tradition of developing datasets to support and challenge the research community. Some of most well-known datasets include ImageNet (?), Places (?), and CIFAR-100 (?). These benchmarks drive competitions for comparing classification and retrieval methods, but because they tend to focus on general (unrelated) categories of images there have been additional efforts towards curating domain-specific datasets, including datasets of classes of cars (?) and birds (?). Most closely related to Hotels-50K are datasets that directly address investigative use-cases, including a database of tattoos (?), and a dataset of advertisements labelled by whether they include a victim of trafficking (?).

Scene and Place Recognition

Recognizing the scene from which an image was captured has been a problem of great interest in the computer vision community. Most work in this area focuses on the problem of identifying the scene category (e.g., park, beach, parking lot) rather than particular locations, but recently there has been increased interest in estimating the precise geographic location of an image.

This place recognition problem can also be formulated as an image retrieval task where geotagged images serve as a database, and a query image’s location is inferred by finding visually similar images in the dataset (?; ?; ?; ?; ?; ?; ?; ?; ?). Increasingly, methods train deep neural networks to produce similar features for images from nearby locations (?; ?; ?; ?; ?).

Algorithms trying recognize a specific place can exploit the fact that the same objects or landmarks appear in the same geometric configuration from different viewpoints. These geometric and matching approaches do not apply to hotel recognition. Within a hotel, the rooms may have some objects that are the same (e.g., every room has the same headboard), some objects that are different (e.g., different artwork on the walls), and those objects may be in different configurations from room to room (e.g., two beds vs. one or furniture on different walls).

Summary

Hotels-50K follows in the tradition of large-scale datasets widely used in the computer vision and machine learning communities. This dataset will support and complement the recent trend for using AI to combat criminal activity, specifically human trafficking. The problem of hotel recognition poses unique challenges and existing methods designed for recognizing outdoor scenes or landmarks are not well-suited to the problem of discriminating between similar-looking hotel rooms.

The Hotels-50K Dataset

Hotels-50K consists of 1,027,871 images from 50,000 unique hotels around the world. Each of the images in the Hotels-50K dataset includes the following metadata: (1) hotel name (2) geographic location, and (3) hotel chain, or Other if the hotel property is not part of a major chain. Figure 3 shows the geographic distribution of the images in our dataset. While the dataset consists of images from around the world, the images are more densely captured in the United States, Western Europe, and coastal regions.

Data Sources

The images in Hotels-50K come from two primary sources: (1) scraped from publicly available travel websites, such as Expedia and (2) captured by the crowdsourcing mobile application, TraffickCam, which allows travelers to submit photos of their hotel room. Figure 4 shows example images from both sources captured at the same hotel. The photos from the travel websites are abundant, accounting for a majority of the images in the dataset. However, these images tend to be taken for promotional purposes, by professional photographers with excellent lighting conditions, of the nicest rooms in a hotel. These images are visually quite different from the types of images referenced in human trafficking investigations. On the other hand, while there are fewer crowdsourced images, these share more visual characteristics with the images used in real-world queries. The crowdsourced images are taken similar devices, at varying orientations, with luggage and other clutter, and without professional lighting.

Dataset Statistics

Of the 50,000 hotel classes in the Hotels-50K training dataset, 13,900 have TraffickCam user-submitted images (a total of 55,061 TraffickCam images are included in the training set). There are no hotels in the dataset that have only TraffickCam images.

Figure 5 show two histograms that characterize the sampling in the dataset. Figure 5(a) shows the number of images per hotel chain for each of the 92 major hotel chains represented in the Hotels-50K dataset. Some chains have many more images than others (Holiday Inn, Hampton and Best Western), consistent with the prevalence of those hotel chains around the world. Figure 5(b) shows a histogram of the number of images per hotel broken down by the source of images (travel websites or TraffickCam mobile application). The average number of images from travel websites per hotel is $19.5$. The average number of images from TraffickCam for the hotels with TraffickCam images is $4.0$.

Observations

While there exist discriminative patterns and unique features visible in the images from the hotels in Hotels-50K, this dataset highlights one of the main challenges in hotel recognition. There can be high intraclass variation, as not every room within a single hotel will have the same shared properties or objects – some rooms contain more amenities and some may have been renovated. On the other hand, there can be low interclass variation, especially from hotels of the same chain, making the recognition of a specific hotel difficult. Figure 6 shows a few specific examples where two rooms in the same hotel look much more different than rooms in two different hotels from the same chain.

Evaluation Protocol

Hotels-50K includes a separate test set of images to support the consistent evaluation of algorithms. Obtaining a large collection of images from real-world investigations is problematic for many reasons. However, in the images in the test set are meant to replicate, as closely as possible the type of data used in these cases.

The test set consists of 17,954 images from the TraffickCam mobile application from 5,000 different hotels, which are a subset of those found in the training set. There is no overlap in the mobile app users between the training and testing sets to avoid the case of near duplicates due to multiple images from the same user with the same device captured at the same time.

To replicate real-world conditions where the regions of the image containing victims are masked prior to image analysis, the images are augmented with increasingly larger ”people-shaped” masks. The masks are generated using silhouettes from ’people’ regions in the MS-COCO semantic labels dataset (?). There are four levels of masking (none, low, medium high), corresponding to the relative sizes of the masked region in each image, where the largest masks can occupy up to 85% of the height of the image. Figure 7 shows examples of masked test images.

The evaluation consists of the following tasks:

Hotel Instance Recognition

The goal for this task is to identify the hotel instance represented for each of the images in the test set.

Hotel Chain Recognition

The goal for this task is to identify the hotel chain represented in the image. Of the test set, 13,136 images are from one of 88 major hotel chains, with the remainder in the ”Other” category.

Evaluation Metrics

Hotel recognition can be framed as both a classification task (i.e., predict the label given the image) and a retrieval task (i.e., find the most similar database images to a query). The evaluation suite for Hotels-50K supports both variants.

For the retrieval variant, the results should be provided as a ranked list of the IDs of the 100 most similar images from the Hotels-50K dataset to each of the test images. The evaluation metric is top-$K$ accuracy, with $K=\{1,10,100\}$ for hotel instance recognition and $K=\{1,3,5\}$ for hotel chain recognition.

For the classification variant, the results should be provided as the posterior probabilities of hotel chains or instances for each of the test images. The evaluation metrics include the average multi-class log loss (lower is better) and top-$K$ classification accuracy with $K=\{1,10,100\}$ for hotel instance recognition and $K=\{1,3,5\}$ for hotel chain recognition.

Results

In order to set the baseline for performance on the Hotels-50K dataset, we compare two ”off-the-shelf” pre-trained networks trained for object and scene recognition to a method using data and augmentation schemes specifically tailored to hotel recognition.

Models

For the pretrained models, we use the fixed feature representations and refer to these as the Fixed-Object and Fixed-Scene methods. The Fixed-Object method is a Resnet-50 network trained on ImageNet (ILSVRC-2012) (?; ?; ?). The feature representation is the 1001-dimensional output from the final fully connected layer. The Fixed-Scene method uses a VGG model trained on the Places365 dataset (?). The feature representation is the 512-dimensional output of the final pooling layer.

Our method uses the Hotels-50K training set as input to fine tune a Resnet-50 model, pre-trained for ImageNet, to output 256-D features. The training scheme is the combinatorial variant of triplet loss described in (?).

In training, we balance the number of crowdsourced and travel website images in each batch. Additionally, we perform a set of data augmentation steps, highlighted in Figure 8. Images from the batch are randomly selected and rotated between -35 and 35 degrees, cropped between 60% and 100% of the original size, modified with color and brightness, and masked with person shaped silhouettes, similar to process used for the test data. The set of masks applied in training do not overlap with those used to generated the Hotels-50K test data and will be made available. Training parameters were selected using cross-validation. The final model was fine-tuned for 65,000 iterations with 120 images per batch.

Retrieval

For retrieval, we compute feature representations for all of the images in the Hotels-50K training set using each method. Feature representations are also computed for each image in the test set, and the database images are ranked by cosine similarity to each test image.

Table 1 shows the image retrieval results by hotel instance and chain for all three methods. For all methods, the retrieval accuracy by hotel instance is significantly lower than the accuracy by hotel chain. This is likely due to the difficulty discriminating between particular instances of hotel chains that look similar. The chain identification task is simple enough that even the fixed methods not fine-tuned to the task achieve nearly 80% top-5 accuracy on this task. Therefore, for our remaining experiments, we focus on the more challenging problem to recognize a hotel instance.

Table 2 shows the image retrieval results for all three methods for the test images with varying sizes of image masking. Our approach has significantly higher retrieval accuracy compared to the pre-trained approaches for all tests, both with and without occlusions.

Figure 9 shows the top 5 results for several query images using Fixed-Object, Fixed-Scene and our approaches. Unlike Fixed-Object and Fixed-Scene, our model appears to encode information about the important colors and objects in a hotel room. In the top example in Figure 9, our approach finds examples from the correct hotel, as well as other images with similar blue walls and headboards. Our model also performs reasonably well even in the case where there is large amounts of clutter in the query image, as seen in the middle example in Figure 9. The last example in Figure 9 highlights the difficulty of hotel instance recognition given the similarity between instances of the same hotel chain – nearly all of the top images retrieved by our model are from the correct hotel chain, but not necessarily the correct hotel.

Classification

For the classification task, we adapt the image embedding approaches used for image retrieval to report class posterior probabilities. For each method for each test image, we find the 1000 most similar images in the database using cosine similarity between the output features. The proportion of each class (hotel instance or hotel chain) in the resulting set is the estimate of the posterior probability.

Table 3 shows the multiclass log loss for each method for varying levels of occlusions in the test images. In all cases, our approach outperforms features from the pretrained models. However, there is still significant room for improved classification performance.

Ablation Study

To quantify the effects of both the inclusion of the crowdsourced data and the augmentation steps in our approach, we compare the results of variants of our method on the hotel instance retrieval task with and without significant occlusions.

Table 4 shows the results for the ablation experiment. We evaluate our approach without the data augmentation steps and additionally without including the crowdsourced images, which are those most similar to the real-world images. The inclusion of the crowdsourced images has a significant impact on the performance both with and without occlusions in the test image. The data augmentation steps do not have an impact on the performance in the un-occluded cases, but in the medium occlusion case, which roughly corresponds to sizes of the masked regions in real-world cases, the benefits of the data augmentation steps are apparent, increasing the top-$K$ accuracy by more than 50% for $K=10$.

Conclusion

In this paper, we introduced Hotels-50K, a dataset of over a million images of hotel rooms from 50,000 different hotels around the world. This dataset should further the state of the art in hotel recognition from images. We present an approach trained on the Hotels-50K dataset that outperforms fixed features from generic object and scene models. The Hotels-50K dataset, pre-trained models and code to replicate our baseline approaches can be found at https://github.com/GWUvision/Hotels-50K. The baseline approach is currently deployed for use by human trafficking investigators, including the National Center for Missing and Exploited Children, and novel algorithms can be quickly deployed to improve search performance in ongoing investigations.

This project is based in part on work supported through the National Institute of Justice (Grant 2018-75-CX-0038) and a gift from Adobe Inc.