Scene Retrieval for Contextual Visual Mapping

TL;DR

This paper introduces a novel scene retrieval approach using a CNN and a new algorithm to improve visual mapping accuracy by better classifying scene types and selectively including images based on scene context.

Contribution

It defines the problem of scene retrieval, proposes a CNN trained with triplet loss for scene classification, and introduces the DMC algorithm to enhance visual mapping with scene context.

Findings

Scene retrieval improves scene classification accuracy by up to 7%.

DMC increases inclusion of scene-specific images by 64%.

DMC enhances localization accuracy by 3-10% across datasets.

Abstract

Visual navigation localizes a query place image against a reference database of place images, also known as a `visual map'. Localization accuracy requirements for specific areas of the visual map, `scene classes', vary according to the context of the environment and task. State-of-the-art visual mapping is unable to reflect these requirements by explicitly targetting scene classes for inclusion in the map. Four different scene classes, including pedestrian crossings and stations, are identified in each of the Nordland and St. Lucia datasets. Instead of re-training separate scene classifiers which struggle with these overlapping scene classes we make our first contribution: defining the problem of `scene retrieval'. Scene retrieval extends image retrieval to classification of scenes defined at test time by associating a single query image to reference images of scene classes. Our second contribution is a triplet-trained convolutional neural network (CNN) to address this problem which increases scene classification accuracy by up to 7% against state-of-the-art networks pre-trained for scene recognition. The second contribution is an algorithm `DMC' that combines our scene classification with distance and memorability for visual mapping. Our analysis shows that DMC includes 64% more images of our chosen scene classes in a visual map than just using distance interval mapping. State-of-the-art visual place descriptors AMOS-Net, Hybrid-Net and NetVLAD are finally used to show that DMC improves scene class localization accuracy by a mean of 3% and localization accuracy of the remaining map images by a mean of 10% across both datasets.