Scene recognition with CNNs: objects, scales and dataset bias

TL;DR

This paper investigates how scale-induced dataset bias affects scene recognition with CNNs and proposes scale-specific CNNs to effectively combine scene and object knowledge, significantly improving accuracy.

Contribution

It introduces scale-specific CNNs to address dataset bias and demonstrates improved scene recognition by combining scene-centric and object-centric CNNs at multiple scales.

Findings

Scale-specific CNNs outperform single-scale models.

Multi-scale combinations improve recognition accuracy.

Achieved up to 70.17% accuracy on SUN397 dataset.

Abstract

Since scenes are composed in part of objects, accurate recognition of scenes requires knowledge about both scenes and objects. In this paper we address two related problems: 1) scale induced dataset bias in multi-scale convolutional neural network (CNN) architectures, and 2) how to combine effectively scene-centric and object-centric knowledge (i.e. Places and ImageNet) in CNNs. An earlier attempt, Hybrid-CNN, showed that incorporating ImageNet did not help much. Here we propose an alternative method taking the scale into account, resulting in significant recognition gains. By analyzing the response of ImageNet-CNNs and Places-CNNs at different scales we find that both operate in different scale ranges, so using the same network for all the scales induces dataset bias resulting in limited performance. Thus, adapting the feature extractor to each particular scale (i.e. scale-specific CNNs) is crucial to improve recognition, since the objects in the scenes have their specific range of scales. Experimental results show that the recognition accuracy highly depends on the scale, and that simple yet carefully chosen multi-scale combinations of ImageNet-CNNs and Places-CNNs, can push the state-of-the-art recognition accuracy in SUN397 up to 66.26% (and even 70.17% with deeper architectures, comparable to human performance).