Scene Recognition with Prototype-agnostic Scene Layout

TL;DR

This paper introduces a novel prototype-agnostic scene layout method and a graph network to model diverse spatial structures in scene images, achieving state-of-the-art recognition results.

Contribution

It proposes a flexible scene layout construction method independent of prototypes and a graph network to incorporate spatial and semantic relations for improved scene recognition.

Findings

Achieves state-of-the-art results on MIT67 and SUN397 datasets.

Demonstrates strong generalization on Places365 dataset.

Outperforms existing methods without multi-scale fusion.

Abstract

Abstract--- Exploiting the spatial structure in scene images is a key research direction for scene recognition. Due to the large intra-class structural diversity, building and modeling flexible structural layout to adapt various image characteristics is a challenge. Existing structural modeling methods in scene recognition either focus on predefined grids or rely on learned prototypes, which all have limited representative ability. In this paper, we propose Prototype-agnostic Scene Layout (PaSL) construction method to build the spatial structure for each image without conforming to any prototype. Our PaSL can flexibly capture the diverse spatial characteristic of scene images and have considerable generalization capability. Given a PaSL, we build Layout Graph Network (LGN) where regions in PaSL are defined as nodes and two kinds of independent relations between regions are encoded as edges. The LGN aims to incorporate two topological structures (formed in spatial and semantic similarity dimensions) into image representations through graph convolution. Extensive experiments show that our approach achieves state-of-the-art results on widely recognized MIT67 and SUN397 datasets without multi-model or multi-scale fusion. Moreover, we also conduct the experiments on one of the largest scale datasets, Places365. The results demonstrate the proposed method can be well generalized and obtains competitive performance.