2.5D Visual Relationship Detection

TL;DR

This paper introduces 2.5D visual relationship detection, a new task that combines object detection with relative depth and occlusion understanding from an egocentric viewpoint, supported by a large annotated dataset.

Contribution

The paper presents the first dataset for 2.5D visual relationship detection and benchmarks existing models, highlighting the need for specialized approaches beyond semantic cues.

Findings

Existing models rely heavily on semantic cues.

Current models perform poorly on 2.5D relationships.

The dataset enables future research in 2.5D perception.

Abstract

Visual 2.5D perception involves understanding the semantics and geometry of a scene through reasoning about object relationships with respect to the viewer in an environment. However, existing works in visual recognition primarily focus on the semantics. To bridge this gap, we study 2.5D visual relationship detection (2.5VRD), in which the goal is to jointly detect objects and predict their relative depth and occlusion relationships. Unlike general VRD, 2.5VRD is egocentric, using the camera's viewpoint as a common reference for all 2.5D relationships. Unlike depth estimation, 2.5VRD is object-centric and not only focuses on depth. To enable progress on this task, we create a new dataset consisting of 220k human-annotated 2.5D relationships among 512K objects from 11K images. We analyze this dataset and conduct extensive experiments including benchmarking multiple state-of-the-art VRD models on this task. Our results show that existing models largely rely on semantic cues and simple heuristics to solve 2.5VRD, motivating further research on models for 2.5D perception. The new dataset is available at https://github.com/google-research-datasets/2.5vrd.