Believing is Seeing: Unobserved Object Detection using Generative Models

TL;DR

This paper explores the novel task of detecting objects that are nearby but not visible in images, using adapted generative models to infer their presence and location, with promising results on indoor scene datasets.

Contribution

It introduces unobserved object detection as a new task and adapts state-of-the-art generative models to address it, providing a benchmark and evaluation metrics.

Findings

Generative models can infer unobserved objects in images.

Proposed metrics effectively evaluate unobserved object detection.

Empirical results show potential for generative models in this task.

Abstract

Can objects that are not visible in an image -- but are in the vicinity of the camera -- be detected? This study introduces the novel tasks of 2D, 2.5D and 3D unobserved object detection for predicting the location of nearby objects that are occluded or lie outside the image frame. We adapt several state-of-the-art pre-trained generative models to address this task, including 2D and 3D diffusion models and vision-language models, and show that they can be used to infer the presence of objects that are not directly observed. To benchmark this task, we propose a suite of metrics that capture different aspects of performance. Our empirical evaluation on indoor scenes from the RealEstate10k and NYU Depth v2 datasets demonstrate results that motivate the use of generative models for the unobserved object detection task.