Estimating Commonsense Scene Composition on Belief Scene Graphs

TL;DR

This paper introduces a framework for commonsense scene composition by estimating the spatial distribution of unseen objects within Belief Scene Graphs, combining graph neural networks and LLM-based ontologies.

Contribution

It proposes two variants of a Correlation Information model for learning spatial probability distributions, integrating neuro-symbolic methods with LLMs and providing a dataset for evaluation.

Findings

Effective spatial interpretation in simulated environments

Successful application in real-world indoor scenes

Demonstrated ability to model unseen object locations

Abstract

This work establishes the concept of commonsense scene composition, with a focus on extending Belief Scene Graphs by estimating the spatial distribution of unseen objects. Specifically, the commonsense scene composition capability refers to the understanding of the spatial relationships among related objects in the scene, which in this article is modeled as a joint probability distribution for all possible locations of the semantic object class. The proposed framework includes two variants of a Correlation Information (CECI) model for learning probability distributions: (i) a baseline approach based on a Graph Convolutional Network, and (ii) a neuro-symbolic extension that integrates a spatial ontology based on Large Language Models (LLMs). Furthermore, this article provides a detailed description of the dataset generation process for such tasks. Finally, the framework has been validated through multiple runs on simulated data, as well as in a real-world indoor environment, demonstrating its ability to spatially interpret scenes across different room types.