Leveraging commonsense for object localisation in partial scenes

TL;DR

This paper introduces a novel graph-based scene representation called D-SCG, enriched with commonsense knowledge, to improve object localisation in partial 3D scenes using a graph neural network with attentional message passing.

Contribution

The paper presents a new scene graph model incorporating commonsense knowledge for better geometric reasoning in object localisation tasks.

Findings

Achieved 5.9% higher localisation accuracy on Partial ScanNet

Faster training speed by a factor of 8x compared to previous methods

Demonstrated effectiveness of commonsense integration in scene understanding

Abstract

We propose an end-to-end solution to address the problem of object localisation in partial scenes, where we aim to estimate the position of an object in an unknown area given only a partial 3D scan of the scene. We propose a novel scene representation to facilitate the geometric reasoning, Directed Spatial Commonsense Graph (D-SCG), a spatial scene graph that is enriched with additional concept nodes from a commonsense knowledge base. Specifically, the nodes of D-SCG represent the scene objects and the edges are their relative positions. Each object node is then connected via different commonsense relationships to a set of concept nodes. With the proposed graph-based scene representation, we estimate the unknown position of the target object using a Graph Neural Network that implements a novel attentional message passing mechanism. The network first predicts the relative positions between the target object and each visible object by learning a rich representation of the objects via aggregating both the object nodes and the concept nodes in D-SCG. These relative positions then are merged to obtain the final position. We evaluate our method using Partial ScanNet, improving the state-of-the-art by 5.9% in terms of the localisation accuracy at a 8x faster training speed.