What is (missing or wrong) in the scene? A Hybrid Deep Boltzmann Machine For Contextualized Scene Modeling

TL;DR

This paper introduces a hybrid deep Boltzmann Machine that models object relations for improved scene reasoning in robotics, outperforming baseline models on scene classification tasks.

Contribution

The paper presents a novel hybrid Boltzmann Machine with tri-way edges to incorporate object relations into scene modeling, enhancing reasoning capabilities.

Findings

Outperforms baseline models in scene classification accuracy

Effectively models object relations for scene understanding

Improves reasoning about missing or incorrect scene elements

Abstract

Scene models allow robots to reason about what is in the scene, what else should be in it, and what should not be in it. In this paper, we propose a hybrid Boltzmann Machine (BM) for scene modeling where relations between objects are integrated. To be able to do that, we extend BM to include tri-way edges between visible (object) nodes and make the network to share the relations across different objects. We evaluate our method against several baseline models (Deep Boltzmann Machines, and Restricted Boltzmann Machines) on a scene classification dataset, and show that it performs better in several scene reasoning tasks.