A Multi-Modal Neuro-Symbolic Approach for Spatial Reasoning-Based Visual Grounding in Robotics

TL;DR

This paper introduces a neuro-symbolic framework that combines panoramic images and 3D point clouds to improve spatial reasoning in visual grounding tasks for robotics, enabling explicit modeling of object relationships.

Contribution

It presents a novel multi-modal neuro-symbolic approach integrating perception and symbolic reasoning for enhanced spatial understanding in robotics.

Findings

Outperforms existing models on JRDB-Reasoning dataset

Provides interpretable scene graphs for spatial relationships

Maintains lightweight design suitable for real-time robotics

Abstract

Visual reasoning, particularly spatial reasoning, is a challenging cognitive task that requires understanding object relationships and their interactions within complex environments, especially in robotics domain. Existing vision_language models (VLMs) excel at perception tasks but struggle with fine-grained spatial reasoning due to their implicit, correlation-driven reasoning and reliance solely on images. We propose a novel neuro_symbolic framework that integrates both panoramic-image and 3D point cloud information, combining neural perception with symbolic reasoning to explicitly model spatial and logical relationships. Our framework consists of a perception module for detecting entities and extracting attributes, and a reasoning module that constructs a structured scene graph to support precise, interpretable queries. Evaluated on the JRDB-Reasoning dataset, our approach demonstrates superior performance and reliability in crowded, human_built environments while maintaining a lightweight design suitable for robotics and embodied AI applications.