Assured Autonomy with Neuro-Symbolic Perception

TL;DR

This paper introduces NeuSPaPer, a neuro-symbolic perception framework that combines object detection and scene graph generation to enhance safety, reliability, and reasoning in autonomous cyber-physical systems.

Contribution

The paper proposes a novel neuro-symbolic perception paradigm that integrates symbolic structure with data-driven models for improved situational awareness in autonomous systems.

Findings

Demonstrates effective scene understanding using joint object detection and scene graph generation.

Shows that structured relational graphs improve the reliability of perception in safety-critical domains.

Validates the approach with simulations and real-world datasets.

Abstract

Many state-of-the-art AI models deployed in cyber-physical systems (CPS), while highly accurate, are simply pattern-matchers.~With limited security guarantees, there are concerns for their reliability in safety-critical and contested domains. To advance assured AI, we advocate for a paradigm shift that imbues data-driven perception models with symbolic structure, inspired by a human's ability to reason over low-level features and high-level context. We propose a neuro-symbolic paradigm for perception (NeuSPaPer) and illustrate how joint object detection and scene graph generation (SGG) yields deep scene understanding.~Powered by foundation models for offline knowledge extraction and specialized SGG algorithms for real-time deployment, we design a framework leveraging structured relational graphs that ensures the integrity of situational awareness in autonomy. Using physics-based simulators and real-world datasets, we demonstrate how SGG bridges the gap between low-level sensor perception and high-level reasoning, establishing a foundation for resilient, context-aware AI and advancing trusted autonomy in CPS.