Assured Autonomy with Neuro-Symbolic Perception
R. Spencer Hallyburton, Miroslav Pajic

TL;DR
This paper introduces NeuSPaPer, a neuro-symbolic perception framework that combines object detection and scene graph generation to enhance safety, reliability, and reasoning in autonomous cyber-physical systems.
Contribution
The paper proposes a novel neuro-symbolic perception paradigm that integrates symbolic structure with data-driven models for improved situational awareness in autonomous systems.
Findings
Demonstrates effective scene understanding using joint object detection and scene graph generation.
Shows that structured relational graphs improve the reliability of perception in safety-critical domains.
Validates the approach with simulations and real-world datasets.
Abstract
Many state-of-the-art AI models deployed in cyber-physical systems (CPS), while highly accurate, are simply pattern-matchers.~With limited security guarantees, there are concerns for their reliability in safety-critical and contested domains. To advance assured AI, we advocate for a paradigm shift that imbues data-driven perception models with symbolic structure, inspired by a human's ability to reason over low-level features and high-level context. We propose a neuro-symbolic paradigm for perception (NeuSPaPer) and illustrate how joint object detection and scene graph generation (SGG) yields deep scene understanding.~Powered by foundation models for offline knowledge extraction and specialized SGG algorithms for real-time deployment, we design a framework leveraging structured relational graphs that ensures the integrity of situational awareness in autonomy. Using physics-based…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsEmotions and Moral Behavior
