NAVER: A Neuro-Symbolic Compositional Automaton for Visual Grounding with Explicit Logic Reasoning

TL;DR

NAVER introduces a neuro-symbolic automaton for visual grounding that combines explicit probabilistic logic reasoning with a self-correcting mechanism, achieving state-of-the-art results in complex reasoning tasks.

Contribution

The paper presents NAVER, a novel compositional visual grounding approach integrating explicit logic reasoning within a finite-state automaton, enhancing robustness and interpretability.

Findings

NAVER outperforms recent baselines in visual grounding tasks.

Explicit logic reasoning improves robustness in complex queries.

Self-correcting mechanism enhances inference accuracy.

Abstract

Visual Grounding (VG) tasks, such as referring expression detection and segmentation tasks are important for linking visual entities to context, especially in complex reasoning tasks that require detailed query interpretation. This paper explores VG beyond basic perception, highlighting challenges for methods that require reasoning like human cognition. Recent advances in large language methods (LLMs) and Vision-Language methods (VLMs) have improved abilities for visual comprehension, contextual understanding, and reasoning. These methods are mainly split into end-to-end and compositional methods, with the latter offering more flexibility. Compositional approaches that integrate LLMs and foundation models show promising performance but still struggle with complex reasoning with language-based logical representations. To address these limitations, we propose NAVER, a compositional visual grounding method that integrates explicit probabilistic logic reasoning within a finite-state automaton, equipped with a self-correcting mechanism. This design improves robustness and interpretability in inference through explicit logic reasoning. Our results show that NAVER achieves SoTA performance comparing to recent end-to-end and compositional baselines. The code is available at https://github.com/ControlNet/NAVER .