WaterVideoQA: ASV-Centric Perception and Rule-Compliant Reasoning via Multi-Modal Agents

TL;DR

WaterVideoQA introduces a large-scale video question answering benchmark and a multi-agent neuro-symbolic system, NaviMind, to enhance autonomous surface vessels' reasoning and decision-making in complex maritime environments.

Contribution

The paper presents the first comprehensive maritime Video QA benchmark and a novel multi-agent reasoning system, advancing knowledge-driven perception for autonomous navigation.

Findings

NaviMind outperforms existing baselines in maritime reasoning tasks.

The benchmark effectively stresses ASV capabilities across diverse waterway conditions.

The system enables regulation-compliant, interpretable decision-making in dynamic environments.

Abstract

While autonomous navigation has achieved remarkable success in passive perception (e.g., object detection and segmentation), it remains fundamentally constrained by a void in knowledge-driven, interactive environmental cognition. In the high-stakes domain of maritime navigation, the ability to bridge the gap between raw visual perception and complex cognitive reasoning is not merely an enhancement but a critical prerequisite for Autonomous Surface Vessels to execute safe and precise maneuvers. To this end, we present WaterVideoQA, the first large-scale, comprehensive Video Question Answering benchmark specifically engineered for all-waterway environments. This benchmark encompasses 3,029 video clips across six distinct waterway categories, integrating multifaceted variables such as volatile lighting and dynamic weather to rigorously stress-test ASV capabilities across a five-tier hierarchical cognitive framework. Furthermore, we introduce NaviMind, a pioneering multi-agent neuro-symbolic system designed for open-ended maritime reasoning. By synergizing Adaptive Semantic Routing, Situation-Aware Hierarchical Reasoning, and Autonomous Self-Reflective Verification, NaviMind transitions ASVs from superficial pattern matching to regulation-compliant, interpretable decision-making. Experimental results demonstrate that our framework significantly transcends existing baselines, establishing a new paradigm for intelligent, trustworthy interaction in dynamic maritime environments.