Real-Time Fusion of Visual and Chart Data for Enhanced Maritime Vision

TL;DR

This paper introduces a real-time system that fuses visual and chart data for maritime navigation, using a transformer-based neural network to improve buoy detection and matching accuracy in challenging environments.

Contribution

It presents a novel transformer-based neural network for end-to-end fusion of visual and chart data, enhancing maritime object localization and association accuracy.

Findings

Significantly improved buoy detection accuracy.

Enhanced object association in dynamic maritime scenes.

Outperforms baseline models like ray-casting and YOLOv7-based methods.

Abstract

This paper presents a novel approach to enhancing marine vision by fusing real-time visual data with chart information. Our system overlays nautical chart data onto live video feeds by accurately matching detected navigational aids, such as buoys, with their corresponding representations in chart data. To achieve robust association, we introduce a transformer-based end-to-end neural network that predicts bounding boxes and confidence scores for buoy queries, enabling the direct matching of image-domain detections with world-space chart markers. The proposed method is compared against baseline approaches, including a ray-casting model that estimates buoy positions via camera projection and a YOLOv7-based network extended with a distance estimation module. Experimental results on a dataset of real-world maritime scenes demonstrate that our approach significantly improves object localization and association accuracy in dynamic and challenging environments.