NanoMVG: USV-Centric Low-Power Multi-Task Visual Grounding based on Prompt-Guided Camera and 4D mmWave Radar

TL;DR

NanoMVG is a low-power, multi-task visual grounding model for USVs that uses camera and 4D radar to locate objects via natural language, suitable for waterway perception in harsh environments.

Contribution

It introduces NanoMVG, a novel low-power multi-task visual grounding model that integrates camera and radar data for USV perception, enabling deployment in real-world waterway scenarios.

Findings

Achieves competitive performance on WaterVG dataset.

Operates with ultra-low power consumption.

Effective in harsh environmental conditions.

Abstract

Recently, visual grounding and multi-sensors setting have been incorporated into perception system for terrestrial autonomous driving systems and Unmanned Surface Vehicles (USVs), yet the high complexity of modern learning-based visual grounding model using multi-sensors prevents such model to be deployed on USVs in the real-life. To this end, we design a low-power multi-task model named NanoMVG for waterway embodied perception, guiding both camera and 4D millimeter-wave radar to locate specific object(s) through natural language. NanoMVG can perform both box-level and mask-level visual grounding tasks simultaneously. Compared to other visual grounding models, NanoMVG achieves highly competitive performance on the WaterVG dataset, particularly in harsh environments and boasts ultra-low power consumption for long endurance.