Multi-model fusion for Aerial Vision and Dialog Navigation based on human attention aids

TL;DR

This paper introduces a multi-model fusion approach using human attention aids for aerial navigation, enabling drones to follow natural language commands more effectively by predicting navigation points and human attention.

Contribution

It proposes a novel fusion training method combining HAA-Transformer and HAA-LSTM models for aerial navigation with human attention guidance.

Findings

Achieves high success rate (SR) and SPL metrics.

Shows a 7% improvement in GP metrics over baseline.

Effectively predicts navigation points and human attention.

Abstract

Drones have been widely used in many areas of our daily lives. It relieves people of the burden of holding a controller all the time and makes drone control easier to use for people with disabilities or occupied hands. However, the control of aerial robots is more complicated compared to normal robots due to factors such as uncontrollable height. Therefore, it is crucial to develop an intelligent UAV that has the ability to talk to humans and follow natural language commands. In this report, we present an aerial navigation task for the 2023 ICCV Conversation History. Based on the AVDN dataset containing more than 3k recorded navigation trajectories and asynchronous human-robot conversations, we propose an effective method of fusion training of Human Attention Aided Transformer model (HAA-Transformer) and Human Attention Aided LSTM (HAA-LSTM) model, which achieves the prediction of the navigation routing points and human attention. The method not only achieves high SR and SPL metrics, but also shows a 7% improvement in GP metrics compared to the baseline model.