Large Models in Dialogue for Active Perception and Anomaly Detection

TL;DR

This paper introduces a novel framework using Large Language Models (LLMs) and deep learning dialogue to actively control drones for enhanced perception and anomaly detection in inaccessible or unfamiliar scenes, demonstrated in simulation.

Contribution

It presents a new LLM-based dialogue approach for active perception and anomaly detection in autonomous aerial monitoring, integrating multimodal VQA and real-time scene exploration.

Findings

Improved scene description accuracy through LLM reasoning.

Effective anomaly detection in simulated environments.

Enhanced perception capabilities over static methods.

Abstract

Autonomous aerial monitoring is an important task aimed at gathering information from areas that may not be easily accessible by humans. At the same time, this task often requires recognizing anomalies from a significant distance or not previously encountered in the past. In this paper, we propose a novel framework that leverages the advanced capabilities provided by Large Language Models (LLMs) to actively collect information and perform anomaly detection in novel scenes. To this end, we propose an LLM based model dialogue approach, in which two deep learning models engage in a dialogue to actively control a drone to increase perception and anomaly detection accuracy. We conduct our experiments in a high fidelity simulation environment where an LLM is provided with a predetermined set of natural language movement commands mapped into executable code functions. Additionally, we deploy a multimodal Visual Question Answering (VQA) model charged with the task of visual question answering and captioning. By engaging the two models in conversation, the LLM asks exploratory questions while simultaneously flying a drone into different parts of the scene, providing a novel way to implement active perception. By leveraging LLMs reasoning ability, we output an improved detailed description of the scene going beyond existing static perception approaches. In addition to information gathering, our approach is utilized for anomaly detection and our results demonstrate the proposed methods effectiveness in informing and alerting about potential hazards.