Zero-Shot Scene Understanding for Automatic Target Recognition Using Large Vision-Language Models

TL;DR

This paper introduces a new pipeline combining open-world detectors with large vision-language models to enable zero-shot recognition of novel objects and environments in automatic target recognition tasks, especially in military scenarios.

Contribution

It proposes a novel system that leverages LVLMs and open-world detectors for improved zero-shot ATR, addressing limitations in localizing objects and recognizing unseen classes in unknown domains.

Findings

LVLMs can recognize military vehicles in zero-shot settings.

The system's performance varies with distance, modality, and prompting methods.

Insights are provided for developing more reliable ATR in novel conditions.

Abstract

Automatic target recognition (ATR) plays a critical role in tasks such as navigation and surveillance, where safety and accuracy are paramount. In extreme use cases, such as military applications, these factors are often challenged due to the presence of unknown terrains, environmental conditions, and novel object categories. Current object detectors, including open-world detectors, lack the ability to confidently recognize novel objects or operate in unknown environments, as they have not been exposed to these new conditions. However, Large Vision-Language Models (LVLMs) exhibit emergent properties that enable them to recognize objects in varying conditions in a zero-shot manner. Despite this, LVLMs struggle to localize objects effectively within a scene. To address these limitations, we propose a novel pipeline that combines the detection capabilities of open-world detectors with the recognition confidence of LVLMs, creating a robust system for zero-shot ATR of novel classes and unknown domains. In this study, we compare the performance of various LVLMs for recognizing military vehicles, which are often underrepresented in training datasets. Additionally, we examine the impact of factors such as distance range, modality, and prompting methods on the recognition performance, providing insights into the development of more reliable ATR systems for novel conditions and classes.