IFShip: Interpretable Fine-grained Ship Classification with Domain Knowledge-Enhanced Vision-Language Models

TL;DR

IFShip introduces an interpretable vision-language model for fine-grained ship classification that leverages domain knowledge and step-by-step reasoning, outperforming existing methods in accuracy and interpretability.

Contribution

The paper presents a novel domain knowledge-enhanced Chain-of-Thought prompt mechanism and a semi-automatic dataset for adapting vision-language models to fine-grained ship classification.

Findings

IFShip outperforms state-of-the-art FGSC algorithms in accuracy and interpretability.

The model provides natural language explanations for its classifications.

IFShip demonstrates superior performance compared to other vision-language models like LLaVA and MiniGPT-4.

Abstract

End-to-end interpretation currently dominates the remote sensing fine-grained ship classification (RS-FGSC) task. However, the inference process remains uninterpretable, leading to criticisms of these models as "black box" systems. To address this issue, we propose a domain knowledge-enhanced Chain-of-Thought (CoT) prompt generation mechanism, which is used to semi-automatically construct a task-specific instruction-following dataset, TITANIC-FGS. By training on TITANIC-FGS, we adapt general-domain vision-language models (VLMs) to the FGSC task, resulting in a model named IFShip. Building upon IFShip, we develop an FGSC visual chatbot that redefines the FGSC problem as a step-by-step reasoning task and conveys the reasoning process in natural language. Experimental results show that IFShip outperforms state-of-the-art FGSC algorithms in both interpretability and classification accuracy. Furthermore, compared to VLMs such as LLaVA and MiniGPT-4, IFShip demonstrates superior performance on the FGSC task. It provides an accurate chain of reasoning when fine-grained ship types are recognizable to the human eye and offers interpretable explanations when they are not. Our dataset is publicly available at: https://github.com/lostwolves/IFShip.