Fine-Grained Scene Image Classification with Modality-Agnostic Adapter

TL;DR

This paper introduces MAA, a novel multi-modal feature fusion method for fine-grained scene image classification that adaptively learns modality importance without prior assumptions, achieving state-of-the-art results.

Contribution

The paper proposes a modality-agnostic Transformer-based fusion approach that automatically learns modality importance and easily incorporates new modalities, improving classification performance.

Findings

Achieves state-of-the-art results on benchmarks

Effectively handles multiple modalities with adaptive importance

Easily integrates new modalities to boost performance

Abstract

When dealing with the task of fine-grained scene image classification, most previous works lay much emphasis on global visual features when doing multi-modal feature fusion. In other words, models are deliberately designed based on prior intuitions about the importance of different modalities. In this paper, we present a new multi-modal feature fusion approach named MAA (Modality-Agnostic Adapter), trying to make the model learn the importance of different modalities in different cases adaptively, without giving a prior setting in the model architecture. More specifically, we eliminate the modal differences in distribution and then use a modality-agnostic Transformer encoder for a semantic-level feature fusion. Our experiments demonstrate that MAA achieves state-of-the-art results on benchmarks by applying the same modalities with previous methods. Besides, it is worth mentioning that new modalities can be easily added when using MAA and further boost the performance. Code is available at https://github.com/quniLcs/MAA.