RXFOOD: Plug-in RGB-X Fusion for Object of Interest Detection

TL;DR

This paper introduces RXFOOD, a novel plug-in attention-based fusion method for RGB-X networks that enhances feature interaction across scales and modalities, improving object detection performance.

Contribution

The paper proposes RXFOOD, a unified attention mechanism with an Energy Exchange Module for multi-scale, multi-modality feature fusion in RGB-X networks, which is easily integrable.

Findings

Improves object detection accuracy across multiple RGB-X tasks.

Demonstrates effectiveness on RGB-NIR, RGB-D, and frequency image detection datasets.

Outperforms naive fusion methods in experiments.

Abstract

The emergence of different sensors (Near-Infrared, Depth, etc.) is a remedy for the limited application scenarios of traditional RGB camera. The RGB-X tasks, which rely on RGB input and another type of data input to resolve specific problems, have become a popular research topic in multimedia. A crucial part in two-branch RGB-X deep neural networks is how to fuse information across modalities. Given the tremendous information inside RGB-X networks, previous works typically apply naive fusion (e.g., average or max fusion) or only focus on the feature fusion at the same scale(s). While in this paper, we propose a novel method called RXFOOD for the fusion of features across different scales within the same modality branch and from different modality branches simultaneously in a unified attention mechanism. An Energy Exchange Module is designed for the interaction of each feature map's energy matrix, who reflects the inter-relationship of different positions and different channels inside a feature map. The RXFOOD method can be easily incorporated to any dual-branch encoder-decoder network as a plug-in module, and help the original backbone network better focus on important positions and channels for object of interest detection. Experimental results on RGB-NIR salient object detection, RGB-D salient object detection, and RGBFrequency image manipulation detection demonstrate the clear effectiveness of the proposed RXFOOD.