Weakly Aligned Feature Fusion for Multimodal Object Detection

TL;DR

This paper introduces AR-CNN, a multimodal object detection method that effectively addresses the position shift problem in unaligned multimodal data through feature alignment, jitter strategies, and reliable feature fusion, improving detection robustness.

Contribution

The paper presents a novel AR-CNN framework with RF alignment, RoI jitter, and feature reweighting to enhance multimodal object detection under misalignment conditions.

Findings

Effective in 2-D and 3-D detection tasks

Improves robustness to position shifts

Outperforms existing methods on multiple datasets

Abstract

To achieve accurate and robust object detection in the real-world scenario, various forms of images are incorporated, such as color, thermal, and depth. However, multimodal data often suffer from the position shift problem, i.e., the image pair is not strictly aligned, making one object has different positions in different modalities. For the deep learning method, this problem makes it difficult to fuse multimodal features and puzzles the convolutional neural network (CNN) training. In this article, we propose a general multimodal detector named aligned region CNN (AR-CNN) to tackle the position shift problem. First, a region feature (RF) alignment module with adjacent similarity constraint is designed to consistently predict the position shift between two modalities and adaptively align the cross-modal RFs. Second, we propose a novel region of interest (RoI) jitter strategy to improve the robustness to unexpected shift patterns. Third, we present a new multimodal feature fusion method that selects the more reliable feature and suppresses the less useful one via feature reweighting. In addition, by locating bounding boxes in both modalities and building their relationships, we provide novel multimodal labeling named KAIST-Paired. Extensive experiments on 2-D and 3-D object detection, RGB-T, and RGB-D datasets demonstrate the effectiveness and robustness of our method.