FoRA: Low-Rank Adaptation Model beyond Multimodal Siamese Network

TL;DR

FoRA introduces Low-rank Modal Adaptors with a shared backbone and adaptive rank allocation for multimodal object detection, achieving significant accuracy improvements and parameter reduction.

Contribution

The paper proposes a novel low-rank adaptation model with shared backbone and adaptive rank strategy for multimodal detection, addressing complex fusion and parameter issues.

Findings

10.4% accuracy improvement on DroneVehicle dataset

149M fewer parameters compared to state-of-the-art

Effective handling of data heterogeneity at feature levels

Abstract

Multimodal object detection offers a promising prospect to facilitate robust detection in various visual conditions. However, existing two-stream backbone networks are challenged by complex fusion and substantial parameter increments. This is primarily due to large data distribution biases of multimodal homogeneous information. In this paper, we propose a novel multimodal object detector, named Low-rank Modal Adaptors (LMA) with a shared backbone. The shared parameters enhance the consistency of homogeneous information, while lightweight modal adaptors focus on modality unique features. Furthermore, we design an adaptive rank allocation strategy to adapt to the varying heterogeneity at different feature levels. When applied to two multimodal object detection datasets, experiments validate the effectiveness of our method. Notably, on DroneVehicle, LMA attains a 10.4% accuracy improvement over the state-of-the-art method with a 149M-parameters reduction. The code is available at https://github.com/zyszxhy/FoRA. Our work was submitted to ACM MM in April 2024, but was rejected. We will continue to refine our work and paper writing next, mainly including proof of theory and multi-task applications of FoRA.