VELoRA: A Low-Rank Adaptation Approach for Efficient RGB-Event based Recognition

TL;DR

VELoRA introduces a parameter-efficient fine-tuning method using low-rank adaptation for RGB-Event recognition, enabling effective multi-modal classification with reduced computational cost.

Contribution

It proposes a novel PEFT strategy with modality-specific LoRA tuning for pre-trained vision models in RGB-Event recognition tasks.

Findings

Effective multi-modal feature learning via LoRA tuning.

Improved efficiency in fine-tuning pre-trained models.

Open-source code and models available for reproducibility.

Abstract

Pattern recognition leveraging both RGB and Event cameras can significantly enhance performance by deploying deep neural networks that utilize a fine-tuning strategy. Inspired by the successful application of large models, the introduction of such large models can also be considered to further enhance the performance of multi-modal tasks. However, fully fine-tuning these models leads to inefficiency and lightweight fine-tuning methods such as LoRA and Adapter have been proposed to achieve a better balance between efficiency and performance. To our knowledge, there is currently no work that has conducted parameter-efficient fine-tuning (PEFT) for RGB-Event recognition based on pre-trained foundation models. To address this issue, this paper proposes a novel PEFT strategy to adapt the pre-trained foundation vision models for the RGB-Event-based classification. Specifically, given the RGB frames and event streams, we extract the RGB and event features based on the vision foundation model ViT with a modality-specific LoRA tuning strategy. The frame difference of the dual modalities is also considered to capture the motion cues via the frame difference backbone network. These features are concatenated and fed into high-level Transformer layers for efficient multi-modal feature learning via modality-shared LoRA tuning. Finally, we concatenate these features and feed them into a classification head to achieve efficient fine-tuning. The source code and pre-trained models will be released on \url{https://github.com/Event-AHU/VELoRA}.