Low-rank Adaptation-based All-Weather Removal for Autonomous Navigation

TL;DR

This paper introduces a low-rank adaptation approach, LoRA, combined with a novel fine-tuning method LoRA-Align, to efficiently adapt all-weather image restoration models for autonomous navigation under unseen weather conditions, preserving original capabilities.

Contribution

The paper proposes LoRA and LoRA-Align methods for efficient model adaptation to new weather conditions while maintaining original task performance.

Findings

LoRA effectively adapts models to new weather conditions.

LoRA-Align preserves pre-trained task performance during adaptation.

Restored images improve autonomous navigation tasks like segmentation and depth estimation.

Abstract

All-weather image restoration (AWIR) is crucial for reliable autonomous navigation under adverse weather conditions. AWIR models are trained to address a specific set of weather conditions such as fog, rain, and snow. But this causes them to often struggle with out-of-distribution (OoD) samples or unseen degradations which limits their effectiveness for real-world autonomous navigation. To overcome this issue, existing models must either be retrained or fine-tuned, both of which are inefficient and impractical, with retraining needing access to large datasets, and fine-tuning involving many parameters. In this paper, we propose using Low-Rank Adaptation (LoRA) to efficiently adapt a pre-trained all-weather model to novel weather restoration tasks. Furthermore, we observe that LoRA lowers the performance of the adapted model on the pre-trained restoration tasks. To address this issue, we introduce a LoRA-based fine-tuning method called LoRA-Align (LoRA-A) which seeks to align the singular vectors of the fine-tuned and pre-trained weight matrices using Singular Value Decomposition (SVD). This alignment helps preserve the model's knowledge of its original tasks while adapting it to unseen tasks. We show that images restored with LoRA and LoRA-A can be effectively used for computer vision tasks in autonomous navigation, such as semantic segmentation and depth estimation.