RAW-Adapter: Adapting Pre-trained Visual Model to Camera RAW Images

TL;DR

RAW-Adapter is a novel framework that adapts pre-trained sRGB models to RAW images by integrating learnable ISP stages and model-level adapters, achieving state-of-the-art results across diverse lighting conditions.

Contribution

It introduces a general adapter-based framework for effectively transferring pre-trained models to RAW image data, addressing interaction gaps between ISP stages and high-level networks.

Findings

Achieves state-of-the-art performance on various datasets.

Effective across different lighting conditions.

Demonstrates efficiency and versatility in real-world applications.

Abstract

sRGB images are now the predominant choice for pre-training visual models in computer vision research, owing to their ease of acquisition and efficient storage. Meanwhile, the advantage of RAW images lies in their rich physical information under variable real-world challenging lighting conditions. For computer vision tasks directly based on camera RAW data, most existing studies adopt methods of integrating image signal processor (ISP) with backend networks, yet often overlook the interaction capabilities between the ISP stages and subsequent networks. Drawing inspiration from ongoing adapter research in NLP and CV areas, we introduce RAW-Adapter, a novel approach aimed at adapting sRGB pre-trained models to camera RAW data. RAW-Adapter comprises input-level adapters that employ learnable ISP stages to adjust RAW inputs, as well as model-level adapters to build connections between ISP stages and subsequent high-level networks. Additionally, RAW-Adapter is a general framework that could be used in various computer vision frameworks. Abundant experiments under different lighting conditions have shown our algorithm's state-of-the-art (SOTA) performance, demonstrating its effectiveness and efficiency across a range of real-world and synthetic datasets.