LLMRA: Multi-modal Large Language Model based Restoration Assistant

TL;DR

This paper introduces LLMRA, a novel multi-modal large language model framework that leverages language and vision models to improve universal image restoration by incorporating degradation priors and user dialogue.

Contribution

The paper proposes a new MLLM-based image restoration framework with innovative modules for integrating degradation information, enhancing accuracy and adaptability in low-level vision tasks.

Findings

Superior performance on universal image restoration tasks

Effective use of degradation priors from MLLMs

Enhanced restoration accuracy and flexibility

Abstract

Multi-modal Large Language Models (MLLMs) have a significant impact on various tasks, due to their extensive knowledge and powerful perception and generation capabilities. However, it still remains an open research problem on applying MLLMs to low-level vision tasks. In this paper, we present a simple MLLM-based Image Restoration framework to address this gap, namely Multi-modal Large Language Model based Restoration Assistant (LLMRA). We exploit the impressive capabilities of MLLMs to obtain the degradation information for universal image restoration. By employing a pretrained multi-modal large language model and a vision language model, we generate text descriptions and encode them as context embedding with degradation information for the degraded image. Through the proposed Context Enhance Module (CEM) and Degradation Context based Transformer Network (DC-former), we integrate these context embedding into the restoration network, contributing to more accurate and adjustable image restoration. Based on the dialogue with the users, our method leverages image degradation priors from MLLMs, providing low-level attributes descriptions of the input low-quality images and the restored high-quality images simultaneously. Extensive experiments demonstrate the superior performance of our LLMRA in universal image restoration tasks.