Learning A Low-Level Vision Generalist via Visual Task Prompt

TL;DR

This paper introduces VPIP, a flexible framework using visual prompts to train a low-level vision generalist model, GenLV, capable of handling diverse tasks with improved performance over existing methods.

Contribution

The paper proposes a novel VPIP framework with prompt cross-attention, enabling a single model to effectively perform multiple low-level vision tasks.

Findings

GenLV outperforms existing methods quantitatively.

GenLV demonstrates strong qualitative results across tasks.

The framework effectively manages diverse input-target domains.

Abstract

Building a unified model for general low-level vision tasks holds significant research and practical value. Current methods encounter several critical issues. Multi-task restoration approaches can address multiple degradation-to-clean restoration tasks, while their applicability to tasks with different target domains (e.g., image stylization) is limited. Methods like PromptGIP can handle multiple input-target domains but rely on the Masked Autoencoder (MAE) paradigm. Consequently, they are tied to the ViT architecture, resulting in suboptimal image reconstruction quality. In addition, these methods are sensitive to prompt image content and often struggle with low-frequency information processing. In this paper, we propose a Visual task Prompt-based Image Processing (VPIP) framework to overcome these challenges. VPIP employs visual task prompts to manage tasks with different input-target domains and allows flexible selection of backbone network suitable for general tasks. Besides, a new prompt cross-attention is introduced to facilitate interaction between the input and prompt information. Based on the VPIP framework, we train a low-level vision generalist model, namely GenLV, on 30 diverse tasks. Experimental results show that GenLV can successfully address a variety of low-level tasks, significantly outperforming existing methods both quantitatively and qualitatively. Codes are available at https://github.com/chxy95/GenLV.