Exploring Scalable Unified Modeling for General Low-Level Vision

TL;DR

This paper introduces VPIP, a unified framework for low-level vision tasks using visual prompts, demonstrating scalable, adaptable performance across diverse tasks and settings, including zero-shot and few-shot scenarios.

Contribution

The paper proposes a novel prompt-based unified model for low-level vision, extending scalability and generalization across over 100 tasks with strong zero-shot and few-shot capabilities.

Findings

Achieves strong performance across diverse low-level vision tasks.

Model generalizes well in zero-shot and few-shot settings.

Increasing task diversity improves model robustness and transferability.

Abstract

Low-level vision involves a wide spectrum of tasks, including image restoration, enhancement, stylization, and feature extraction, which differ significantly in both task formulation and output domains. To address the challenge of unified modeling across such diverse tasks, we propose a Visual task Prompt-based Image Processing (VPIP) framework that leverages input-target image pairs as visual prompts to guide the model in performing a variety of low-level vision tasks. The framework comprises an end-to-end image processing backbone, a prompt encoder, and a prompt interaction module, enabling flexible integration with various architectures and effective utilization of task-specific visual representations. Based on this design, we develop a unified low-level vision model, GenLV, and evaluate its performance across multiple representative tasks. To explore the scalability of this approach, we extend the framework along two dimensions: model capacity and task diversity. We construct a large-scale benchmark consisting of over 100 low-level vision tasks and train multiple versions of the model with varying scales. Experimental results show that the proposed method achieves considerable performance across a wide range of tasks. Notably, increasing the number of training tasks enhances generalization, particularly for tasks with limited data, indicating the model's ability to learn transferable representations through joint training. Further evaluations in zero-shot generalization, few-shot transfer, and task-specific fine-tuning scenarios demonstrate the model's strong adaptability, confirming the effectiveness, scalability, and potential of the proposed framework as a unified foundation for general low-level vision modeling.