PaAgent: Portrait-Aware Image Restoration Agent via Subjective-Objective Reinforcement Learning

TL;DR

PaAgent is a novel portrait-aware image restoration agent that uses a self-evolving portrait bank and reinforcement learning to select optimal restoration tools, significantly improving performance across diverse complex scenarios.

Contribution

Introduces PaAgent, which combines a portrait bank, retrieval-augmented generation, and subjective-objective reinforcement learning for enhanced image restoration.

Findings

Outperforms existing IR methods on 8 benchmarks.

Effectively handles complex, mixed-degradation scenarios.

Demonstrates robustness in diverse real-world conditions.

Abstract

Image Restoration (IR) agents, leveraging multimodal large language models to perceive degradation and invoke restoration tools, have shown promise in automating IR tasks. However, existing IR agents typically lack an insight summarization mechanism for past interactions, which results in an exhaustive search for the optimal IR tool. To address this limitation, we propose a portrait-aware IR agent, dubbed PaAgent, which incorporates a self-evolving portrait bank for IR tools and Retrieval-Augmented Generation (RAG) to select a suitable IR tool for input. Specifically, to construct and evolve the portrait bank, the PaAgent continuously enriches it by summarizing the characteristics of various IR tools with restored images, selected IR tools, and degraded images. In addition, the RAG is employed to select the optimal IR tool for the input image by retrieving relevant insights from the portrait bank. Furthermore, to enhance PaAgent's ability to perceive degradation in complex scenes, we propose a subjective-objective reinforcement learning strategy that considers both image quality scores and semantic insights in reward generation, which accurately provides the degradation information even under partial and non-uniform degradation. Extensive experiments across 8 IR benchmarks, covering six single-degradation and eight mixed-degradation scenarios, validate PaAgent's superiority in addressing complex IR tasks. Our project page is \href{https://wyjgr.github.io/PaAgent.html}{PaAgent}.