LossAgent: Towards Any Optimization Objectives for Image Processing with LLM Agents

TL;DR

LossAgent leverages large language models to understand and optimize complex, customized objectives in low-level image processing tasks, surpassing traditional loss functions.

Contribution

This work introduces LossAgent, the first LLM-based loss agent capable of handling diverse and complex optimization objectives in image processing tasks.

Findings

Effective in super-resolution and restoration tasks

Supports complex perceptual and human feedback objectives

Demonstrates adaptability across multiple image processing scenarios

Abstract

We present the first loss agent, dubbed LossAgent, for low-level image processing tasks, e.g., image super-resolution and restoration, intending to achieve any customized optimization objectives of low-level image processing in different practical applications. Notably, not all optimization objectives, such as complex hand-crafted perceptual metrics, text description, and intricate human feedback, can be instantiated with existing low-level losses, e.g., MSE loss, which presents a crucial challenge in optimizing image processing networks in an end-to-end manner. To eliminate this, our LossAgent introduces the powerful large language model (LLM) as the loss agent, where the rich textual understanding of prior knowledge empowers the loss agent with the potential to understand complex optimization objectives, trajectory, and state feedback from external environments in the optimization process of the low-level image processing networks. In particular, we establish the loss repository by incorporating existing loss functions that support the end-to-end optimization for low-level image processing. Then, we design the optimization-oriented prompt engineering for the loss agent to actively and intelligently decide the compositional weights for each loss in the repository at each optimization interaction, thereby achieving the required optimization trajectory for any customized optimization objectives. Extensive experiments on three typical low-level image processing tasks and multiple optimization objectives have shown the effectiveness and applicability of our proposed LossAgent.