AdaptiveISP: Learning an Adaptive Image Signal Processor for Object Detection

TL;DR

AdaptiveISP is a novel, scene-adaptive image signal processing framework that uses deep reinforcement learning to optimize ISP pipelines for improved object detection performance in dynamic scenes.

Contribution

It introduces a task-driven, adaptive ISP method that dynamically adjusts processing modules and parameters to enhance downstream detection tasks, outperforming prior fixed or quality-focused approaches.

Findings

Surpasses state-of-the-art detection performance.

Effectively balances detection accuracy and computational cost.

Adapts to scenes with large dynamic range variations.

Abstract

Image Signal Processors (ISPs) convert raw sensor signals into digital images, which significantly influence the image quality and the performance of downstream computer vision tasks. Designing ISP pipeline and tuning ISP parameters are two key steps for building an imaging and vision system. To find optimal ISP configurations, recent works use deep neural networks as a proxy to search for ISP parameters or ISP pipelines. However, these methods are primarily designed to maximize the image quality, which are sub-optimal in the performance of high-level computer vision tasks such as detection, recognition, and tracking. Moreover, after training, the learned ISP pipelines are mostly fixed at the inference time, whose performance degrades in dynamic scenes. To jointly optimize ISP structures and parameters, we propose AdaptiveISP, a task-driven and scene-adaptive ISP. One key observation is that for the majority of input images, only a few processing modules are needed to improve the performance of downstream recognition tasks, and only a few inputs require more processing. Based on this, AdaptiveISP utilizes deep reinforcement learning to automatically generate an optimal ISP pipeline and the associated ISP parameters to maximize the detection performance. Experimental results show that AdaptiveISP not only surpasses the prior state-of-the-art methods for object detection but also dynamically manages the trade-off between detection performance and computational cost, especially suitable for scenes with large dynamic range variations. Project website: https://openimaginglab.github.io/AdaptiveISP/.