DISeR: Designing Imaging Systems with Reinforcement Learning

TL;DR

This paper introduces DISeR, a reinforcement learning-based method for automatically designing imaging systems by jointly optimizing camera components and perception models, leading to superior task-specific performance.

Contribution

It formulates imaging system design as a CFG and applies reinforcement learning to efficiently search for optimal configurations, a novel approach in this domain.

Findings

Outperforms industry standards in depth estimation tasks.

Generates superior camera rigs for autonomous vehicles.

Automates imaging system design process.

Abstract

Imaging systems consist of cameras to encode visual information about the world and perception models to interpret this encoding. Cameras contain (1) illumination sources, (2) optical elements, and (3) sensors, while perception models use (4) algorithms. Directly searching over all combinations of these four building blocks to design an imaging system is challenging due to the size of the search space. Moreover, cameras and perception models are often designed independently, leading to sub-optimal task performance. In this paper, we formulate these four building blocks of imaging systems as a context-free grammar (CFG), which can be automatically searched over with a learned camera designer to jointly optimize the imaging system with task-specific perception models. By transforming the CFG to a state-action space, we then show how the camera designer can be implemented with reinforcement learning to intelligently search over the combinatorial space of possible imaging system configurations. We demonstrate our approach on two tasks, depth estimation and camera rig design for autonomous vehicles, showing that our method yields rigs that outperform industry-wide standards. We believe that our proposed approach is an important step towards automating imaging system design.