Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging

TL;DR

This paper explores the shift from physics-based to data-driven, task-specific camera design, proposing a framework that highlights the integration of physics and data in modern imaging systems and discusses future challenges.

Contribution

It introduces a comprehensive framework for understanding end-to-end camera and algorithm design, emphasizing the integration of physics and data-driven methods in task-specific imaging.

Findings

Physics and data-driven methods are increasingly combined in imaging.

End-to-end design frameworks are emerging for task-specific cameras.

Barriers to progress include hardware constraints and data availability.

Abstract

Cameras were originally designed using physics-based heuristics to capture aesthetic images. In recent years, there has been a transformation in camera design from being purely physics-driven to increasingly data-driven and task-specific. In this paper, we present a framework to understand the building blocks of this nascent field of end-to-end design of camera hardware and algorithms. As part of this framework, we show how methods that exploit both physics and data have become prevalent in imaging and computer vision, underscoring a key trend that will continue to dominate the future of task-specific camera design. Finally, we share current barriers to progress in end-to-end design, and hypothesize how these barriers can be overcome.