JOCA: Task-Driven Joint Optimisation of Camera Hardware and Adaptive Camera Control Algorithms

TL;DR

This paper presents a unified framework for jointly optimizing camera hardware and adaptive control algorithms to enhance perception tasks, especially under challenging conditions like low light and fast motion.

Contribution

It introduces a novel joint optimization method combining gradient-based and derivative-free techniques for both hardware and adaptive control parameters.

Findings

Outperforms baseline methods in perception tasks.

Effective under low light and fast motion conditions.

Supports both continuous and discrete parameter optimization.

Abstract

The quality of captured images strongly influences the performance of downstream perception tasks. Recent works on co-designing camera systems with perception tasks have shown improved task performance. However, most prior approaches focus on optimising fixed camera parameters set at manufacturing, while many parameters, such as exposure settings, require adaptive control at runtime. This paper introduces a method that jointly optimises camera hardware and adaptive camera control algorithms with downstream vision tasks. We present a unified optimisation framework that integrates gradient-based and derivative-free methods, enabling support for both continuous and discrete parameters, non-differentiable image formation processes, and neural network-based adaptive control algorithms. To address non-differentiable effects such as motion blur, we propose DF-Grad, a hybrid optimisation strategy that trains adaptive control networks using signals from a derivative-free optimiser alongside unsupervised task-driven learning. Experiments show that our method outperforms baselines that optimise static and dynamic parameters separately, particularly under challenging conditions such as low light and fast motion. These results demonstrate that jointly optimising hardware parameters and adaptive control algorithms improves perception performance and provides a unified approach to task-driven camera system design.