Learning to Sense for Driving: Joint Optics-Sensor-Model Co-Design for Semantic Segmentation

TL;DR

This paper introduces a unified end-to-end RAW-to-task co-design framework for autonomous driving, optimizing optics, sensors, and segmentation models jointly to improve robustness and efficiency in perception tasks.

Contribution

It presents a novel joint optimization approach integrating realistic sensor models with lightweight segmentation networks, enhancing robustness and accuracy in autonomous driving perception pipelines.

Findings

Improved mIoU on KITTI-360 dataset with co-designed sensors.

Robustness gains under blur, noise, and low-bit conditions.

Achieves real-time performance with a compact model (~1M parameters).

Abstract

Traditional autonomous driving pipelines decouple camera design from downstream perception, relying on fixed optics and handcrafted ISPs that prioritize human viewable imagery rather than machine semantics. This separation discards information during demosaicing, denoising, or quantization, while forcing models to adapt to sensor artifacts. We present a task-driven co-design framework that unifies optics, sensor modeling, and lightweight semantic segmentation networks into a single end-to-end RAW-to-task pipeline. Building on DeepLens[19], our system integrates realistic cellphone-scale lens models, learnable color filter arrays, Poisson-Gaussian noise processes, and quantization, all optimized directly for segmentation objectives. Evaluations on KITTI-360 show consistent mIoU improvements over fixed pipelines, with optics modeling and CFA learning providing the largest gains, especially for thin or low-light-sensitive classes. Importantly, these robustness gains are achieved with a compact ~1M-parameter model running at ~28 FPS, demonstrating edge deployability. Visual and quantitative analyses further highlight how co-designed sensors adapt acquisition to semantic structure, sharpening boundaries and maintaining accuracy under blur, noise, and low bit-depth. Together, these findings establish full-stack co-optimization of optics, sensors, and networks as a principled path toward efficient, reliable, and deployable perception in autonomous systems.