Hybrid Event Frame Sensors: Modeling, Calibration, and Simulation

TL;DR

This paper introduces a unified noise model for hybrid sensors combining APS and EVS, along with a calibration pipeline and a realistic simulator validated on multiple imaging tasks.

Contribution

It presents the first comprehensive, statistics-based noise model for hybrid APS-EVS sensors, including calibration and a realistic simulation framework.

Findings

The noise model accurately describes APS and EVS behaviors.

Calibration pipeline effectively estimates noise parameters from real data.

Simulator demonstrates strong transferability to real-world imaging tasks.

Abstract

Event frame hybrid sensors integrate an Active Pixel Sensor (APS) and an Event Vision Sensor (EVS) within a single chip, combining the high dynamic range and low latency of the EVS with the rich spatial intensity information from the APS. While this tight integration offers compact, temporally precise imaging, the complex circuit architecture introduces non-trivial noise patterns that remain poorly understood and unmodeled. In this work, we present the first unified, statistics-based imaging noise model that jointly describes the noise behavior of APS and EVS pixels. Our formulation explicitly incorporates photon shot noise, dark current noise, fixed-pattern noise, and quantization noise, and links EVS noise to illumination level and dark current. Based on this formulation, we further develop a calibration pipeline to estimate noise parameters from real data and offer a detailed analysis of both APS and EVS noise behaviors. Finally, we propose HESIM, a statistically grounded simulator that generates RAW frames and events under realistic, jointly calibrated noise statistics. Experiments on two hybrid sensors validate our model across multiple imaging tasks (e.g., video frame interpolation and deblurring), demonstrating strong transfer from simulation to real data.