RGB-Event ISP: The Dataset and Benchmark
Yunfan Lu, Yanlin Qian, Ziyang Rao, Junren Xiao, Liming Chen, Hui, Xiong
TL;DR
This paper introduces the first event-guided ISP framework, a new dataset, and benchmark methods, aiming to improve image processing by leveraging event sensor data for better RGB image quality.
Contribution
It presents a novel event-RAW dataset, a conventional ISP pipeline for reference, and the first event-guided ISP method, establishing a new research direction.
Findings
New dataset with 3373 RAW images and aligned events across 24 scenes.
Evaluation of existing learnable ISP methods on the new dataset.
Proposed a simple event-guided ISP method demonstrating potential benefits.
Abstract
Event-guided imaging has received significant attention due to its potential to revolutionize instant imaging systems. However, the prior methods primarily focus on enhancing RGB images in a post-processing manner, neglecting the challenges of image signal processor (ISP) dealing with event sensor and the benefits events provide for reforming the ISP process. To achieve this, we conduct the first research on event-guided ISP. First, we present a new event-RAW paired dataset, collected with a novel but still confidential sensor that records pixel-level aligned events and RAW images. This dataset includes 3373 RAW images with 2248 x 3264 resolution and their corresponding events, spanning 24 scenes with 3 exposure modes and 3 lenses. Second, we propose a conventional ISP pipeline to generate good RGB frames as reference. This conventional ISP pipleline performs basic ISP operations,…
Peer Reviews
Decision·ICLR 2025 Poster
1. The present paper introduces a event-RAW paired dataset that addresses a gap in the field of ISP and facilitates further research on event-guided ISP. 2. The trainable ISP methods are evaluated on this event-RAW paired dataset.
This paper, as a work primarily contributing a dataset, provides insufficient information about the dataset itself and focuses heavily on performance comparisons with existing methods, but the proposed method fails to outperform the current ones across all metrics. The overall logic and structure of the paper need improvement and refinement. 1. As a dataset-centric paper, it presents too few sample images, making it difficult for readers to intuitively grasp the specific content of the dataset a
1. Dataset: This is the first dataset with aligned (in both time and spatial space) RAW and events from a new HVS equipment, which may provide some avenues for developing new ISP algorithms. 2. Benchmark Task: This work evaluates the proposed dataset by testing various ISP baseline methods along with an event-guided ISP approach. Some conclusions and insights can be drawn from these experiments.
1. Scale of the dataset: From my understanding, a robust dataset should have both scale and diversity. Although the authors mention that the dataset is relatively small due to the low stability of the HVS sensor, I am not completely convinced by it, and I still believe it would be beneficial to expand the dataset further to increase its richness and variety. Additionally, the authors could include suggestions in the future work section on strategies for scaling up the dataset. 2. Reproducibility
1. The relevant background knowledge of this paper is clearly explained. 2. The topic of RGB-event ISP is very interesting topic and meanful for future camera. 3. The writing is very well and easy to understand.
1. I greatly appreciate the effort to create a dataset for RGB-Event ISP, which opens up opportunities for event-assisted RGB ISP tasks. However, the dataset's scale is quite limited, with only 3,373 samples, which may not be sufficient to support data-driven learning methods. This raises concerns about the dataset's ability to serve as a professional, standardized, and challenging benchmark. If this is primarily a workload issue, could the authors consider generating some simulated datasets? I
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsIndustrial Vision Systems and Defect Detection
MethodsSoftmax · Attention Is All You Need · Focus