BlinkVision: A Benchmark for Optical Flow, Scene Flow and Point Tracking Estimation using RGB Frames and Events

TL;DR

BlinkVision introduces a comprehensive benchmark dataset combining event data and RGB images with dense annotations for optical flow, scene flow, and point tracking, facilitating advances in correspondence tasks with diverse, naturalistic data.

Contribution

It is the first large-scale benchmark dataset that includes both event data and images with dense annotations for multiple correspondence tasks.

Findings

Enables extensive benchmarking of image-based and event-based methods.

Provides insights into the performance differences across modalities.

Supports diverse naturalistic scenarios for robust evaluation.

Abstract

Recent advances in event-based vision suggest that these systems complement traditional cameras by providing continuous observation without frame rate limitations and a high dynamic range, making them well-suited for correspondence tasks such as optical flow and point tracking. However, there is still a lack of comprehensive benchmarks for correspondence tasks that include both event data and images. To address this gap, we propose BlinkVision, a large-scale and diverse benchmark with multiple modalities and dense correspondence annotations. BlinkVision offers several valuable features: 1) Rich modalities: It includes both event data and RGB images. 2) Extensive annotations: It provides dense per-pixel annotations covering optical flow, scene flow, and point tracking. 3) Large vocabulary: It contains 410 everyday categories, sharing common classes with popular 2D and 3D datasets like LVIS and ShapeNet. 4) Naturalistic: It delivers photorealistic data and covers various naturalistic factors, such as camera shake and deformation. BlinkVision enables extensive benchmarks on three types of correspondence tasks (optical flow, point tracking, and scene flow estimation) for both image-based and event-based methods, offering new observations, practices, and insights for future research. The benchmark website is https://www.blinkvision.net/.