Event-based vision on FPGAs -- a survey

TL;DR

This survey reviews how FPGAs are used to process event camera data across various applications, highlighting their benefits in energy efficiency and real-time performance in embedded vision systems.

Contribution

It provides a comprehensive overview of FPGA-based event data processing techniques and discusses current trends and challenges in the field.

Findings

FPGAs enable real-time processing of event data in embedded systems.

Applications include filtering, stereovision, optical flow, and AI acceleration.

FPGAs improve energy efficiency and low latency in event-based vision systems.

Abstract

In recent years there has been a growing interest in event cameras, i.e. vision sensors that record changes in illumination independently for each pixel. This type of operation ensures that acquisition is possible in very adverse lighting conditions, both in low light and high dynamic range, and reduces average power consumption. In addition, the independent operation of each pixel results in low latency, which is desirable for robotic solutions. Nowadays, Field Programmable Gate Arrays (FPGAs), along with general-purpose processors (GPPs/CPUs) and programmable graphics processing units (GPUs), are popular architectures for implementing and accelerating computing tasks. In particular, their usefulness in the embedded vision domain has been repeatedly demonstrated over the past 30 years, where they have enabled fast data processing (even in real-time) and energy efficiency. Hence, the combination of event cameras and reconfigurable devices seems to be a good solution, especially in the context of energy-efficient real-time embedded systems. This paper gives an overview of the most important works, where FPGAs have been used in different contexts to process event data. It covers applications in the following areas: filtering, stereovision, optical flow, acceleration of AI-based algorithms (including spiking neural networks) for object classification, detection and tracking, and applications in robotics and inspection systems. Current trends and challenges for such systems are also discussed.