Deep Learning-based Event Data Coding: A Joint Spatiotemporal and Polarity Solution

TL;DR

This paper introduces a novel deep learning-based lossy event data coding method that uses a single-point cloud representation to efficiently compress neuromorphic vision sensor data while maintaining task performance.

Contribution

It proposes a unique joint coding approach leveraging a single-point cloud representation and adaptive binarization strategies, outperforming existing lossless and lossy coding standards.

Findings

Significant compression gains over state-of-the-art methods.

Lossy coding maintains classification accuracy with reduced data rates.

Adaptive strategies optimize for quality or task-specific performance.

Abstract

Neuromorphic vision sensors, commonly referred to as event cameras, generate a massive number of pixel-level events, composed by spatiotemporal and polarity information, thus demanding highly efficient coding solutions. Existing solutions focus on lossless coding of event data, assuming that no distortion is acceptable for the target use cases, mostly including computer vision tasks such as classification and recognition. One promising coding approach exploits the similarity between event data and point clouds, both being sets of 3D points, thus allowing to use current point cloud coding solutions to code event data, typically adopting a two-point clouds representation, one for each event polarity. This paper proposes a novel lossy Deep Learning-based Joint Event data Coding (DL-JEC) solution, which adopts for the first time a single-point cloud representation, where the event polarity plays the role of a point cloud attribute, thus enabling to exploit the correlation between the geometry/spatiotemporal and polarity event information. Moreover, this paper also proposes novel adaptive voxel binarization strategies which may be used in DL-JEC, optimized for either quality-oriented or computer vision task-oriented purposes which allow to maximize the performance for the task at hand. DL-JEC can achieve significant compression performance gains when compared with relevant conventional and DL-based state-of-the-art event data coding solutions, notably the MPEG G-PCC and JPEG Pleno PCC standards. Furthermore, it is shown that it is possible to use lossy event data coding, with significantly reduced rate regarding lossless coding, without compromising the target computer vision task performance, notably event classification, thus changing the current event data coding paradigm.