# End-to-End Learning of Representations for Asynchronous Event-Based Data

## TL;DR

This paper introduces a unified, end-to-end learnable framework for converting asynchronous event camera data into grid-based representations, improving performance in optical flow and object recognition tasks.

## Contribution

It presents a differentiable framework that learns event representations jointly with task networks, unifying existing methods and enabling novel representations.

## Key findings

- Approximately 12% improvement in optical flow estimation.
- Approximately 12% improvement in object recognition.
- Framework unifies and extends existing event representations.

## Abstract

Event cameras are vision sensors that record asynchronous streams of per-pixel brightness changes, referred to as "events". They have appealing advantages over frame-based cameras for computer vision, including high temporal resolution, high dynamic range, and no motion blur. Due to the sparse, non-uniform spatiotemporal layout of the event signal, pattern recognition algorithms typically aggregate events into a grid-based representation and subsequently process it by a standard vision pipeline, e.g., Convolutional Neural Network (CNN). In this work, we introduce a general framework to convert event streams into grid-based representations through a sequence of differentiable operations. Our framework comes with two main advantages: (i) allows learning the input event representation together with the task dedicated network in an end to end manner, and (ii) lays out a taxonomy that unifies the majority of extant event representations in the literature and identifies novel ones. Empirically, we show that our approach to learning the event representation end-to-end yields an improvement of approximately 12% on optical flow estimation and object recognition over state-of-the-art methods.