# Fast Trajectory End-Point Prediction with Event Cameras for Reactive Robot Control

## TL;DR

This paper presents a method using event cameras and a Stateful LSTM network to accurately predict the end-point of a trajectory for reactive robot control, improving interception success in dynamic tasks.

## Contribution

It introduces a novel approach combining event cameras with a specialized LSTM for real-time trajectory prediction in robotic interception tasks.

## Key findings

- Event cameras increase the number of successful interceptions.
- The proposed method outperforms traditional frame-based sampling.
- Simulation training accelerates model development.

## Abstract

Prediction skills can be crucial for the success of tasks where robots have limited time to act or joints actuation power. In such a scenario, a vision system with a fixed, possibly too low, sampling rate could lead to the loss of informative points, slowing down prediction convergence and reducing the accuracy. In this paper, we propose to exploit the low latency, motion-driven sampling, and data compression properties of event cameras to overcome these issues. As a use-case, we use a Panda robotic arm to intercept a ball bouncing on a table. To predict the interception point, we adopt a Stateful LSTM network, a specific LSTM variant without fixed input length, which perfectly suits the event-driven paradigm and the problem at hand, where the length of the trajectory is not defined. We train the network in simulation to speed up the dataset acquisition and then fine-tune the models on real trajectories. Experimental results demonstrate how using a dense spatial sampling (i.e. event cameras) significantly increases the number of intercepted trajectories as compared to a fixed temporal sampling (i.e. frame-based cameras).