Differentiable Event Stream Simulator for Non-Rigid 3D Tracking

TL;DR

This paper presents the first differentiable event stream simulator that enables non-rigid 3D tracking from event camera data without requiring large datasets or explicit trajectories, outperforming RGB-based methods.

Contribution

A novel differentiable simulator for event streams that facilitates non-rigid 3D tracking directly from event data, eliminating the need for explicit trajectories or large datasets.

Findings

Outperforms RGB-based methods in 3D error metrics

Works effectively on various non-rigid objects

Does not require large-scale datasets or explicit event trajectories

Abstract

This paper introduces the first differentiable simulator of event streams, i.e., streams of asynchronous brightness change signals recorded by event cameras. Our differentiable simulator enables non-rigid 3D tracking of deformable objects (such as human hands, isometric surfaces and general watertight meshes) from event streams by leveraging an analysis-by-synthesis principle. So far, event-based tracking and reconstruction of non-rigid objects in 3D, like hands and body, has been either tackled using explicit event trajectories or large-scale datasets. In contrast, our method does not require any such processing or data, and can be readily applied to incoming event streams. We show the effectiveness of our approach for various types of non-rigid objects and compare to existing methods for non-rigid 3D tracking. In our experiments, the proposed energy-based formulations outperform competing RGB-based methods in terms of 3D errors. The source code and the new data are publicly available.