TETO: Tracking Events with Teacher Observation for Motion Estimation and Frame Interpolation

TL;DR

This paper introduces TETO, a novel teacher-student framework that learns event-based motion estimation from limited real-world data, significantly improving point tracking, optical flow, and frame interpolation performance.

Contribution

TETO is the first approach to learn event motion estimation solely from real-world data using knowledge distillation, reducing reliance on synthetic datasets.

Findings

State-of-the-art point tracking on EVIMO2

Superior optical flow on DSEC

Enhanced frame interpolation quality on BS-ERGB and HQ-EVFI

Abstract

Event cameras capture per-pixel brightness changes with microsecond resolution, offering continuous motion information lost between RGB frames. However, existing event-based motion estimators depend on large-scale synthetic data that often suffers from a significant sim-to-real gap. We propose TETO (Tracking Events with Teacher Observation), a teacher-student framework that learns event motion estimation from only $\sim$25 minutes of unannotated real-world recordings through knowledge distillation from a pretrained RGB tracker. Our motion-aware data curation and query sampling strategy maximizes learning from limited data by disentangling object motion from dominant ego-motion. The resulting estimator jointly predicts point trajectories and dense optical flow, which we leverage as explicit motion priors to condition a pretrained video diffusion transformer for frame interpolation. We achieve state-of-the-art point tracking on EVIMO2 and optical flow on DSEC using orders of magnitude less training data, and demonstrate that accurate motion estimation translates directly to superior frame interpolation quality on BS-ERGB and HQ-EVFI.