Interp3R: Continuous-time 3D Geometry Estimation with Frames and Events

TL;DR

Interp3R is a novel method that uses asynchronous event data to interpolate 3D scene geometry and camera poses at arbitrary times, surpassing existing discrete-time approaches in accuracy and temporal continuity.

Contribution

It introduces the first continuous-time 3D geometry estimation method leveraging event data to interpolate pointmaps from frame-based models.

Findings

Outperforms state-of-the-art baselines in accuracy

Generalizes well across synthetic and real-world data

Provides temporally continuous 3D scene representations

Abstract

In recent years, 3D visual foundation models pioneered by pointmap-based approaches such as DUSt3R have attracted a lot of interest, achieving impressive accuracy and strong generalization across diverse scenes. However, these methods are inherently limited to recovering scene geometry only at the discrete time instants when images are captured, leaving the scene evolution during the blind time between consecutive frames largely unexplored. We introduce Interp3R, to the best of our knowledge the first method that enhances pointmap-based models to estimate depth and camera poses at arbitrary time instants. Interp3R leverages asynchronous event data to interpolate pointmaps produced by frame-based models, enabling temporally continuous geometric representations. Depth and camera poses are then jointly recovered by aligning the interpolated pointmaps together with those predicted by the underlying frame-based models into a consistent spatial framework. We train Interp3R exclusively on a synthetic dataset, yet demonstrate strong generalization across a wide range of synthetic and real-world benchmarks. Extensive experiments show that Interp3R outperforms by a considerable margin state-of-the-art baselines that follow a two-stage pipeline of 2D video frame interpolation followed by 3D geometry estimation.