Light Field Based 6DoF Tracking of Previously Unobserved Objects

TL;DR

This paper presents a light field-based 6DoF object tracking method that does not rely on pre-trained models, effectively handling complex visual effects like reflections, and demonstrating competitive performance on challenging datasets.

Contribution

Introduces a novel light field image-based tracking approach that uses view-dependent Gaussian splats and does not depend on pre-trained models, improving robustness to complex visual appearances.

Findings

Competitive with state-of-the-art trackers on challenging reflective objects

Supports differentiable rendering and pose optimization

Provides a new dataset with challenging reflective objects and ground truth poses

Abstract

Object tracking is an important step in robotics and reautonomous driving pipelines, which has to generalize to previously unseen and complex objects. Existing high-performing methods often rely on pre-captured object views to build explicit reference models, which restricts them to a fixed set of known objects. However, such reference models can struggle with visually complex appearance, reducing the quality of tracking. In this work, we introduce an object tracking method based on light field images that does not depend on a pre-trained model, while being robust to complex visual behavior, such as reflections. We extract semantic and geometric features from light field inputs using vision foundation models and convert them into view-dependent Gaussian splats. These splats serve as a unified object representation, supporting differentiable rendering and pose optimization. We further introduce a light field object tracking dataset containing challenging reflective objects with precise ground truth poses. Experiments demonstrate that our method is competitive with state-of-the-art model-based trackers in these difficult cases, paving the way toward universal object tracking in robotic systems. Code/data available at https://github.com/nagonch/LiFT-6DoF.