TrackAgent: 6D Object Tracking via Reinforcement Learning

TL;DR

TrackAgent introduces a reinforcement learning-based method for 6D object pose tracking using point cloud alignment, simplifying the process and reducing data requirements compared to RGB(D)-based approaches.

Contribution

The paper presents a novel RL approach for 6D object tracking that operates on point clouds, enabling effective tracking with limited data and integrated reinitialization strategies.

Findings

Effective 6D pose tracking with limited point cloud data

Reinforcement learning improves frame-to-model alignment

Uncertainty and mask propagation aid reinitialization

Abstract

Tracking an object's 6D pose, while either the object itself or the observing camera is moving, is important for many robotics and augmented reality applications. While exploiting temporal priors eases this problem, object-specific knowledge is required to recover when tracking is lost. Under the tight time constraints of the tracking task, RGB(D)-based methods are often conceptionally complex or rely on heuristic motion models. In comparison, we propose to simplify object tracking to a reinforced point cloud (depth only) alignment task. This allows us to train a streamlined approach from scratch with limited amounts of sparse 3D point clouds, compared to the large datasets of diverse RGBD sequences required in previous works. We incorporate temporal frame-to-frame registration with object-based recovery by frame-to-model refinement using a reinforcement learning (RL) agent that jointly solves for both objectives. We also show that the RL agent's uncertainty and a rendering-based mask propagation are effective reinitialization triggers.