Object Agnostic 3D Lifting in Space and Time

TL;DR

This paper introduces a novel approach for category-agnostic 3D lifting of 2D keypoints over time, leveraging information across similar objects and temporal context to improve accuracy across multiple animal categories.

Contribution

It proposes a new spatio-temporal method that outperforms existing techniques by utilizing cross-object information and temporal windows, and releases a synthetic dataset for animals.

Findings

Outperforms state-of-the-art on multiple animal categories

Leverages cross-object information for better performance

Uses temporal context for sequence consistency

Abstract

We present a spatio-temporal perspective on category-agnostic 3D lifting of 2D keypoints over a temporal sequence. Our approach differs from existing state-of-the-art methods that are either: (i) object-agnostic, but can only operate on individual frames, or (ii) can model space-time dependencies, but are only designed to work with a single object category. Our approach is grounded in two core principles. First, general information about similar objects can be leveraged to achieve better performance when there is little object-specific training data. Second, a temporally-proximate context window is advantageous for achieving consistency throughout a sequence. These two principles allow us to outperform current state-of-the-art methods on per-frame and per-sequence metrics for a variety of animal categories. Lastly, we release a new synthetic dataset containing 3D skeletons and motion sequences for a variety of animal categories.