Where Is The Ball: 3D Ball Trajectory Estimation From 2D Monocular Tracking

TL;DR

This paper introduces an LSTM-based approach for estimating 3D ball trajectories from 2D monocular video sequences, using a novel camera-invariant representation to handle arbitrary views and achieve state-of-the-art results.

Contribution

The authors propose a new canonical 3D representation and an LSTM pipeline that generalizes from simulated data to real-world scenarios for 3D ball tracking.

Findings

Achieves state-of-the-art performance on synthetic and real datasets.

Generalizes well to real-world scenarios despite training only on simulated data.

Handles multiple trajectories in complex scenes.

Abstract

We present a method for 3D ball trajectory estimation from a 2D tracking sequence. To overcome the ambiguity in 3D from 2D estimation, we design an LSTM-based pipeline that utilizes a novel canonical 3D representation that is independent of the camera's location to handle arbitrary views and a series of intermediate representations that encourage crucial invariance and reprojection consistency. We evaluated our method on four synthetic and three real datasets and conducted extensive ablation studies on our design choices. Despite training solely on simulated data, our method achieves state-of-the-art performance and can generalize to real-world scenarios with multiple trajectories, opening up a range of applications in sport analysis and virtual replay. Please visit our page: https://where-is-the-ball.github.io.