Temporally-consistent 3D Reconstruction of Birds

TL;DR

This paper presents a novel pipeline for temporally consistent 3D reconstruction of seabirds from monocular videos, enabling detailed analysis of bird behavior and shape changes, with state-of-the-art results on a new dataset.

Contribution

It introduces a full pipeline including detection, tracking, segmentation, and a temporal loss for 3D bird reconstruction, along with a new dataset of seabird videos.

Findings

Achieves state-of-the-art 3D reconstruction accuracy on seabird videos.

Provides a large dataset of annotated seabird videos for research.

Demonstrates effective temporal consistency in 3D pose and shape estimation.

Abstract

This paper deals with 3D reconstruction of seabirds which recently came into focus of environmental scientists as valuable bio-indicators for environmental change. Such 3D information is beneficial for analyzing the bird's behavior and physiological shape, for example by tracking motion, shape, and appearance changes. From a computer vision perspective birds are especially challenging due to their rapid and oftentimes non-rigid motions. We propose an approach to reconstruct the 3D pose and shape from monocular videos of a specific breed of seabird - the common murre. Our approach comprises a full pipeline of detection, tracking, segmentation, and temporally consistent 3D reconstruction. Additionally, we propose a temporal loss that extends current single-image 3D bird pose estimators to the temporal domain. Moreover, we provide a real-world dataset of 10000 frames of video observations on average capture nine birds simultaneously, comprising a large variety of motions and interactions, including a smaller test set with bird-specific keypoint labels. Using our temporal optimization, we achieve state-of-the-art performance for the challenging sequences in our dataset.