From Sparse to Dense: Spatio-Temporal Fusion for Multi-View 3D Human Pose Estimation with DenseWarper

TL;DR

This paper introduces a novel spatio-temporal fusion method for multi-view 3D human pose estimation, leveraging sparse interleaved inputs to enhance temporal resolution and performance, with the DenseWarper model utilizing epipolar geometry.

Contribution

It proposes a new sparse interleaved input approach and the DenseWarper model, improving 3D pose estimation by capturing rich spatio-temporal information and increasing frame rate.

Findings

Outperforms traditional dense multi-view methods on Human3.6M and MPI-INF-3DHP datasets.

Achieves state-of-the-art performance with sparse interleaved inputs.

Theoretically increases pose frame rate by N times with N cameras.

Abstract

In multi-view 3D human pose estimation, models typically rely on images captured simultaneously from different camera views to predict a pose at a specific moment. While providing accurate spatial information, this traditional approach often overlooks the rich temporal dependencies between adjacent frames. We propose a novel 3D human pose estimation input method: the sparse interleaved input to address this. This method leverages images captured from different camera views at various time points (e.g., View 1 at time $t$ and View 2 at time $t+\delta$), allowing our model to capture rich spatio-temporal information and effectively boost performance. More importantly, this approach offers two key advantages: First, it can theoretically increase the output pose frame rate by N times with N cameras, thereby breaking through single-view frame rate limitations and enhancing the temporal resolution of the production. Second, using a sparse subset of available frames, our method can reduce data redundancy and simultaneously achieve better performance. We introduce the DenseWarper model, which leverages epipolar geometry for efficient spatio-temporal heatmap exchange. We conducted extensive experiments on the Human3.6M and MPI-INF-3DHP datasets. Results demonstrate that our method, utilizing only sparse interleaved images as input, outperforms traditional dense multi-view input approaches and achieves state-of-the-art performance. The source code for this work is available at: https://github.com/lingli1724/DenseWarper-ICLR2026