Beyond Static Frames: Temporal Aggregate-and-Restore Vision Transformer for Human Pose Estimation

TL;DR

This paper introduces TAR-ViTPose, a novel video-based human pose estimation method that leverages temporal aggregation and restoration in Vision Transformers to improve accuracy and stability over static image methods.

Contribution

The paper proposes a joint-centric temporal aggregation and global restoring attention mechanism to enhance ViT-based pose estimation in videos, addressing temporal coherence issues.

Findings

Achieves +2.3 mAP on PoseTrack2017 benchmark.

Outperforms existing state-of-the-art video pose methods.

Provides higher runtime frame rate in real-world scenarios.

Abstract

Vision Transformers (ViTs) have recently achieved state-of-the-art performance in 2D human pose estimation due to their strong global modeling capability. However, existing ViT-based pose estimators are designed for static images and process each frame independently, thereby ignoring the temporal coherence that exists in video sequences. This limitation often results in unstable predictions, especially in challenging scenes involving motion blur, occlusion, or defocus. In this paper, we propose TAR-ViTPose, a novel Temporal Aggregate-and-Restore Vision Transformer tailored for video-based 2D human pose estimation. TAR-ViTPose enhances static ViT representations by aggregating temporal cues across frames in a plug-and-play manner, leading to more robust and accurate pose estimation. To effectively aggregate joint-specific features that are temporally aligned across frames, we introduce a joint-centric temporal aggregation (JTA) that assigns each joint a learnable query token to selectively attend to its corresponding regions from neighboring frames. Furthermore, we develop a global restoring attention (GRA) to restore the aggregated temporal features back into the token sequence of the current frame, enriching its pose representation while fully preserving global context for precise keypoint localization. Extensive experiments demonstrate that TAR-ViTPose substantially improves upon the single-frame baseline ViTPose, achieving a +2.3 mAP gain on the PoseTrack2017 benchmark. Moreover, our approach outperforms existing state-of-the-art video-based methods, while also achieving a noticeably higher runtime frame rate in real-world applications. Project page: https://github.com/zgspose/TARViTPose.