Overlooked Poses Actually Make Sense: Distilling Privileged Knowledge for Human Motion Prediction

TL;DR

This paper introduces a novel human motion prediction approach that leverages overlooked future poses as privileged information, using interpolation and knowledge distillation to improve accuracy in complex multivariate time series data.

Contribution

It proposes a new prediction pattern utilizing privileged future poses, with a dual-network architecture and a PK-Simulator for enhanced motion prediction.

Findings

Achieves state-of-the-art results on benchmark datasets.

Effective in both short-term and long-term predictions.

Introduces a novel interpolation-based prediction framework.

Abstract

Previous works on human motion prediction follow the pattern of building a mapping relation between the sequence observed and the one to be predicted. However, due to the inherent complexity of multivariate time series data, it still remains a challenge to find the extrapolation relation between motion sequences. In this paper, we present a new prediction pattern, which introduces previously overlooked human poses, to implement the prediction task from the view of interpolation. These poses exist after the predicted sequence, and form the privileged sequence. To be specific, we first propose an InTerPolation learning Network (ITP-Network) that encodes both the observed sequence and the privileged sequence to interpolate the in-between predicted sequence, wherein the embedded Privileged-sequence-Encoder (Priv-Encoder) learns the privileged knowledge (PK) simultaneously. Then, we propose a Final Prediction Network (FP-Network) for which the privileged sequence is not observable, but is equipped with a novel PK-Simulator that distills PK learned from the previous network. This simulator takes as input the observed sequence, but approximates the behavior of Priv-Encoder, enabling FP-Network to imitate the interpolation process. Extensive experimental results demonstrate that our prediction pattern achieves state-of-the-art performance on benchmarked H3.6M, CMU-Mocap and 3DPW datasets in both short-term and long-term predictions.