Diffusion Masked Pretraining for Dynamic Point Cloud

TL;DR

DiMP introduces a diffusion-based self-supervised pretraining framework for dynamic point clouds, addressing positional leakage and multimodal motion uncertainty to improve downstream tasks.

Contribution

It applies diffusion modeling to both positional inference and motion learning, removing positional leakage and modeling multimodal motion distributions.

Findings

DiMP achieves 11.21% improvement on offline action segmentation.

DiMP improves online inference accuracy by 13.65%.

The method outperforms backbone models across multiple benchmarks.

Abstract

Dynamic point cloud pretraining is still dominated by masked reconstruction objectives. However, these objectives inherit two key limitations. Existing methods inject ground-truth tube centers as decoder positional embeddings, causing spatio-temporal positional leakage. Moreover, they supervise inter-frame motion with deterministic proxy targets that systematically discard distributional structure by collapsing multimodal trajectory uncertainty into conditional means. To address these limitations, we propose Diffusion Masked Pretraining (DiMP), a unified self-supervised framework for dynamic point clouds. DiMP introduces diffusion modeling into both positional inference and motion learning. It first applies forward diffusion noise only to masked tube centers, then predicts clean centers from visible spatio-temporal context. This removes positional leakage while preserving visible coordinates as clean temporal anchors. DiMP also reformulates point-wise inter-frame displacement supervision as a DDPM noise-prediction objective conditioned on decoded representations. This design drives the encoder to target the full conditional distribution of plausible motions under a variational surrogate, rather than collapsing to a single deterministic estimate. Extensive experiments demonstrate that DiMP consistently improves downstream accuracy over the backbone alone, with absolute gains of 11.21% on offline action segmentation and 13.65% under causally constrained online inference.Codes are available at https://github.com/InitalZ/DiMP.git.