MoFM: A Large-Scale Human Motion Foundation Model

TL;DR

MoFM is a large-scale, versatile human motion foundation model that leverages a novel motion dictionary and heatmap encoding to understand complex motions across various tasks.

Contribution

Introduces MoFM, a pioneering motion foundation model inspired by LLMs, utilizing MotionBook and Thermal Cubes for scalable, efficient human motion understanding.

Findings

MoFM demonstrates strong generalization across diverse motion tasks.

The MotionBook dataset enables large-scale training of motion models.

Thermal Cubes effectively encode spatio-temporal motion heatmaps.

Abstract

Foundation Models (FM) have increasingly drawn the attention of researchers due to their scalability and generalization across diverse tasks. Inspired by the success of FMs and the principles that have driven advancements in Large Language Models (LLMs), we introduce MoFM as a novel Motion Foundation Model. MoFM is designed for the semantic understanding of complex human motions in both time and space. To facilitate large-scale training, MotionBook, a comprehensive human motion dictionary of discretized motions is designed and employed. MotionBook utilizes Thermal Cubes to capture spatio-temporal motion heatmaps, applying principles from discrete variational models to encode human movements into discrete units for a more efficient and scalable representation. MoFM, trained on a large corpus of motion data, provides a foundational backbone adaptable to diverse downstream tasks, supporting paradigms such as one-shot, unsupervised, and supervised tasks. This versatility makes MoFM well-suited for a wide range of motion-based applications.