Human Motion Instruction Tuning

TL;DR

LLaMo introduces a multimodal framework that retains native human motion data for instruction tuning, significantly improving understanding and prediction of complex behaviors in motion-intensive scenarios.

Contribution

It is the first to retain motion in native form during instruction tuning, enhancing interpretation of complex human behaviors compared to traditional tokenization methods.

Findings

Effective in high-complexity domains like human behaviors and professional activities.

Improves comprehension and prediction accuracy in motion-intensive scenarios.

Demonstrates broad applicability from sports analytics to behavioral prediction.

Abstract

This paper presents LLaMo (Large Language and Human Motion Assistant), a multimodal framework for human motion instruction tuning. In contrast to conventional instruction-tuning approaches that convert non-linguistic inputs, such as video or motion sequences, into language tokens, LLaMo retains motion in its native form for instruction tuning. This method preserves motion-specific details that are often diminished in tokenization, thereby improving the model's ability to interpret complex human behaviors. By processing both video and motion data alongside textual inputs, LLaMo enables a flexible, human-centric analysis. Experimental evaluations across high-complexity domains, including human behaviors and professional activities, indicate that LLaMo effectively captures domain-specific knowledge, enhancing comprehension and prediction in motion-intensive scenarios. We hope LLaMo offers a foundation for future multimodal AI systems with broad applications, from sports analytics to behavioral prediction. Our code and models are available on the project website: https://github.com/ILGLJ/LLaMo.