ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Model

TL;DR

This paper investigates the application of scaling laws to motion generation, introducing a framework that confirms predictable relationships between model size, data, and performance, enabling optimal model configuration predictions.

Contribution

First to demonstrate the existence of scaling laws in motion generation, providing a foundation for optimizing model design based on compute budgets.

Findings

Normalized test loss follows a logarithmic law with compute

Power law relationships between parameters and data tokens

Predicted optimal model configurations match experimental results

Abstract

The scaling law has been validated in various domains, such as natural language processing (NLP) and massive computer vision tasks; however, its application to motion generation remains largely unexplored. In this paper, we introduce a scalable motion generation framework that includes the motion tokenizer Motion FSQ-VAE and a text-prefix autoregressive transformer. Through comprehensive experiments, we observe the scaling behavior of this system. For the first time, we confirm the existence of scaling laws within the context of motion generation. Specifically, our results demonstrate that the normalized test loss of our prefix autoregressive models adheres to a logarithmic law in relation to compute budgets. Furthermore, we also confirm the power law between Non-Vocabulary Parameters, Vocabulary Parameters, and Data Tokens with respect to compute budgets respectively. Leveraging the scaling law, we predict the optimal transformer size, vocabulary size, and data requirements for a compute budget of $1e18$. The test loss of the system, when trained with the optimal model size, vocabulary size, and required data, aligns precisely with the predicted test loss, thereby validating the scaling law.