How Much Do Large Language Models Know about Human Motion? A Case Study in 3D Avatar Control

TL;DR

This study investigates the extent of large language models' understanding of human motion by using them to generate 3D avatar animations from instructions, revealing strengths in high-level planning and weaknesses in detailed positioning.

Contribution

The paper introduces a novel framework for evaluating LLMs' knowledge of human motion through a two-step planning process for 3D avatar control, highlighting their capabilities and limitations.

Findings

LLMs excel at high-level movement interpretation

Struggle with precise body part positioning

Decomposition improves motion planning

Abstract

We explore the human motion knowledge of Large Language Models (LLMs) through 3D avatar control. Given a motion instruction, we prompt LLMs to first generate a high-level movement plan with consecutive steps (High-level Planning), then specify body part positions in each step (Low-level Planning), which we linearly interpolate into avatar animations. Using 20 representative motion instructions that cover fundamental movements and balance body part usage, we conduct comprehensive evaluations, including human and automatic scoring of both high-level movement plans and generated animations, as well as automatic comparison with oracle positions in low-level planning. Our findings show that LLMs are strong at interpreting high-level body movements but struggle with precise body part positioning. While decomposing motion queries into atomic components improves planning, LLMs face challenges in multi-step movements involving high-degree-of-freedom body parts. Furthermore, LLMs provide reasonable approximations for general spatial descriptions, but fall short in handling precise spatial specifications. Notably, LLMs demonstrate promise in conceptualizing creative motions and distinguishing culturally specific motion patterns.