Towards Motion Turing Test: Evaluating Human-Likeness in Humanoid Robots

TL;DR

This paper introduces the Motion Turing Test framework and the HHMotion dataset to evaluate and quantify how human-like humanoid robot motions are, revealing current limitations and proposing a baseline for automatic assessment.

Contribution

The paper presents the first Motion Turing Test framework, a comprehensive HHMotion dataset, and a baseline model for automatic human-likeness evaluation of humanoid robot motions.

Findings

Humanoid motions still significantly deviate from human movements.

Current multimodal large language models are inadequate for motion assessment.

A simple baseline model outperforms several LLM-based methods.

Abstract

Humanoid robots have achieved significant progress in motion generation and control, exhibiting movements that appear increasingly natural and human-like. Inspired by the Turing Test, we propose the Motion Turing Test, a framework that evaluates whether human observers can discriminate between humanoid robot and human poses using only kinematic information. To facilitate this evaluation, we present the Human-Humanoid Motion (HHMotion) dataset, which consists of 1,000 motion sequences spanning 15 action categories, performed by 11 humanoid models and 10 human subjects. All motion sequences are converted into SMPL-X representations to eliminate the influence of visual appearance. We recruited 30 annotators to rate the human-likeness of each pose on a 0-5 scale, resulting in over 500 hours of annotation. Analysis of the collected data reveals that humanoid motions still exhibit noticeable deviations from human movements, particularly in dynamic actions such as jumping, boxing, and running. Building on HHMotion, we formulate a human-likeness evaluation task that aims to automatically predict human-likeness scores from motion data. Despite recent progress in multimodal large language models, we find that they remain inadequate for assessing motion human-likeness. To address this, we propose a simple baseline model and demonstrate that it outperforms several contemporary LLM-based methods. The dataset, code, and benchmark will be publicly released to support future research in the community.