Arnold: a generalist muscle transformer policy

TL;DR

Arnold is a versatile transformer-based policy that learns multiple musculoskeletal control tasks and embodiments, enabling rapid adaptation and providing insights into biological motor control.

Contribution

It introduces Arnold, a generalist muscle transformer policy that combines behavior cloning and PPO, with a novel sensorimotor vocabulary for multi-task, multi-embodiment learning.

Findings

Achieves expert performance in 14 control tasks

Supports rapid adaptation to new tasks

Provides insights into muscle synergy transferability

Abstract

Controlling high-dimensional and nonlinear musculoskeletal models of the human body is a foundational scientific challenge. Recent machine learning breakthroughs have heralded policies that master individual skills like reaching, object manipulation and locomotion in musculoskeletal systems with many degrees of freedom. However, these agents are merely "specialists", achieving high performance for a single skill. In this work, we develop Arnold, a generalist policy that masters multiple tasks and embodiments. Arnold combines behavior cloning and fine-tuning with PPO to achieve expert or super-expert performance in 14 challenging control tasks from dexterous object manipulation to locomotion. A key innovation is Arnold's sensorimotor vocabulary, a compositional representation of the semantics of heterogeneous sensory modalities, objectives, and actuators. Arnold leverages this vocabulary via a transformer architecture to deal with the variable observation and action spaces of each task. This framework supports efficient multi-task, multi-embodiment learning and facilitates rapid adaptation to novel tasks. Finally, we analyze Arnold to provide insights into biological motor control, corroborating recent findings on the limited transferability of muscle synergies across tasks.