Muscles in Action

TL;DR

This paper introduces Muscles in Action, a dataset combining video and electromyography data to model muscle activity and human motion, enabling more realistic virtual human representations.

Contribution

The paper presents a new dataset and a bidirectional model linking muscle activity and motion, advancing the integration of internal muscle data into computer vision.

Findings

Model predicts muscle activation from video

Reconstructs motion from muscle signals

Enhances virtual human realism

Abstract

Human motion is created by, and constrained by, our muscles. We take a first step at building computer vision methods that represent the internal muscle activity that causes motion. We present a new dataset, Muscles in Action (MIA), to learn to incorporate muscle activity into human motion representations. The dataset consists of 12.5 hours of synchronized video and surface electromyography (sEMG) data of 10 subjects performing various exercises. Using this dataset, we learn a bidirectional representation that predicts muscle activation from video, and conversely, reconstructs motion from muscle activation. We evaluate our model on in-distribution subjects and exercises, as well as on out-of-distribution subjects and exercises. We demonstrate how advances in modeling both modalities jointly can serve as conditioning for muscularly consistent motion generation. Putting muscles into computer vision systems will enable richer models of virtual humans, with applications in sports, fitness, and AR/VR.