Morpheus: A Neural-driven Animatronic Face with Hybrid Actuation and Diverse Emotion Control

TL;DR

Morpheus introduces a hybrid-actuated animatronic face controlled by neural networks, enabling diverse and nuanced emotional expressions from speech, overcoming previous hardware and software limitations in expressive robotic faces.

Contribution

The paper presents a novel hybrid actuation design combined with neural network-based control for expressive animatronic faces, including a self-modeling network for automatic motor-to-expression mapping.

Findings

Able to generate a range of emotions from speech input

Achieves precise control of key facial features with hybrid actuation

Demonstrates nuanced, emotion-specific facial expressions

Abstract

Previous animatronic faces struggle to express emotions effectively due to hardware and software limitations. On the hardware side, earlier approaches either use rigid-driven mechanisms, which provide precise control but are difficult to design within constrained spaces, or tendon-driven mechanisms, which are more space-efficient but challenging to control. In contrast, we propose a hybrid actuation approach that combines the best of both worlds. The eyes and mouth-key areas for emotional expression-are controlled using rigid mechanisms for precise movement, while the nose and cheek, which convey subtle facial microexpressions, are driven by strings. This design allows us to build a compact yet versatile hardware platform capable of expressing a wide range of emotions. On the algorithmic side, our method introduces a self-modeling network that maps motor actions to facial landmarks, allowing us to automatically establish the relationship between blendshape coefficients for different facial expressions and the corresponding motor control signals through gradient backpropagation. We then train a neural network to map speech input to corresponding blendshape controls. With our method, we can generate distinct emotional expressions such as happiness, fear, disgust, and anger, from any given sentence, each with nuanced, emotion-specific control signals-a feature that has not been demonstrated in earlier systems. We release the hardware design and code at https://github.com/ZZongzheng0918/Morpheus-Hardware and https://github.com/ZZongzheng0918/Morpheus-Software.