CLIP-Actor: Text-Driven Recommendation and Stylization for Animating Human Meshes

TL;DR

CLIP-Actor is a novel system that generates human mesh animations from text prompts by recommending motions and applying neural style optimization, ensuring temporally consistent and realistic results.

Contribution

It introduces a zero-shot neural style optimization method and a text-driven motion recommendation system for 3D human meshes, addressing limitations of prior pose-dependent approaches.

Findings

Produces plausible, human-recognizable 3D animations from text

Ensures temporal consistency and pose-agnostic stylization

Leverages multi-frame data for stable optimization

Abstract

We propose CLIP-Actor, a text-driven motion recommendation and neural mesh stylization system for human mesh animation. CLIP-Actor animates a 3D human mesh to conform to a text prompt by recommending a motion sequence and optimizing mesh style attributes. We build a text-driven human motion recommendation system by leveraging a large-scale human motion dataset with language labels. Given a natural language prompt, CLIP-Actor suggests a text-conforming human motion in a coarse-to-fine manner. Then, our novel zero-shot neural style optimization detailizes and texturizes the recommended mesh sequence to conform to the prompt in a temporally-consistent and pose-agnostic manner. This is distinctive in that prior work fails to generate plausible results when the pose of an artist-designed mesh does not conform to the text from the beginning. We further propose the spatio-temporal view augmentation and mask-weighted embedding attention, which stabilize the optimization process by leveraging multi-frame human motion and rejecting poorly rendered views. We demonstrate that CLIP-Actor produces plausible and human-recognizable style 3D human mesh in motion with detailed geometry and texture solely from a natural language prompt.