Implementing a Machine Learning Deformer for CG Crowds: Our Journey

TL;DR

This paper shares the journey and lessons learned from developing a machine learning-based deformer for CG crowd characters, aiming to improve facial and detailed deformations in crowd rigs.

Contribution

It introduces a novel machine learning approach to compress and replicate complex rig deformations for CG crowds, addressing a key industry limitation.

Findings

Initial tests showed promise for neural network-based deformation compression

Faced multiple technical challenges and false leads during development

Provided practical insights and lessons learned from the development process

Abstract

CG crowds have become increasingly popular this last decade in the VFX and animation industry: formerly reserved to only a few high end studios and blockbusters, they are now widely used in TV shows or commercials. Yet, there is still one major limitation: in order to be ingested properly in crowd software, studio rigs have to comply with specific prerequisites, especially in terms of deformations. Usually only skinning, blend shapes and geometry caches are supported preventing close-up shots with facial performances on crowd characters. We envisioned two approaches to tackle this: either reverse engineer the hundreds of deformer nodes available in the major DCCs/plugins and incorporate them in our crowd package, or surf the machine learning wave to compress the deformations of a rig using a neural network architecture. Considering we could not commit 5+ man/years of development into this problem, and that we were excited to dip our toes in the machine learning pool, we went for the latter. From our first tests to a minimum viable product, we went through hopes and disappointments: we hit multiple pitfalls, took false shortcuts and dead ends before reaching our destination. With this paper, we hope to provide a valuable feedback by sharing the lessons we learnt from this experience.