High-Fidelity Mobile Avatars with Pruned Local Blendshapes

TL;DR

This paper introduces a mobile-friendly method for high-fidelity human avatar reconstruction from multi-view video, emphasizing local linear blendshapes to balance detail and computational efficiency.

Contribution

It presents a novel local linear blendshape approach that reduces computation and model size while maintaining high avatar detail on mobile devices.

Findings

Achieves 120 FPS at 2K resolution on mobile devices.

Produces high-quality avatars with better details than previous methods.

Uses WebGPU for cross-device implementation.

Abstract

We propose a method to reconstruct high-fidelity human avatars from multi-view video that can run on mobile devices. Many works can model high-quality Gaussian-based full-body avatars from multi-view video. However, these methods require heavy computation to obtain pose-dependent appearance, making deployment on mobile devices very difficult. Recent methods distill from pretrained models and model pose-dependent nonlinear Gaussian attributes by linearly combining global pose features with blendshapes. Although they can run on mobile devices, they suffer some loss of detail. We observe that nearby Gaussians are often highly correlated within a local region of the body, and can be linearly modeled with less error. Therefore, we use local linear blendshapes in small body parts to capture global nonlinear changes of Gaussian attributes. To further reduce computation and model size, we propose to remove blendshapes for Gaussians whose attributes change little, yielding a minimal blendshape representation. Our method is an end-to-end training method without a pretrained model. To make it run on multiple devices, we implement our method using WebGPU. Experiments show that our method can render high-quality human avatars with better details, and can reach 120 FPS at 2K resolution on mobile devices.