Continual Learning of Personalized Generative Face Models with Experience Replay

TL;DR

This paper addresses continual learning for personalized face generative models, proposing a novel experience replay method using StyleGAN's latent space to prevent forgetting during sequential updates.

Contribution

It introduces a convex hull-based experience replay algorithm that improves retention of past face representations in continual learning scenarios.

Findings

Convex hull-based replay outperforms random sampling in preventing forgetting.

Simple experience replay mitigates forgetting with large storage, but struggles with limited memory.

The proposed method enhances long-term model stability in continual face modeling.

Abstract

We introduce a novel continual learning problem: how to sequentially update the weights of a personalized 2D and 3D generative face model as new batches of photos in different appearances, styles, poses, and lighting are captured regularly. We observe that naive sequential fine-tuning of the model leads to catastrophic forgetting of past representations of the individual's face. We then demonstrate that a simple random sampling-based experience replay method is effective at mitigating catastrophic forgetting when a relatively large number of images can be stored and replayed. However, for long-term deployment of these models with relatively smaller storage, this simple random sampling-based replay technique also forgets past representations. Thus, we introduce a novel experience replay algorithm that combines random sampling with StyleGAN's latent space to represent the buffer as an optimal convex hull. We observe that our proposed convex hull-based experience replay is more effective in preventing forgetting than a random sampling baseline and the lower bound.