Frugal Incremental Generative Modeling using Variational Autoencoders

TL;DR

This paper introduces a replay-free incremental learning model using Variational Autoencoders with a multi-modal latent space and orthogonality criterion, achieving state-of-the-art accuracy while significantly reducing memory usage.

Contribution

The work presents a novel incremental generative model with a multi-modal latent space and an orthogonality criterion to prevent catastrophic forgetting, without increasing model size.

Findings

Achieves state-of-the-art accuracy in incremental learning.

Reduces memory usage by at least an order of magnitude.

Effective in preventing catastrophic forgetting.

Abstract

Continual or incremental learning holds tremendous potential in deep learning with different challenges including catastrophic forgetting. The advent of powerful foundation and generative models has propelled this paradigm even further, making it one of the most viable solution to train these models. However, one of the persisting issues lies in the increasing volume of data particularly with replay-based methods. This growth introduces challenges with scalability since continuously expanding data becomes increasingly demanding as the number of tasks grows. In this paper, we attenuate this issue by devising a novel replay-free incremental learning model based on Variational Autoencoders (VAEs). The main contribution of this work includes (i) a novel incremental generative modelling, built upon a well designed multi-modal latent space, and also (ii) an orthogonality criterion that mitigates catastrophic forgetting of the learned VAEs. The proposed method considers two variants of these VAEs: static and dynamic with no (or at most a controlled) growth in the number of parameters. Extensive experiments show that our method is (at least) an order of magnitude more ``memory-frugal'' compared to the closely related works while achieving SOTA accuracy scores.