Temporal-Difference Variational Continual Learning

TL;DR

This paper introduces a novel variational continual learning method that integrates multiple past posterior estimates to reduce approximation errors and employs Temporal-Difference concepts, significantly improving knowledge retention in sequential learning tasks.

Contribution

It proposes new learning objectives combining past posterior regularization with TD-inspired techniques, addressing catastrophic forgetting more effectively than existing variational methods.

Findings

Outperforms existing Variational CL methods on benchmarks

Effectively mitigates catastrophic forgetting

Connects variational updates with Temporal-Difference learning

Abstract

Machine Learning models in real-world applications must continuously learn new tasks to adapt to shifts in the data-generating distribution. Yet, for Continual Learning (CL), models often struggle to balance learning new tasks (plasticity) with retaining previous knowledge (memory stability). Consequently, they are susceptible to Catastrophic Forgetting, which degrades performance and undermines the reliability of deployed systems. In the Bayesian CL literature, variational methods tackle this challenge by employing a learning objective that recursively updates the posterior distribution while constraining it to stay close to its previous estimate. Nonetheless, we argue that these methods may be ineffective due to compounding approximation errors over successive recursions. To mitigate this, we propose new learning objectives that integrate the regularization effects of multiple previous posterior estimations, preventing individual errors from dominating future posterior updates and compounding over time. We reveal insightful connections between these objectives and Temporal-Difference methods, a popular learning mechanism in Reinforcement Learning and Neuroscience. Experiments on challenging CL benchmarks show that our approach effectively mitigates Catastrophic Forgetting, outperforming strong Variational CL methods.