Explaining Robustness to Catastrophic Forgetting Through Incremental Concept Formation

TL;DR

This paper investigates why hierarchical concept formation models like Cobweb/4V are robust to catastrophic forgetting, highlighting adaptive restructuring, sparse updates, and information-theoretic learning as key factors.

Contribution

It introduces Cobweb/4V and compares it with neural baselines, demonstrating how structural reorganization, sparse updates, and information-theoretic learning improve continual learning stability.

Findings

Adaptive restructuring enhances learning plasticity.

Sparse updates reduce interference between tasks.

Information-theoretic learning preserves prior knowledge.

Abstract

Catastrophic forgetting remains a central challenge in continual learning, where models are required to integrate new knowledge over time without losing what they have previously learned. In prior work, we introduced Cobweb/4V, a hierarchical concept formation model that exhibited robustness to catastrophic forgetting in visual domains. Motivated by this robustness, we examine three hypotheses regarding the factors that contribute to such stability: (1) adaptive structural reorganization enhances knowledge retention, (2) sparse and selective updates reduce interference, and (3) information-theoretic learning based on sufficiency statistics provides advantages over gradient-based backpropagation. To test these hypotheses, we compare Cobweb/4V with neural baselines, including CobwebNN, a neural implementation of the Cobweb framework introduced in this work. Experiments on datasets of varying complexity (MNIST, Fashion-MNIST, MedMNIST, and CIFAR-10) show that adaptive restructuring enhances learning plasticity, sparse updates help mitigate interference, and the information-theoretic learning process preserves prior knowledge without revisiting past data. Together, these findings provide insight into mechanisms that can mitigate catastrophic forgetting and highlight the potential of concept-based, information-theoretic approaches for building stable and adaptive continual learning systems.