Dream2Learn: Structured Generative Dreaming for Continual Learning
Salvatore Calcagno, Matteo Pennisi, Federica Proietto Salanitri, Amelia Sorrenti, Simone Palazzo, Concetto Spampinato, Giovanni Bellitto
TL;DR
Dream2Learn introduces a novel continual learning framework where a model autonomously generates structured synthetic experiences through internal simulation, improving knowledge retention and transfer without relying on past data replay.
Contribution
The paper presents Dream2Learn, a new approach that uses internally generated, semantically distinct dreamed classes for self-training, enhancing continual learning without data rehearsal.
Findings
Outperforms rehearsal-based baselines on Mini-ImageNet, FG-ImageNet, and ImageNet-R.
Achieves positive forward transfer, improving adaptability.
Effectively reorganizes memory and supports future task learning.
Abstract
Continual learning requires balancing plasticity and stability while mitigating catastrophic forgetting. Inspired by human dreaming as a mechanism for internal simulation and knowledge restructuring, we introduce Dream2Learn (D2L), a framework in which a model autonomously generates structured synthetic experiences from its own internal representations and uses them for self-improvement. Rather than reconstructing past data as in generative replay, D2L enables a classifier to create novel, semantically distinct dreamed classes that are coherent with its learned knowledge yet do not correspond to previously observed data. These dreamed samples are produced by conditioning a frozen diffusion model through soft prompt optimization driven by the classifier itself. The generated data are not used to replace memory, but to expand and reorganize the representation space, effectively allowing…
Peer Reviews
Decision·ICLR 2026 Conference Withdrawn Submission
1) The paper provides an interesting motivation based on human sleep based replay for CL. 2) Results on standard benchmarks demonstrate superior performance of D2L compared to other replay based methods.
1) While catastrophic forgetting is an important problem to consider in CL, the main reason to not do full replay of prior data to avoid forgetting is to reduce the amount of prior data storage, and additional training on prior data to save compute. The proposed method uses significantly larger models to generate more classes to train a small CNN on a relatively small dataset (ImageNet-100). Not only does the model need to train on a large number of generated class images, the method still relie
- To my knowledge, the central idea of using generative replay for forward transfer rather than mitigating forgetting is novel and interesting. - Experiments and ablations show the effectiveness of the proposed method.
1. The method involves multiple class mappings and set operations, and their description lacks clarity (see questions below). It might be helpful to point each operation in Algorithm 1 to the sections or equations where that operation is described. 2. The method seems to rely on several class embedding manipulations, and the experiments are performed using ResNet-18. It is not clear what challenges might arise when scaling up this approach to larger data or to settings where classes are not disj
1. The paper is well organized and can be read straightforwardly. 2. The method and the experiments consider forward transfer which is overlooked in many CL works, yet is very important in CL. 3. D2L introduces an oracle network that learns when to stop the soft prompt optimization. This idea is novel and to my knowledge unexplored in previous works. 4. Experimental setup is sound and demonstrate that D2L is effective.
1. Addressing catastrophic forgetting based on generative replay is a relatively old idea in CL, including several works not referenced in the paper, and hence the novelty of this work is limited. It is true that implementation of this idea is new but the core idea is not mew. 2. D2L relies on a diffusion model which is a large model augmented to the base ResNet-like classifier. This addition makes the model far more complex and given the scope of experiments, one can argue just to use several
- The paper is well written and easy to follow. - The proposed method is novel - rather than retrospective replay, D2L introduces a prospective generation mechanism that structures the representation space for future tasks. - Oracle-guided optimization is an interesting and effective solution to avoid dream collapse. - Extensive experiments are conducted, including comparisons with SOTA methods and ablation studies. And experimental results demonstrated strong imporvements.
- The “dreaming vs. replay” distinction is interesting but not sharply formalized, and the boundary seems vague. There're OOD test, but not sufficient, as being “not old classes” doesn’t prove they’re future-oriented or structurally bridging. - D2L adopt a pretrained diffusion backbone for generating “dreamed classes.” However, how the choice of this generator impacts results is not analyzed.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Generative Adversarial Networks and Image Synthesis · Sleep and Wakefulness Research