MoRe: Modular Representations for Principled Continual Representation Learning on Sequential Data

TL;DR

MoRe introduces a modular representation framework inspired by the human brain to improve continual learning by decomposing knowledge into hierarchical modules with guaranteed identifiability.

Contribution

MoRe uniquely identifies modularity within representations rather than architecture, enabling principled reuse, alignment, and expansion during continual adaptation.

Findings

Demonstrates interpretable hierarchical structure in synthetic and real-world data.

Shows improved plasticity-stability trade-offs in experiments.

Validates the approach on LLM activation data.

Abstract

Continual learning requires models to adapt to new data while preserving previously acquired knowledge. At its core, this challenge can be viewed as principled one-step adaptation: incorporating new information with minimal interference to existing representations. Most existing approaches address this challenge by modifying model parameters or architectures in a supervised, task-specific manner. However, the underlying issue is representational: tasks require distinct yet structured representations that can be selectively updated without disrupting representations, while structure should reflect intrinsic organization in the data rather than task boundaries. In sequential data, time-delayed dependencies provide a natural signal for uncovering this organization, revealing how fundamental representations give rise to more specific ones. Inspired by the modular organization of the human brain, we propose MoRe, a framework that identifies modularity in the representation itself rather than allocating it at the architectural level. MoRe decomposes knowledge into a hierarchy of fundamental and specific modules with identifiability guarantees, enabling principled module reuse, alignment, and expansion during adaptation while preserving old modules by construction. Experiments on synthetic benchmarks and real-world LLM activations demonstrate interpretable hierarchical structure, improved plasticity-stability trade-offs, suggesting MoRe as a principled foundation for continual adaptation