Mitigating Task-Order Sensitivity and Forgetting via Hierarchical Second-Order Consolidation

TL;DR

This paper presents HTCL, a hierarchical second-order consolidation framework that improves continual learning by reducing task-order sensitivity and forgetting, with theoretical guarantees and significant empirical gains.

Contribution

It introduces a novel hierarchical Taylor series-based method for continual learning that combines local adaptation with global consolidation, extending to multiple levels for multiscale knowledge integration.

Findings

Achieves 7-25% accuracy improvements across datasets.

Reduces variability in final accuracy by up to 68%.

Provides theoretical guarantees for task sequence integration.

Abstract

We introduce $\textbf{Hierarchical Taylor Series-based Continual Learning (HTCL)}$, a framework that couples fast local adaptation with conservative, second-order global consolidation to address the high variance introduced by random task ordering. To address task-order effects, HTCL identifies the best intra-group task sequence and integrates the resulting local updates through a Hessian-regularized Taylor expansion, yielding a consolidation step with theoretical guarantees. The approach naturally extends to an $L$-level hierarchy, enabling multiscale knowledge integration in a manner not supported by conventional single-level CL systems. Across a wide range of datasets and replay and regularization baselines, HTCL acts as a model-agnostic consolidation layer that consistently enhances performance, yielding mean accuracy gains of $7\%$ to $25\%$ while reducing the standard deviation of final accuracy by up to $68\%$ across random task permutations.