Pathway-based Progressive Inference (PaPI) for Energy-Efficient Continual Learning

TL;DR

This paper introduces Pathway-based Progressive Inference (PaPI), a theoretical framework for energy-efficient continual learning that improves stability-plasticity trade-offs and reduces forgetting through pathway selection, with formal guarantees and empirical validation.

Contribution

PaPI provides a novel pathway-based approach with formal convergence guarantees, improving continual learning stability and energy efficiency over existing methods.

Findings

PaPI achieves an $ ext{O}(K)$ improvement in stability-plasticity trade-off.

PaPI's energy consumption scales with active parameters, not total model size.

PaPI outperforms EWC and GEM in theoretical guarantees and empirical benchmarks.

Abstract

Continual learning systems face the dual challenge of preventing catastrophic forgetting while maintaining energy efficiency, particularly in resource-constrained environments. This paper introduces Pathway-based Progressive Inference (PaPI), a novel theoretical framework that addresses these challenges through a mathematically rigorous approach to pathway selection and adaptation. We formulate continual learning as an energy-constrained optimization problem and provide formal convergence guarantees for our pathway routing mechanisms. Our theoretical analysis demonstrates that PaPI achieves an $\mathcal{O}(K)$ improvement in the stability-plasticity trade-off compared to monolithic architectures, where $K$ is the number of pathways. We derive tight bounds on forgetting rates using Fisher Information Matrix analysis and prove that PaPI's energy consumption scales with the number of active parameters rather than the total model size. Comparative theoretical analysis shows that PaPI provides stronger guarantees against catastrophic forgetting than Elastic Weight Consolidation (EWC) while maintaining better energy efficiency than both EWC and Gradient Episodic Memory (GEM). Our experimental validation confirms these theoretical advantages across multiple benchmarks, demonstrating PaPI's effectiveness for continual learning in energy-constrained settings. Our codes are available at https://github.com/zser092/PAPI_FILES.