More Than Memory Savings: Zeroth-Order Optimization Mitigates Forgetting in Continual Learning

TL;DR

This paper explores how zeroth-order optimization can reduce forgetting in continual learning by promoting flatter loss landscapes, but it also introduces a new hybrid method, ZO-FC, that balances stability and plasticity effectively.

Contribution

The paper introduces ZO-FC, a hybrid approach applying ZO to a PEFT module with FO classifier, balancing stability and plasticity in continual learning.

Findings

ZO optimization leads to flatter loss landscapes and reduces forgetting.

ZO-FC achieves a better stability-plasticity trade-off in continual learning.

ZOFc maintains memory efficiency while improving adaptability.

Abstract

Zeroth-order (ZO) optimization has gained attention as a memory-efficient alternative to first-order (FO) methods, particularly in settings where gradient computation is expensive or even impractical. Beyond its memory efficiency, in this work, we investigate ZO optimization for continual learning (CL) as a novel approach to address the plasticity-stability-efficiency trilemma. Through theoretical analysis and empirical evidence, we show that ZO optimization naturally leads to flatter loss landscapes, which in turn reduce forgetting in CL. However, this stability comes at a cost of plasticity: due to its imprecise gradient estimates and slower convergence, ZO optimization tends to be less effective than FO in acquiring new task-specific knowledge, particularly under constrained training budgets. To better understand this trade-off, we conduct a holistic evaluation of ZO optimization applied to various existing CL methods. Our findings reveal that ZO optimization enhances stability but often undermines plasticity, particularly when used with learnable classifiers. Motivated by this insight, we propose ZO-FC, a simple but effective approach that applies ZO optimization to a single adapter-based PEFT module with FO optimized classifier. This design leverages the stability benefits of ZO while preserving the adaptability of FO updates with negligible memory overhead. Experiments demonstrate that ZO-FC achieves an effective balance between stability and plasticity, offering a practical and memory-efficient solution for on-device CL.