Balancing Stability and Plasticity in Sequentially Trained Early-Exiting Neural Networks

TL;DR

This paper introduces methods to improve sequential training of early-exiting neural networks by balancing stability and plasticity, leading to better accuracy and efficiency.

Contribution

It proposes two novel approaches, Elastic Weight Consolidation and Learning without Forgetting, to mitigate interference in sequentially trained early-exit neural networks.

Findings

Enhanced accuracy of early exits over existing methods

Achieved significant speedups at low computational budgets

Consistent performance improvements across benchmarks

Abstract

Early-exiting neural networks enable adaptive inference by allowing inputs to exit at intermediate classifiers, reducing computation for easy samples while maintaining high accuracy. In practice, exits can be trained sequentially by incrementally adding them to a shared backbone; however, this sequential training can cause newly introduced exits to interfere with previously learned ones, degrading the performance of earlier classifiers. We address this problem by retaining the knowledge embedded in existing exits while allowing new ones to specialize. We propose two alternative approaches that operate at different levels of the model. The first constrains learning by protecting parameters that are important for previously trained exits, while the second preserves the output distributions of earlier exits as the network adapts. These alternatives directly reflect the stability-plasticity trade-off studied in continual learning. Accordingly, we leverage \textit{Elastic Weight Consolidation} to constrain critical weights and \textit{Learning without Forgetting} to preserve output distributions. Experiments on standard benchmarks show that our approaches consistently improve early-exit performance, achieving higher accuracy over existing sequential training methods and significant performance speedups at low computational budgets.