A Study on Efficiency in Continual Learning Inspired by Human Learning

TL;DR

This paper investigates the efficiency of continual learning systems inspired by human learning, focusing on pruning algorithms like PackNet, and explores the analogy between sleep cycles and pruning phases to optimize performance.

Contribution

The study analyzes the efficiency of pruning-based continual learning, revealing that weight freezing increases resource usage and drawing parallels between sleep cycles and pruning phases for optimization.

Findings

Weight freezing can double the number of weights used for the same performance.

Pruning phases with a time budget relate to human sleep cycles.

Optimal iteration versus epoch settings depend on the task.

Abstract

Humans are efficient continual learning systems; we continually learn new skills from birth with finite cells and resources. Our learning is highly optimized both in terms of capacity and time while not suffering from catastrophic forgetting. In this work we study the efficiency of continual learning systems, taking inspiration from human learning. In particular, inspired by the mechanisms of sleep, we evaluate popular pruning-based continual learning algorithms, using PackNet as a case study. First, we identify that weight freezing, which is used in continual learning without biological justification, can result in over $2\times$ as many weights being used for a given level of performance. Secondly, we note the similarity in human day and night time behaviors to the training and pruning phases respectively of PackNet. We study a setting where the pruning phase is given a time budget, and identify connections between iterative pruning and multiple sleep cycles in humans. We show there exists an optimal choice of iteration v.s. epochs given different tasks.