Same accuracy, twice as fast: continuous training surpasses retraining from scratch

TL;DR

This paper demonstrates that continuous training methods can achieve the same accuracy as retraining from scratch but with up to 2.7 times less computational effort in computer vision tasks.

Contribution

The authors propose and evaluate optimization techniques for continuous training that significantly reduce computational costs while maintaining or improving performance.

Findings

Up to 2.7x reduction in training time.

Effective optimization methods for initialization, regularization, data selection, and hyper-parameters.

Framework for quantifying computational savings in continual learning.

Abstract

Continual learning aims to enable models to adapt to new datasets without losing performance on previously learned data, often assuming that prior data is no longer available. However, in many practical scenarios, both old and new data are accessible. In such cases, good performance on both datasets is typically achieved by abandoning the model trained on the previous data and re-training a new model from scratch on both datasets. This training from scratch is computationally expensive. In contrast, methods that leverage the previously trained model and old data are worthy of investigation, as they could significantly reduce computational costs. Our evaluation framework quantifies the computational savings of such methods while maintaining or exceeding the performance of training from scratch. We identify key optimization aspects -- initialization, regularization, data selection, and hyper-parameters -- that can each contribute to reducing computational costs. For each aspect, we propose effective first-step methods that already yield substantial computational savings. By combining these methods, we achieve up to 2.7x reductions in computation time across various computer vision tasks, highlighting the potential for further advancements in this area.